|
|
|
Measurement,
Evaluation, and Research Methodology Program
Program
Description and Plan
Table of Contents
Philosophy
and Mandate
Vision
Statements
The
Graduate Program in MERM
General
Description
Typical
Patterns of Graduation
Program
Planning
Priorities
for Planning.
Research
Interests of Core MERM Faculty.
Research
Interests for Individuals Who Periodically Teach in the MERM Program...
APPENDIX
A: Degree Programs
(M.Ed., M.A, and Ph.D.).
APPENDIX
B: A Survey of Competencies and Characteristics of Entry-Level Measurement
Specialists and/or Quantitative Methodologists.
Last Updated: December 20, 2005.
Measurement,
Evaluation, and Research Methodology Program
The
University of British Columbia
The
Measurement, Evaluation, and Research Methodology (MERM) program
recognizes that measurement, evaluation and research methodology
is an evolving field that is trans-disciplinary by nature and is
at the core of many of the research activities in the Department
of Educational and Counseling Psychology, and Special Education,
the Faculty of Education, and in many of the human and health sciences
in the University.
As evidence for this trans-disciplinary nature, we note that
our classes are often in high demand not only by the various disciplines
in the Faculty of Education, including Human Kinetics, but also
by students in Commerce, Health Studies, Nursing and Psychology.
Furthermore, the faculty members in MERM often advise colleagues,
sit on graduate student committees across the university, and provide
advice to graduate students for whom we are not formal committee
members.
In
recognition of this trans-disciplinary feature MERM plays two important
roles within the Department, Faculty, and University.
1. We develop and share our collective knowledge in:
- assessment,
evaluation, testing, and other aspects of educational and psychological
measurement. We are involved in the construction and use of standardized
tests; new forms of assessment, including the development and
testing of measurement methodology,
- program
evaluation, including methods for quasi-experiments, hierarchical
or multi-level models, and the analysis of growth and change,
- qualitative
methods, and
- data
analysis and statistical methods for research problems in psychology
and education,
2. We prepare graduate students as methodological
and measurement specialists. Emphasis in the MERM area is placed
on advanced research as applied to educational, psychological, health,
and social contexts.
An
essential feature of the above roles is the generation and sharing
of knowledge and information through our publications, seminars,
workshops, and membership on technical advisory panels.
We
strive to promote in our research, student supervision, and teaching
the highest standards of measurement and research methodology practice
in our discipline. This has meant that we have two shared goals:
(a) providing courses for researchers, graduate students,
and practitioners outside of MERM, and (b) the preparation of graduate
students as methodological and measurement specialists.
Integrating
research, practice, and teaching to promote the highest
standards of measurement, evaluation, and research methodology
|
Vision
Statements
- We will
work toward expanding the enrollments in our M.A., M.Ed., and
Ph.D. MERM program.
- We
will diversify and monitor our course offerings to meet proactively
the increasing complexity and diversity of data analysis and measurement
methods used in educational and psychological research.
- We
will continue to be a resource to our colleagues and graduate
students outside of the MERM area, and to various governmental
and non-governmental organizations in Canada and around the world.
The
Graduate Program in MERM
General Description
The
Measurement, Evaluation, and Research Methodology program
at the University of British Columbia is comprised of five
core faculty members (Ercikan, Hubley, Kishor, Mathison, and Zumbo) and five associate
faculty members . The term "core faculty" is being used in the same way that
it is used in APA and CPA accreditation processes: core faculty are those
individuals who have active roles in the development and governance of the
program and assume primary responsibility for the training of the program’s
students.
The
MERM program is a small program compared to other
larger Programs (e.g., Counseling Psychology) in our Department.
The MERM Program is comprised of faculty whose primary focus is
MERM and others for whom MERM is a secondary or tertiary areas of
scholarly interest.
The MERM program offers excellent opportunities for students
with the aptitude and interest for this field. Emphasis in the MERM
area is placed on advanced research in a variety of contexts including
educational, psychological, health, and social services contexts.
Students concentrating in this area will generally fit into one
of three categories.
1.
Students who have an applied interest in measurement, program evaluation,
or data analysis. While they often have some preparation in measurement
and data analysis in their undergraduate studies, this is not always
the case. These students are more oriented toward the use of measurement,
program evaluation, or data analysis techniques in substantive research
in the areas of test development, personality assessment, clinical
measurement, computer applications, program planning or evaluation,
and so on.
2.
Students possessing strong theoretical interests in technical problems
in test theory, item response theory, assessment, statistics, factor
analysis, multi-level modeling, and the like. Also, some of these
students come to the Program with some statistical and/or mathematical
background -- often obtained while studying in another social science
such as Psychology or Sociology although sometimes students arrive
with degrees in statistics or mathematics.
3.
Some students will find it compatible with their career goals to
give equal attention to both these major aspects (i.e., applied
and theoretical) of this program.
Appendix A provides a detailed description of our M.A., M.Ed., and
Ph.D. programs.
The
M.A. and Ph.D. students in this field will be expected to become
involved in research activities culminating in the execution of
their own Masters thesis or Doctoral dissertation research. The
M.Ed. degree is primarily a course-based Masters degree.
MERM
students interested in the technical side of psychometrics may also
take related courses outside the department, including Statistics,
Computer Science, Psychology, and Epidemiology.
Upon
completion of our program our graduates often take positions as:
university faculty; test developers; school district and provincial
government testing and research directors; professional evaluators;
educational statisticians; research consultants; testing specialists
in business, industry, education, community programs; and licensure,
certification, and credentialing professionals.
Recent
graduates from our Program have taken positions at the Rutgers University
(U.S.A.), Department of Pharmaceutical Sciences UBC, and testing
an research organizations such as CTB/McGraw-Hill as Psychometricians
and Research Scientists.
In
order to get a sense of the graduation patterns of MERM-type students,
the report by Sanderson and Dugoni (1999) Summary report 1997:
Doctorate recipients from United States universities (Chicago:
National Opinion Research Center), was consulted to obtain the number
of doctoral degrees awarded in psychology and those granted in educational
and school psychology in departments, colleges, and faculties of
education.
Although this is data from the United States, it will be
useful in giving us a sense of the potential candidate pool that
MERM may expect and the proportion of graduates we can expect relative
to the other programs in our department. Only the sub-fields that
are currently in our department are listed in Table 1.
Table
1. Doctoral Degrees Awarded in 1996 by Subfield
Subfield
|
Number
|
Percent
|
Counseling
|
464
|
34.9
|
Educational
PhD/EdD
|
401
|
30.1
|
School
PhD/EdD
|
196
|
14.7
|
Developmental
& Child
|
188
|
14.1
|
Family/Marriage
Counseling
|
52
|
3.9
|
Psychometrics
and Quantitative
|
30
|
2.3
|
Total
|
1331
|
100
|
Table
1 suggests that if the US pattern fits in our context we can reasonably
expect that approximately 2 out of every 100 students our department
graduates would be MERM graduates (of course, this implies that
our enrollments would be approximately 2 percent of the graduate
students in our department).
As of January 2001, there were 398 masters and doctoral students
in our department, of which 12 (3%) are MERM. However, the department
has 78 PhD students of which 5 (6.4%) are MERM students .
Clearly, although student recruitment is a priority for us,
our enrollments are in line with national and international trends.
What
is expected of our MERM Graduate Students
Appendix
B contains a report of a survey conducted in the Fall of 2000 that
was planned to determine the required competencies and characteristics
of entry-level measurement specialists and/or quantitative methodologists.
With regards to academic program planning, the results speak to
several recommendations:
- With
regard to a longstanding question of striking a balance between
measurement and evaluation training (i.e., specialist training)
and knowledge of a domain outside of measurement, evaluation,
and research methodology, the results highlight that students
must be prepared to demonstrate expertise in another domain besides
measurement and quantitative methods. What is unclear, however,
is the level of expertise expected. We believe that our graduate
students meet this expectation because (a) our graduate programs
require our students to complete coursework outside and (b) our
graduate students typically come from undergraduate degrees in
fields outside of MERM, per se (e.g., teacher education, psychology,
history, sociology, political science).
- At
the same time that potential employers would like expertise outside
of MERM the question of the importance of competencies and characteristics
(Table 1 of Appendix B) highlighted that there is an expectation
that mathematical and statistical knowledge of the measurement
models is important.
- With
regard to written and oral communication skills, most, if not
all, of our MERM courses (except, perhaps the general service
data analysis course) require our students to write and present
papers/projects.
- Although
over half of the respondents to the survey required their applicants
to have a doctorate, a substantial portion of the market still
has a place for individuals who have less than a doctorate. This highlights the importance of our M.Ed.
and M.A. programs in MERM.
MERM Courses
In
our discussion of MERM courses it is important to use the terminology
of “introductory”, “intermediate”, and “advanced” courses. In this
light, all of our courses, irrespective of the level of the course,
are open to students across the Department (and actually across
the Faculty, and in some cases, the University). What this means
is that we are no longer using the language of
"specialized" courses and also "service"
versus "non-service" courses. Instead, the language of
"introductory, intermediate, and advanced" courses makes
it clearer that all of our courses are required by several of the
ECPS Programs.
Of
course, all of the Programs in ECPS require that their students
take some combination of introductory to advanced courses. For example,
all SCPS, HLDI, SPED, and CNPS doctoral students have methodological
requirements and some of those requirements are met through our
intermediate and advanced courses.
Given
the above description, every academic year we try and strike a balance
so that we have at least one intermediate or advanced course in
each of the winter terms.
What
follows is an example of course offerings during an academic year:
Introductory/basic
(EPSE 481, 482, 528, 592, 595):
15 sections
Intermediate
(EPSE 596) 1 section
Advanced
(EPSE 597, 681) 2 sections
Of
course, whether a course is considered introductory, intermediate,
or advanced is somewhat arbitrary but you will get a sense of our
course offerings from the list above.
Priorities
for Planning
Annually we have courses on advanced topics such as IRT, factor
analysis and structural equation modeling, test construction, equating,
DIF, multi-level modeling, and measurement and analysis of change
and growth.
From
the survey and discussion among MERM faculty we set the following
priorties:
- Priority
should be given to offer IRT and advanced measurement models on
a regular cycle (at least every two years)
- A
course should be offered regularly on survey sampling theory and
design for educational and psychological research
- Likewise
a course on categorical data analysis would be useful.
A
strategic plan will be prepared for recruiting MERM students. The
following tasks will be a part of this plan:
- The
MERM webpage will be further developed
- A
brochure and poster will be prepared that will be sent to various
departments of psychology and sociology as well as faculties of
Education across Canada, U.S.A, and key recruiting sites internationally.
- Formalizing
a minor in MERM
Core
Faculty are those individuals who have active roles in the development and
governance of the program and assume primary responsibility for the training
of the program’s students. The
following individuals regularly teach MERM courses and supervise
MERM Graduate Students.
Kadriye
Ercikan, Ph.D. Associate Professor (MERM
Program) Research design and methodology; measurement and psychometrics;
item response theory; cross-cultural issues in assessment, adaptation
and translation issues in assessment, psychometric issues in large-scale
assessments; international assessments. http://www.ecps.educ.ubc.ca/faculty/k_erikan.htm
Anita
Hubley, Ph.D. Associate Professor
(MERM Program) Applied measurement and test development (social,
cognitive-neuropsychological measures); adult assessment (personality,
intelligence, neuropsychology); lifespan development with a focus
on adulthood and aging (e.g., age identity, body image, depression,
memory); quantitative and qualitative research methods and analyses.
http://www.educ.ubc.ca/faculty/hubley/index.html
Nand
Kishor, Ph.D. Associate Professor (MERM
Program) Educational and psychological measurement; scaling; item
response modeling; equating & standard setting; computer adaptive
and dynamic assessment; program evaluation; performance evaluation;
multivariate statistical methods; cross-cultural research methodology;
meta-analysis methods; cognition: judgement and decision-making. http://www.ecps.educ.ubc.ca/faculty/n_kishor.htm
Sandra Mathison, Ph.D. Professor (MERM Program). Educational
evaluation, Democratic and participatory evaluation, Sociology of
assessment, Impact of mandated standardized testing on students,
teachers and schools, Qualitative research methods, Uses of alternative
representation in research and evaluation. http://www.ecps.educ.ubc.ca/faculty/s_mathison.htm
Bruno
D. Zumbo, Ph.D. Professor (MERM Program). My interests focus in measurement and statistical science. Most
recently I have worked on latent variable and observed score psychometric
models including complex tasks, complex survey, measurement of change,
and differential item functioning. My methodological developments
are applied and tested, by me and others, in the domains of educational
and psychological research, language testing, health studies, and
in quality of life and subjective well-being research.
Recent papers include studies of Bayesian methods in IRT,
nonparametric statistics, DIF detection methods, and reliability
theory. I am also formally affiliated with the Department of Statistics
and the Department of Psychology at UBC.
/zumbo/zumbo.html
Although
MERM is not their primary area of research and teaching, the following
individuals teach MERM courses.
Marshall
Arlin, Ph.D. Professor
(MERM and HLDI Programs) Critical thinking, problem solving, learning
strategies, creativity. Research design and analysis of variance.
Deborah
Butler, Ph.D. Associate Professor
(SPED, HLDI, and MERM Programs) Learning disabilities in adolescence
and adulthood; instructional strategies for promoting self-regulated
and strategic learning; cognitive learning theories; research design
and methodology; qualitative and quantitative methods and analyses.
Joe
Lucyshyn, Ph.D. Assistant Professor (SPED and MERM Programs)
Developmental disabilities; positive behavioral support; assessment
and intervention with families of children and youth with behavior
problems; home-school collaboration; instructional technology; small
N research design; qualitative research methods.
William
McKee, Ph.D.
Assistant Professor (SCPS, SPED, and MERM Programs) Professional
issues in school psychology and special education including service
delivery, ethics, legislation and training; consultation processes;
intervention for behavioural disorders; intervention implementation
and treatment integrity; applied behaviour analysis; single case
and small N research design.
Jennifer D. Shapka, Ph.D. Assistant Professor
(HLDI ) Developmental perspectives on gender, motivation and education (technology,
science, and math education). Identifying risk and protective factors
impacting high risk youth populations (i.e., teenage mothers and their children).
Examining patterns of development at the population level, using large-scale
longitudinal databases (i.e., NLSCY).
Bruno
D. Zumbo, Anita M. Hubley, and Bonnie Davidson,
University
of British Columbia
We
conducted a survey of potential employers to determine the required
competencies and characteristics of entry-level measurement specialists
and/or quantitative methodologists. Our main goal was to obtain
information for academic program planning but along the way it became
clear that that we could provide information to those who are interviewing
and hiring entry-level candidates, and to entry-level measurement
applicants (i.e., upcoming and recent graduates). The term “entry-level”
is used to connote starting or base-level positions which, depending
on the hiring organization, may be doctoral, masters, or bachelors
prepared candidates.
Methods
An
invitation to participate in our survey was sent to six active internet
measurement-related listserves (MNET, NCME, AERA-D, SEMNET, IRT,
and the ITC distribution list). We invited responses from anyone
who works for an organization, institution, or company that hires
measurement specialists, psychometricians, or quantitative methodologists.
We purposefully broadened the scope of respondents beyond
simply those managers who make the final offer or members of the
human resources departments because we were interested in soliciting
input from individuals involved at any level of the process, including
screening applicant files, interviewing, making recommendations,
or eventually supervising the successful applicant’s work. Although
there are limitations to an internet-based survey, given that many
individuals in the field of measurement and quantitative methods
(by necessity) are active computer and internet users, we chose
an electronic internet survey rather than a postal survey.
Results
Survey
Respondents.
There were 160 respondents, 46% of which were employed at
a university or college, 22% at a testing company or licensure board,
11% at a consulting or research firm, 9% at a school board or district,
8% at a government department or ministry of education, and
4% at a hospital or health-care research setting. Seventy-two percent
of respondents indicated that they were in the United States, 13%
in Canada, 3% each in Australia and the United Kingdom, and the
remainder were single respondents from Bahrain, Finland, Germany,
Iceland, Iran, Jordan, Mexico, Netherlands, New Zealand, Norway,
South Africa, and Switzerland.
Clearly, our findings will reflect North American perspectives;
however, we believe that with regard to compentencies and characteristics
of entry-level applicants, there is a great deal in common internationally.
Respondents
were asked to indicate the kind of entry-level position (e.g., job
title) being considered. Overall, 23% indicated that they were considering
an assistant professor position (i.e., an entry-level professorial
position) and the remainder were a mix of titles such as assessment
officers, measurement specialists/statisticians, quantitative methodologists,
research scientists, data analysts/consultants, researchers and
psychometricians. An unexpected finding was that while 74 respondents
indicated that they were at a university or college, less than half
(i.e., 36) of these individuals were describing a professorial position.
The remainder of these university or college respondents were describing
some sort of testing and assessment specialists or coordinators,
research associates, and evaluators who would be working at the
university or college but were not professorial positions.
When
asked the minimum academic qualification that would be considered
for the entry-level position, 53% indicated a doctorate (Ph.D, Ed.D.,
or Psy.D.), 38% masters (M.A., M.Ed., M.Sc.), and 9% bachelors degrees.
Clearly, although over half of the respondents required their applicants
to have a doctorate, a substantial portion of the market still has
a place for individuals who have less than a doctorate.
Competencies
and Characteristics.
Table 1 summarizes the responses to the question: Please
rate the importance of each of the following variables or characteristics
in the hiring process by clicking on the down-arrow and selecting
one of the following responses: "not at all important"
(NAI), "somewhat important" (SI), or "vitally important"
(VI). The table
is divided into three sections: academic (professorial) positions,
doctoral prepared nonacademic positions, and masters prepared nonacademic
positions. Because there were only 13 respondents, we decided not
to include the results for the Bachelors-prepared applicants.
Table
1. Percentage Of Respondents For The Importance Ratings.
Applicants
for an entry-level professorial position (n=34)
|
Applicant’s
…
|
NAI
|
SI
|
VI
|
number
of publications
|
0
|
59
|
41
|
journal
quality of publications
|
0
|
53
|
47
|
number
of conference presentations
|
9
|
76
|
15
|
Grades
in graduate school
|
24
|
50
|
26
|
mathematical
and statistical knowledge of measurement models
|
0
|
32
|
68
|
the
university from which the applicant has or will graduate
|
15
|
65
|
20
|
academic/research
supervisor’s reputation in the discipline
|
12
|
62
|
26
|
written
communication skills
|
3
|
21
|
76
|
oral
communication skills
|
0
|
18
|
82
|
consulting
experience
|
38
|
56
|
6
|
applicant’s
personal characteristics (e.g., team-player, easy to get along
with)
|
6
|
41
|
53
|
Doctoral
Prepared Applicants for non-Academic Positions (n=50)
|
Applicant’s
…
|
NAI
|
SI
|
VI
|
number
of publications
|
22
|
70
|
8
|
journal
quality of publications
|
24
|
60
|
16
|
number
of conference presentations
|
20
|
74
|
6
|
grades
in graduate school
|
28
|
54
|
18
|
mathematical
and statistical knowledge of measurement models
|
0
|
12
|
88
|
the
university from which the applicant has or will graduate
|
12
|
84
|
4
|
academic/research
supervisor’s reputation in the discipline
|
22
|
66
|
12
|
written
communication skills
|
0
|
24
|
76
|
oral
communication skills
|
2
|
18
|
80
|
consulting
experience
|
34
|
52
|
14
|
applicant’s
personal characteristics (e.g., team-player, easy to get along
with)
|
4
|
35
|
64
|
Masters
Prepared Applicants for non-Academic Positions (n=59)
|
Applicant’s
…
|
NAI
|
SI
|
VI
|
number
of publications
|
58
|
37
|
5
|
journal
quality of publications
|
66
|
32
|
2
|
number
of conference presentations
|
30
|
68
|
2
|
grades
in graduate school
|
26
|
54
|
20
|
mathematical
and statistical knowledge of measurement models
|
3
|
20
|
77
|
the
university from which the applicant has or will graduate
|
44
|
48
|
8
|
academic/research
supervisor’s reputation in the discipline
|
46
|
44
|
10
|
written
communication skills
|
2
|
17
|
81
|
oral
communication skills
|
2
|
23
|
75
|
consulting
experience
|
34
|
53
|
13
|
applicant’s
personal characteristics (e.g., team-player, easy to get along
with)
|
2
|
22
|
76
|
A
longstanding question in curricular planning for measurement and
quantitative methods programs is the balance between measurement
training and knowledge of a domain outside of measurement and quantitative
methods. To address this matter we asked the question: Does your
organization look for entry-level applicants who have demonstrated
expertise in another domain besides measurement and quantitative
methods? And if so, what? For those individuals looking
for a professorial colleague, 76% responded yes whereas for the
non-professorial positions, 67% and 75% for the doctoral and master-prepared
respectively responded yes. The list of other domains is quite extensive
covering nearly all areas of educational, social, behavioral, and
health sciences.
Another
longstanding question in curricular and hiring discussions asks:
At a minimum, in which topics must an applicant be knowledgeable?
We addressed this question by asking that the respondent
choose no more than five topics from a list of 17 topics such as
item response theory (IRT), standard setting methods, structural
equation modeling (SEM), equating, and multi-level models / hierarchical
linear models. Knowing that our list was certainly not going to
be complete, we allowed respondents to add topics of their choice.
Our reasoning for restricting the choices to five topics
is that we wanted to avoid respondents stating that all of the topics
were important and simulate the realistic trade-off situation wherein
entry-level applicants do not know all topics equally well.
We
transcribed and tallied all of the responses provided, and then
selected the top five most commonly listed topics for the professorial
postions and non-professorial doctoral and masters prepared. Two
topics were listed in the top five for all three types of positions:
(i) item response theory and (ii) survey sampling theory & design.
Individuals interested in academic positions are also expected to
know about structural equation modeling, quasi-experimental design,
and classroom assessment.
Individuals who have doctorates and are interested in a non-professorial
positions are also expected to know about equating, differential
item functioning, and (a tie between) structural equation modeling
and the analysis of categorical data. Masters prepared individuals
are also expected to know about categorical data analysis, criterion-referenced
approaches, and program evaluation. Clearly then, although we agree
that entry-level applicants are expected to know a whole range of
topics in measurement and data analysis, item response theory, and
survey sampling theory & design are a safe bet for topics that
will be expected.
1Thank
you to Lynda McDicken, Graduate Secretary, for this information.
2This
survey was conducted and the project was completed in November
December 2000.
top
|
|