UBC Faculty of Education
Department of Educational and Counselling Pscyhology, and Special Education


Welcome
Graduate Programs Diploma Programs
Courses
Apply Online
Research
Faculty and Staff
News and Events
Graduate Resources
Admission
Frequent Questions
Links
Contact Us





Programs


Measurement, Evaluation, and Research Methodology Program

Program Description and Plan

 


Table of Contents

Philosophy and Mandate

Vision Statements

The Graduate Program in MERM

        General Description

        Typical Patterns of Graduation

        Program Planning

Priorities for Planning.

Research Interests of Core MERM Faculty.

Research Interests for Individuals Who Periodically Teach in the MERM Program...

APPENDIX A:  Degree Programs (M.Ed., M.A, and Ph.D.).

APPENDIX B: A Survey of Competencies and Characteristics of Entry-Level Measurement Specialists and/or Quantitative Methodologists.

 Last Updated: December 20, 2005.

Measurement, Evaluation, and Research Methodology Program

The University of British Columbia

 Philosophy and Mandate

The Measurement, Evaluation, and Research Methodology (MERM) program recognizes that measurement, evaluation and research methodology is an evolving field that is trans-disciplinary by nature and is at the core of many of the research activities in the Department of Educational and Counseling Psychology, and Special Education, the Faculty of Education, and in many of the human and health sciences in the University.    As evidence for this trans-disciplinary nature, we note that our classes are often in high demand not only by the various disciplines in the Faculty of Education, including Human Kinetics, but also by students in Commerce, Health Studies, Nursing and Psychology. Furthermore, the faculty members in MERM often advise colleagues, sit on graduate student committees across the university, and provide advice to graduate students for whom we are not formal committee members.

 In recognition of this trans-disciplinary feature MERM plays two important roles within the Department, Faculty, and University.

     1. We develop and share our collective knowledge in:

  • assessment, evaluation, testing, and other aspects of educational and psychological measurement. We are involved in the construction and use of standardized tests; new forms of assessment, including the development and testing of measurement methodology,
  • program evaluation, including methods for quasi-experiments, hierarchical or multi-level models, and the analysis of growth and change,
  • qualitative methods, and
  • data analysis and statistical methods for research problems in psychology and education,


    2. We prepare graduate students as methodological and measurement specialists. Emphasis in the MERM area is placed on advanced research as applied to educational, psychological, health, and social contexts.

An essential feature of the above roles is the generation and sharing of knowledge and information through our publications, seminars, workshops, and membership on technical advisory panels.

We strive to promote in our research, student supervision, and teaching the highest standards of measurement and research methodology practice in our discipline. This has meant that we have two shared goals:    (a) providing courses for researchers, graduate students, and practitioners outside of MERM, and (b) the preparation of graduate students as methodological and measurement specialists.

 Integrating research, practice, and teaching to promote the highest standards of measurement, evaluation, and research methodology

 

 Vision Statements

  • We will work toward expanding the enrollments in our M.A., M.Ed., and Ph.D. MERM program.
  • We will diversify and monitor our course offerings to meet proactively the increasing complexity and diversity of data analysis and measurement methods used in educational and psychological research.
  • We will continue to be a resource to our colleagues and graduate students outside of the MERM area, and to various governmental and non-governmental organizations in Canada and around the world.

The Graduate Program in MERM

             General Description

The Measurement, Evaluation, and Research Methodology program at the University of British Columbia is comprised of five core faculty members (Ercikan, Hubley, Kishor, Mathison, and Zumbo) and five associate faculty members . The term "core faculty" is being used in the same way that it is used in APA and CPA accreditation processes: core faculty are those individuals who have active roles in the development and governance of the program and assume primary responsibility for the training of the program’s students.

The MERM program is a small program compared to other larger Programs (e.g., Counseling Psychology) in our Department. The MERM Program is comprised of faculty whose primary focus is MERM and others for whom MERM is a secondary or tertiary areas of scholarly interest.   The MERM program offers excellent opportunities for students with the aptitude and interest for this field. Emphasis in the MERM area is placed on advanced research in a variety of contexts including educational, psychological, health, and social services contexts. Students concentrating in this area will generally fit into one of three categories.   

 1. Students who have an applied interest in measurement, program evaluation, or data analysis. While they often have some preparation in measurement and data analysis in their undergraduate studies, this is not always the case. These students are more oriented toward the use of measurement, program evaluation, or data analysis techniques in substantive research in the areas of test development, personality assessment, clinical measurement, computer applications, program planning or evaluation, and so on.

2. Students possessing strong theoretical interests in technical problems in test theory, item response theory, assessment, statistics, factor analysis, multi-level modeling, and the like. Also, some of these students come to the Program with some statistical and/or mathematical background -- often obtained while studying in another social science such as Psychology or Sociology although sometimes students arrive with degrees in statistics or mathematics.

3. Some students will find it compatible with their career goals to give equal attention to both these major aspects (i.e., applied and theoretical) of this program.


Appendix A provides a detailed description of our M.A., M.Ed., and Ph.D. programs.

The M.A. and Ph.D. students in this field will be expected to become involved in research activities culminating in the execution of their own Masters thesis or Doctoral dissertation research. The M.Ed. degree is primarily a course-based Masters degree.

MERM students interested in the technical side of psychometrics may also take related courses outside the department, including Statistics, Computer Science, Psychology, and Epidemiology.

Upon completion of our program our graduates often take positions as: university faculty; test developers; school district and provincial government testing and research directors; professional evaluators; educational statisticians; research consultants; testing specialists in business, industry, education, community programs; and licensure, certification, and credentialing professionals.

Recent graduates from our Program have taken positions at the Rutgers University (U.S.A.), Department of Pharmaceutical Sciences UBC, and testing an research organizations such as CTB/McGraw-Hill as Psychometricians and Research Scientists.

 

Typical Patterns of Graduation

 In order to get a sense of the graduation patterns of MERM-type students, the report by Sanderson and Dugoni (1999) Summary report 1997: Doctorate recipients from United States universities (Chicago: National Opinion Research Center), was consulted to obtain the number of doctoral degrees awarded in psychology and those granted in educational and school psychology in departments, colleges, and faculties of education.        Although this is data from the United States, it will be useful in giving us a sense of the potential candidate pool that MERM may expect and the proportion of graduates we can expect relative to the other programs in our department. Only the sub-fields that are currently in our department are listed in Table 1.   

Table 1. Doctoral Degrees Awarded in 1996 by Subfield

Subfield

Number

Percent

Counseling

464

34.9

Educational PhD/EdD

401

30.1

School PhD/EdD

196

14.7

Developmental & Child

188

14.1

Family/Marriage Counseling

52

3.9

Psychometrics and Quantitative

30

2.3

Total

1331

100

Table 1 suggests that if the US pattern fits in our context we can reasonably expect that approximately 2 out of every 100 students our department graduates would be MERM graduates (of course, this implies that our enrollments would be approximately 2 percent of the graduate students in our department).  As of January 2001, there were 398 masters and doctoral students in our department, of which 12 (3%) are MERM. However, the department has 78 PhD students of which 5 (6.4%) are MERM students [1].  Clearly, although student recruitment is a priority for us, our enrollments are in line with national and international trends.

What is expected of our MERM Graduate Students

Appendix B contains a report of a survey conducted in the Fall of 2000 that was planned to determine the required competencies and characteristics of entry-level measurement specialists and/or quantitative methodologists. With regards to academic program planning, the results speak to several recommendations:

  • With regard to a longstanding question of striking a balance between measurement and evaluation training (i.e., specialist training) and knowledge of a domain outside of measurement, evaluation, and research methodology, the results highlight that students must be prepared to demonstrate expertise in another domain besides measurement and quantitative methods. What is unclear, however, is the level of expertise expected. We believe that our graduate students meet this expectation because (a) our graduate programs require our students to complete coursework outside and (b) our graduate students typically come from undergraduate degrees in fields outside of MERM, per se (e.g., teacher education, psychology, history, sociology, political science).
  • At the same time that potential employers would like expertise outside of MERM the question of the importance of competencies and characteristics (Table 1 of Appendix B) highlighted that there is an expectation that mathematical and statistical knowledge of the measurement models is important.
  • With regard to written and oral communication skills, most, if not all, of our MERM courses (except, perhaps the general service data analysis course) require our students to write and present papers/projects.
  • Although over half of the respondents to the survey required their applicants to have a doctorate, a substantial portion of the market still has a place for individuals who have less than a doctorate.    This highlights the importance of our M.Ed. and M.A. programs in MERM.

 

Program Planning

MERM Courses

In our discussion of MERM courses it is important to use the terminology of “introductory”, “intermediate”, and “advanced” courses. In this light, all of our courses, irrespective of the level of the course, are open to students across the Department (and actually across the Faculty, and in some cases, the University). What this means is that we are no longer using the language of  "specialized" courses and also "service" versus "non-service" courses. Instead, the language of "introductory, intermediate, and advanced" courses makes it clearer that all of our courses are required by several of the ECPS Programs.

Of course, all of the Programs in ECPS require that their students take some combination of introductory to advanced courses. For example, all SCPS, HLDI, SPED, and CNPS doctoral students have methodological requirements and some of those requirements are met through our intermediate and advanced courses.

Given the above description, every academic year we try and strike a balance so that we have at least one intermediate or advanced course in each of the winter terms.

What follows is an example of course offerings during an academic year:

Introductory/basic (EPSE 481, 482, 528, 592, 595):     15 sections

Intermediate  (EPSE 596)    1 section

Advanced (EPSE 597, 681)  2 sections

Of course, whether a course is considered introductory, intermediate, or advanced is somewhat arbitrary but you will get a sense of our course offerings from the list above.

Priorities for Planning

Annually we have courses on advanced topics such as IRT, factor analysis and structural equation modeling, test construction, equating, DIF, multi-level modeling, and measurement and analysis of change and growth.

From the survey and discussion among MERM faculty we set the following priorties:

  • Priority should be given to offer IRT and advanced measurement models on a regular cycle (at least every two years) 
  • A course should be offered regularly on survey sampling theory and design for educational and psychological research 
  • Likewise a course on categorical data analysis would be useful. 

A strategic plan will be prepared for recruiting MERM students. The following tasks will be a part of this plan:

  • The MERM webpage will be further developed
  • A brochure and poster will be prepared that will be sent to various departments of psychology and sociology as well as faculties of Education across Canada, U.S.A, and key recruiting sites internationally.
  • Formalizing a minor in MERM 

Research Interests of Core MERM Faculty

Core Faculty are those individuals who have active roles in the development and governance of the program and assume primary responsibility for the training of the program’s students. The following individuals regularly teach MERM courses and supervise MERM Graduate Students.

Kadriye Ercikan, Ph.D. Associate Professor (MERM Program) Research design and methodology; measurement and psychometrics; item response theory; cross-cultural issues in assessment, adaptation and translation issues in assessment, psychometric issues in large-scale assessments; international assessments.
http://www.ecps.educ.ubc.ca/faculty/k_erikan.htm

Anita Hubley, Ph.D. Associate Professor (MERM Program) Applied measurement and test development (social, cognitive-neuropsychological measures); adult assessment (personality, intelligence, neuropsychology); lifespan development with a focus on adulthood and aging (e.g., age identity, body image, depression, memory); quantitative and qualitative research methods and analyses.
http://www.educ.ubc.ca/faculty/hubley/index.html

Nand Kishor, Ph.D. Associate Professor (MERM Program) Educational and psychological measurement; scaling; item response modeling; equating & standard setting; computer adaptive and dynamic assessment; program evaluation; performance evaluation; multivariate statistical methods; cross-cultural research methodology; meta-analysis methods; cognition: judgement and decision-making.
http://www.ecps.educ.ubc.ca/faculty/n_kishor.htm

Sandra Mathison, Ph.D. Professor (MERM Program). Educational evaluation, Democratic and participatory evaluation, Sociology of assessment, Impact of mandated standardized testing on students, teachers and schools, Qualitative research methods, Uses of alternative representation in research and evaluation. http://www.ecps.educ.ubc.ca/faculty/s_mathison.htm

Bruno D. Zumbo, Ph.D. Professor (MERM Program). My interests focus in measurement and statistical science. Most recently I have worked on latent variable and observed score psychometric models including complex tasks, complex survey, measurement of change, and differential item functioning. My methodological developments are applied and tested, by me and others, in the domains of educational and psychological research, language testing, health studies, and in quality of life and subjective well-being research.    Recent papers include studies of Bayesian methods in IRT, nonparametric statistics, DIF detection methods, and reliability theory. I am also formally affiliated with the Department of Statistics and the Department of Psychology at UBC.
/zumbo/zumbo.html
 

Research Interests for Individuals Who Periodically Teach in the MERM Program 

Although MERM is not their primary area of research and teaching, the following individuals teach MERM courses.

Marshall Arlin, Ph.D. Professor (MERM and HLDI Programs) Critical thinking, problem solving, learning strategies, creativity. Research design and analysis of variance.

Deborah Butler, Ph.D. Associate Professor (SPED, HLDI, and MERM Programs) Learning disabilities in adolescence and adulthood; instructional strategies for promoting self-regulated and strategic learning; cognitive learning theories; research design and methodology; qualitative and quantitative methods and analyses.

Joe Lucyshyn, Ph.D. Assistant Professor (SPED and MERM Programs) Developmental disabilities; positive behavioral support; assessment and intervention with families of children and youth with behavior problems; home-school collaboration; instructional technology; small N research design; qualitative research methods.

William McKee, Ph.D. Assistant Professor (SCPS, SPED, and MERM Programs) Professional issues in school psychology and special education including service delivery, ethics, legislation and training; consultation processes; intervention for behavioural disorders; intervention implementation and treatment integrity; applied behaviour analysis; single case and small N research design. 

Jennifer D. Shapka, Ph.D. Assistant Professor  (HLDI ) Developmental perspectives on gender, motivation and education (technology, science, and math education).  Identifying risk and protective factors impacting high risk youth populations (i.e., teenage mothers and their children).  Examining patterns of development at the population level, using large-scale longitudinal databases (i.e., NLSCY).

APPENDIX B: A Survey of Competencies and Characteristics of Entry-Level Measurement Specialists and/or Quantitative Methodologists

Bruno D. Zumbo, Anita M. Hubley, and Bonnie Davidson, [2]

University of British Columbia

We conducted a survey of potential employers to determine the required competencies and characteristics of entry-level measurement specialists and/or quantitative methodologists. Our main goal was to obtain information for academic program planning but along the way it became clear that that we could provide information to those who are interviewing and hiring entry-level candidates, and to entry-level measurement applicants (i.e., upcoming and recent graduates). The term “entry-level” is used to connote starting or base-level positions which, depending on the hiring organization, may be doctoral, masters, or bachelors prepared candidates.

Methods

An invitation to participate in our survey was sent to six active internet measurement-related listserves (MNET, NCME, AERA-D, SEMNET, IRT, and the ITC distribution list). We invited responses from anyone who works for an organization, institution, or company that hires measurement specialists, psychometricians, or quantitative methodologists.    We purposefully broadened the scope of respondents beyond simply those managers who make the final offer or members of the human resources departments because we were interested in soliciting input from individuals involved at any level of the process, including screening applicant files, interviewing, making recommendations, or eventually supervising the successful applicant’s work. Although there are limitations to an internet-based survey, given that many individuals in the field of measurement and quantitative methods (by necessity) are active computer and internet users, we chose an electronic internet survey rather than a postal survey.

Results

Survey Respondents.    There were 160 respondents, 46% of which were employed at a university or college, 22% at a testing company or licensure board, 11% at a consulting or research firm, 9% at a school board or district,    8% at a government department or ministry of education, and 4% at a hospital or health-care research setting. Seventy-two percent of respondents indicated that they were in the United States, 13% in Canada, 3% each in Australia and the United Kingdom, and the remainder were single respondents from Bahrain, Finland, Germany, Iceland, Iran, Jordan, Mexico, Netherlands, New Zealand, Norway, South Africa, and Switzerland.   Clearly, our findings will reflect North American perspectives; however, we believe that with regard to compentencies and characteristics of entry-level applicants, there is a great deal in common internationally.

Respondents were asked to indicate the kind of entry-level position (e.g., job title) being considered. Overall, 23% indicated that they were considering an assistant professor position (i.e., an entry-level professorial position) and the remainder were a mix of titles such as assessment officers, measurement specialists/statisticians, quantitative methodologists, research scientists, data analysts/consultants, researchers and psychometricians. An unexpected finding was that while 74 respondents indicated that they were at a university or college, less than half (i.e., 36) of these individuals were describing a professorial position. The remainder of these university or college respondents were describing some sort of testing and assessment specialists or coordinators, research associates, and evaluators who would be working at the university or college but were not professorial positions.

When asked the minimum academic qualification that would be considered for the entry-level position, 53% indicated a doctorate (Ph.D, Ed.D., or Psy.D.), 38% masters (M.A., M.Ed., M.Sc.), and 9% bachelors degrees. Clearly, although over half of the respondents required their applicants to have a doctorate, a substantial portion of the market still has a place for individuals who have less than a doctorate.

Competencies and Characteristics.    Table 1 summarizes the responses to the question: Please rate the importance of each of the following variables or characteristics in the hiring process by clicking on the down-arrow and selecting one of the following responses: "not at all important" (NAI), "somewhat important" (SI), or "vitally important" (VI).  The table is divided into three sections: academic (professorial) positions, doctoral prepared nonacademic positions, and masters prepared nonacademic positions. Because there were only 13 respondents, we decided not to include the results for the Bachelors-prepared applicants.

Table 1. Percentage Of Respondents For The Importance Ratings.

Applicants for an entry-level professorial position (n=34)

Applicant’s …

NAI

SI

VI

number of publications

0

59

41

journal quality of publications

0

53

47

number of conference presentations

9

76

15

Grades in graduate school

24

50

26

mathematical and statistical knowledge of measurement models

0

32

68

the university from which the applicant has or will graduate

15

65

20

academic/research supervisor’s reputation in the discipline

12

62

26

written communication skills

3

21

76

oral communication skills

0

18

82

consulting experience

38

56

6

applicant’s personal characteristics (e.g., team-player, easy to get along with)

6

41

53

Doctoral Prepared Applicants for non-Academic Positions (n=50)

Applicant’s …

NAI

SI

VI

number of publications

22

70

8

journal quality of publications

24

60

16

number of conference presentations

20

74

6

grades in graduate school

28

54

18

mathematical and statistical knowledge of measurement models

0

12

88

the university from which the applicant has or will graduate

12

84

4

academic/research supervisor’s reputation in the discipline

22

66

12

written communication skills

0

24

76

oral communication skills

2

18

80

consulting experience

34

52

14

applicant’s personal characteristics (e.g., team-player, easy to get along with)

4

35

64

Masters Prepared Applicants for non-Academic Positions (n=59)

Applicant’s …

NAI

SI

VI

number of publications

58

37

5

journal quality of publications

66

32

2

number of conference presentations

30

68

2

grades in graduate school

26

54

20

mathematical and statistical knowledge of measurement models

3

20

77

the university from which the applicant has or will graduate

44

48

8

academic/research supervisor’s reputation in the discipline

46

44

10

written communication skills

2

17

81

oral communication skills

2

23

75

consulting experience

34

53

13

applicant’s personal characteristics (e.g., team-player, easy to get along with)

2

22

76

A longstanding question in curricular planning for measurement and quantitative methods programs is the balance between measurement training and knowledge of a domain outside of measurement and quantitative methods. To address this matter we asked the question: Does your organization look for entry-level applicants who have demonstrated expertise in another domain besides measurement and quantitative methods? And if so, what? For those individuals looking for a professorial colleague, 76% responded yes whereas for the non-professorial positions, 67% and 75% for the doctoral and master-prepared respectively responded yes. The list of other domains is quite extensive covering nearly all areas of educational, social, behavioral, and health sciences.

Another longstanding question in curricular and hiring discussions asks: At a minimum, in which topics must an applicant be knowledgeable?   We addressed this question by asking that the respondent choose no more than five topics from a list of 17 topics such as item response theory (IRT), standard setting methods, structural equation modeling (SEM), equating, and multi-level models / hierarchical linear models. Knowing that our list was certainly not going to be complete, we allowed respondents to add topics of their choice.    Our reasoning for restricting the choices to five topics is that we wanted to avoid respondents stating that all of the topics were important and simulate the realistic trade-off situation wherein entry-level applicants do not know all topics equally well.

We transcribed and tallied all of the responses provided, and then selected the top five most commonly listed topics for the professorial postions and non-professorial doctoral and masters prepared. Two topics were listed in the top five for all three types of positions: (i) item response theory and (ii) survey sampling theory & design. Individuals interested in academic positions are also expected to know about structural equation modeling, quasi-experimental design, and classroom assessment.    Individuals who have doctorates and are interested in a non-professorial positions are also expected to know about equating, differential item functioning, and (a tie between) structural equation modeling and the analysis of categorical data. Masters prepared individuals are also expected to know about categorical data analysis, criterion-referenced approaches, and program evaluation. Clearly then, although we agree that entry-level applicants are expected to know a whole range of topics in measurement and data analysis, item response theory, and survey sampling theory & design are a safe bet for topics that will be expected. 

 

1Thank you to Lynda McDicken, Graduate Secretary, for this information.

2This survey was conducted and the project was completed in November ­ December 2000.


top



ECPS - Department Home Page | Site Map | Search

Faculty of Education Home Page | UBC Home Page