UMassAmherst: Research and Evaluation Methods
Research and Evaluation Methods

Courses Fall 2004 through Spring 2008

Spring, 2005
Education 555 Wells
Education 656 Keller
Education 661 Hambleton
Education 731 Wells
Education 763 (794E) Hambleton
Education 772 Keller
Education 794O Sireci

Fall, 2005
Education 509 (591J) Sireci
Education 555 Keller, Wells
Education 632 Hambleton
Education 656 Wells
Education 661 TBA
Education 735 Hambleton
Education 771 Keller

Spring, 2006
Education 555 Wells
Education 656 TBA
Education 661 Sireci
Education 756 (793B) Sireci
Education 772 Keller
Education 803(797W) Keller and Wells

Fall, 2006
Education 555 TBA
Education 632 Hambleton
Education 656 TBA
Education 661 TBA
Education 736 Hambleton
Education 751 (795M) Sireci
Education 771 Keller

Spring, 2007
Education 555 TBA
Education 656 TBA
Education 731 Wells
Education 763 (794E) Hambleton
Education 772 Keller
Education 794O Sireci

Fall, 2007
Education 509 (591J) Sireci
Education 555 TBA
Education 632 Hambleton
Education 656 TBA
Education 735 Hambleton
Education 771 Keller

Spring, 2008
Education 555 TBA
Education 656 TBA
Education 661 TBA
Education 731 Wells
Education 756 (793B) Hambleton
Education 772 Keller
Education 801 Hambleton
Education 803 (797W) Keller and Wells
TBA means that the course will be taught, with the instructor to be named later. In the spring semester of 2006, Education 756 (793B) will focus on Educational Assessment Policy, and Professor Stephen Sireci, will take the lead with the course.

Research and Evaluation Methods Program Courses (Nov. 21, 2004)

Educ 553 Modern Assessment Methods and Practices (Hambleton, Sireci)

The course will begin with a consideration of technical advances in the development, evaluation, and uses of criterion-referenced assessment. The focus of course content will then shift to newer technical developments associated with performance assessment, standard-setting, score reporting, and current issues in assessment such as (1) the role of computers in assessment, (2) adaptation of tests into multiple languages and cultures, (3) National Assessment of Educational Progress, and (4) the implications of No Child Left Behind Legislation for school, district, and state testing programs. Prerequisites: None.

Educ 555 Introduction to Statistics and Computer Analysis I (Keller, Sireci, Wells)

The purpose of this course is to give students in the social sciences and, in particular, education, skills in statistical reasoning so that they will be a) critical readers of research literature in their fields and b) in a position to design research studies and analyze data on their own. More specifically, the purposes of this course are to provide students with (i) a conceptual understanding of the basic statistical procedures used in educational and social science research and (ii) the computational skills necessary to carry out the procedures. Prerequisites: None

Educ 591J Fundamentals of Test Construction (Sireci)

This course provides students with the skills and information needed to build and evaluate educational tests. In addition to learning the fundamentals of reliability and validity, students will learn about the advantages and disadvantages of different assessment formats such as selected response items, performance assessments, and computer-based testing. They will also learn how to operationally define educational objectives and testing purposes, develop a variety of item formats including multiple-choice and constructed response items, develop answer keys and scoring rubrics for different item formats, and evaluate tests using statistical and qualitative methods. The common theme unifying these knowledge and skill areas is the promotion of equity and fairness in testing. In addition, the course stresses the role of educational testing in improving student learning. Prerequisites: None

Educ 632 Principles of Educational and Psychological Testing (Hambleton)

The course has been designed to provide graduate education students with a solid foundation of educational and psychological testing and measurement skills. By the end of the semester, students should be able to: (a) define basic testing, measurement, and statistical terms; (b) identify situations where educational and psychological tests can be helpful in making decisions about individuals and/or programs; (c) describe and use reliability and validity concepts and methods; (d) interpret and use scores reported on common educational and psychological scales; (e) be familiar with various cultural and environmental factors that influence test performance; (f) distinguish norm-referenced testing from criterion-referenced testing and describe the ways in which each type of test is developed, evaluated, and used; (g) identify popular achievement, aptitude, and personality tests; (h) locate commercially available instruments to meet particular assessment needs and to evaluate the technical quality of the instruments; (i) identify current testing issues and developments (e.g., computers and testing, misuse of state-wide testing programs, coaching to increase test performance, performance testing, testing competencies for educators); and (j) be familiar with basic statistical methods for analyzing and interpreting test scores. Prerequisites: None

Educ 656 Introduction to Statistics & Computer Analysis II ( Keller, Sireci, Wells)

This course provides students with the knowledge necessary for understanding and critiquing educational research, and the skills necessary for conducting statistical analyses on educational research data. Successful completion of this course will enable students to: 1) formulate designs for collecting data, and 2) analyze data of different measurement levels using univariate statistical models such as one-way analysis of variance (ANOVA), factorial ANOVA, repeated measures ANOVA, and multiple regression. In addition, related univariate and multivariate procedures will be discussed. Students will also gain proficiency in using the SPSS statistical software package to perform data analyses. Prerequisite: Educ 555, or equivalent

Educ 661 Educational Research Methods I (Hambleton, Sireci)

The purpose of this course is to provide graduate students with the skills they will need to carry-out empirical research studies in the field of education. Topics covered in the course include: (1) purposes and types of educational research, (2) steps in conducting research and preparing a research proposal, (3) selection of research questions for investigation, (4) literature reviews, (5) basic statistical methods for quantitative data analysis, (6) development and validation of instrumentation, (7) principles of sampling, (8) research designs, (9) data collection techniques, (10) introduction to SPSS (a package of computer programs for data analysis), and (11) interpreting results, drawing conclusions, and reporting research results. Survey, correlational, and experimental and quasi-experimental research methods and practices will be emphasized in the course. Prerequisites: None

Educ 731 Structural Equation Modeling (Wells)

The purpose of this course is to provide students with the skills necessary to apply structural equation modeling techniques in the context of the social sciences. The theory and mathematics behind structural equation modeling will be covered including model identification, evaluating model fit, parameter estimation, confirmatory factor analysis, and hybrid models. Prerequisites: Educ 771, 772 (or equivalent), or permission by the instructor

Educ 735 Classical Test Models and Practices (Hambleton)

The course is intended to provide students with a solid foundation in classical psychometric models and methods, and knowledge of important results such as the attenuation formulas, range restriction corrections, and estimates of measurement error. The classical test model and derivations from it, reliability and validity issues and methods, and test development, are emphasized in the course. Consideration is given, too, to important topics such as scale construction, norming, classical test score equating, and emerging issues in the testing and assessment field such as new item formats and computer-based assessment. Prerequisite: Educ 555 (preferred, but not required)

Educ 736 Fundamentals of Item Response Theory (Hambleton)

Over the past 30 years many agencies who construct educational assessments, aptitude, and personality tests, have switched from the use of classical test theory (CTT) and practices to item response theory (IRT) and practices. The reasons for the switch include the increased flexibility of IRT over CTT, useful properties of IRT item and ability parameters, and evidence that IRT provides a better and more flexible measurement solution to some technical problems that arise in assessment and testing. In this course, basic IRT concepts, models, features, parameter estimation, model fit, and computer software will be introduced. In addition, specific applications of IRT to test score equating, computer-administered tests, the identification of biased items, and test development will be introduced. Prerequisites: Educ 555 (preferred, but not required)

Educ 771 Applied Multivariate Analysis I (Keller)

This course is designed to provide students in the social and behavioral sciences advanced techniques for the analysis of univariate and multivariate data. Rather than providing the students with a compendium of statistical techniques, the course is designed to provide the students with a unified treatment of the procedures for analyzing data. To this end, a rigorous treatment of mathematics and statistical theory underlying the techniques covered in the course is provided. Understanding multivariate statistics requires knowledge of linear algebra. Since most students in the social and behavioral sciences do not have a deep understanding of linear algebra or a background in it, the first part of the course is designed to provide students skills in linear algebra. The topics covered are: basic operations with vectors and matrices, scalar functions of matrices such as trace and determinants, linear dependence and rank of a matrix, inverse of a matrix, solution of linear equations, and characteristic roots and vectors of a matrix. The second part of the course deals with statistical foundations. Topics covered include: joint, conditional, and marginal distributions; the multivariate normal distribution; and, include: joint, conditional, and marginal distributions; the multivariate normal distribution; and, algebra of expectations. These concepts are applied to the linear model and include discussions of the general linear model, the general linear hypothesis, hypothesis testing procedures, the problem of multiple comparison, and applications to regression and experimental design models. The multivariate linear model is introduced and its implications are discussed. The course concludes with a discussion of multivariate analysis of variance and discriminant analysis. The students are taught to write their own programs for the analysis of multivariate data using matrix operation packages. In addition, they are expected to reconcile results obtained with their hand-crafted programs with that obtained using such package programs as SPSS. Prerequisites: Educ 555, 656

Educ 772 Applied Multivariate Analysis II (Keller)

This course is a continuation of Education 771 and includes further topics on multivariate analysis. The course begins with a review of MANOVA and continues with more detailed analysis of repeated measures designs. The general multivariate model is introduced and applications to growth curve analysis and classification analysis are covered. The second part of the course deals with the structure of multivariate data. Topics covered include principal components analysis, factor analysis with a discussion of the problem of rotation, the interpretation of factor loading, and methods of factor extraction procedures.. Prerequisite: Education 771

Educ 793B Advanced Measurement Seminar (Hambleton, Sireci)

The purpose of this advanced level measurement seminar is to provide an opportunity for discussion of emerging technical topics in the educational assessment field. Topics that have been addressed in recent seminars include scale construction and score reporting, adapting tests from one language and culture to another, setting standards on educational tests, psychometric foundations for performance assessment (e.g., item construction and validity), estimating item statistics without data, detection of overexposed test items in CAT environments, introduction to structural equation modeling, and emerging laws and implications for educational assessment. Graduate student research and thesis topics are also addressed. Prerequisites: Educ 555, 735

Educ 794E Advances in Item Response Theory (Hambleton)

Educational assessments are changing and new psychometric models are needed for scoring, test development, and evaluation. In this course, students will be introduced to IRT models for analyzing polytomous and multidimensional data which are becoming popular in school, state, and national assessment programs. These models include the nominal response, partial credit, generalized partial credit, graded response, and the rating scale models, and the normal ogive and logistic multidimensional models. Applications of these new models to test scoring, the detection of bias, equating, computer-based testing, test development, and score reporting will be emphasized. Also, students can expect to learn to use new IRT software such as Multilog, Parscale, and NOHARM. Prerequisites: Educ 555, 736

Educ 7940 Advanced Validity Theory and Test Validation (Sireci)

This course presents and discusses the major theories regarding the concept of "test validity" and the major practices involved in test validation. The theory portion of the course covers the most influential theorists in this area (e.g., Cronbach, Kane, Messick, Shepard, etc.). The application portion of the course reviews methods for gathering validity evidence and analyzing validity data. Students will acquire hands-on experience in conducting such studies. By taking this course, students will learn: (a) how approaches to test validation vary according to testing purposes, (b) how to design studies for evaluating the validity of a test for a particular purpose, and (c) how to conduct such studies. The skills taught in this course will enable students to be experts in testing tests. The concepts addressed in the course are advanced and specialized. Knowledge of basic statistical techniques is necessary. Prerequisites: Educ 555, 656; and Educ 591J or 632

Educ 795M Scaling Methods for the Behavioral Sciences (Sireci)

Scaling is fundamental to the process of measurement. The purpose of scaling is to form a metric, or continuum, along which the magnitude of a variable can be measured. Scaling issues in the behavioral sciences are complex due to the myriad of unobservable variables studied such as abilities, attitudes, perceptions, traits, and proficiencies. The purpose of this course is to introduce and explore scaling methods essential for research in the social sciences. In particular, techniques of unidimensional scaling, multidimensional scaling, and classification are covered. Unidimensional scaling topics include scaling surveys, tests, and rating forms. Multidimensional scaling topics focus on uncovering the hidden structure of multivariate data. Classification techniques covered include cluster analysis and partitioning methods. Application of these scaling techniques to the areas of educational, psychological, and sociological research is emphasized. Prerequisites: Educ 555, 632, 656

Educ 797W Psychometric and Statistical Modeling (Keller)

The purpose of this course is to provide advanced students in the areas of psychometrics and statistics with techniques for carrying out research using computer simulations. The topics covered are: Programming with FORTRAN 90 language, data manipulation, simulation of data according to statistical and psychometric models, numerical techniques for matrix operations, solution of non-linear equations, and optimization. Weekly assignments will be given. The students will be expected to carry out a research project simulation techniques and make a presentation. Prerequisites: Students must have taken courses in classical and modern test theory (Educ 735, 736, 794E, 801), psychometric methods (Educ 731; 795M), and multivariate analysis (Educ 771, 772).

Educ 801 Psychometric Methods I (Hambleton)

This course was designed as a follow-up to Education 735. Students will be introduced to a number of advanced topics that are especially important in the assessment field today: item analysis, horizontal and vertical equating, standard-setting, introduction to generalizability theory, criterion-referenced test theory and models, measurement of changes, score scales and norms, and the detection of differentially functioning test items. Prerequisites: Educ 555, 656, 735

Resources

Center for Educational Assessment

Key Links

Contact

Find out more!