Our Research – CEA

Standardized Assessment Training and Support for Massachusetts Adult Education. [4 years]

As of July 2018, REMP has contracted with the Massachusetts Department of Elementary and Secondary Education’s Adult and Community Learning Services unit to provide support and professional development to adult education providers with respect to all of the standardized assessments used in Massachusetts adult education programs. [In addition to the MAPT-CCR, programs have five other assessments that can be used to measure student gains, depending on student goals and content being assessed.] UMass coordinates both face to face and online training for administration and scoring of these assessments, as well as annual recertification for two of the assessments. We serve as the primary point of contact for the adult education field in Massachusetts with respect to assessment-related questions.  In addition, REMP faculty, staff, and students are developing a library of short video resources for adult education instructors and the students themselves.  Some of the videos are focused on general topics in assessment such as tips for taking tests, and understanding measurement error and score changes, while others are intended to be informational and somewhat more test-specific, on topics such as the MAPT-CCR score reports or recertification procedures for certain tests.

For more information about our work to support assessment in adult education in Massachusetts, please check out our resource blog. 

A Complex Test, A Single Cut: Methodological Approaches to Setting and Validating Cut Scores on the Next Generation NCLEX. [1 year]

Over the past several years, REMP faculty and staff have had two projects funded by the National Council of State Boards of Nursing (NCSBN), as that agency embarks on the development of the next-generation NCLEX exam. Most recently, we completed a project to look into the emerging methods for standard setting in the context of complex assessments.  Credentialing tests measure examinee knowledge, skills, and abilities (KSAs) in the context of professional employment and training. While credentialing tests have long focused on content knowledge and asking questions about processes, with the increasing prevalence of technology in many professions and advances in test delivery, more and more credentialing tests are investigating the feasibility of assessing more complex cognitive processes (i.e., clinical judgement and decision making) that job analyses define as necessary for success as an entry-level candidate.  To accurately measure this wide range of skills, test developers are increasing the complexity of tests by including a range of item types, mixed scoring models, and adaptive test administration.  Given the increasing complexity of next-generation credentialing tests, the process for determining the pass-fail cut-score that delineates a qualified candidate from an unqualified candidate should be informed by these kinds of test elements in order to ensure that the recommended passing score reflects the test and testing context appropriately. The goal of this exploratory study was to develop a deeper understanding of (1) the current and emerging practices in standard setting for complex assessments and (2) the nature of current and optimal approaches to gathering validity evidence for standard setting.  We conducted seven in-person or phone interviews with nationally-renown standard setting experts. The analysis of the interviews revealed four general themes, along with several sub-themes. The findings of this study were used to inform recommendations for evaluation, or re-evaluation, of standard setting practices, including the construction of a validity argument related to standard setting for complex tests. 

This research was presented at the 2018 AERA Research-in-Progress gala in New York City in 2018, and an extension of the project has been accepted on the 2018 NCME program in Toronto as part of a symposium on visualizations in standard setting.

Technical Assistance to Support the National Board of Professional Teaching Standards Examinations. [2 years]

Lewis, J., & Sireci, S. G. (2018). Analysis of the National Board for Professional Teaching Standards Candidate Survey. Center for Educational Research Report No. 964. Amherst, MA: Center for Educational Assessment.

Since 2003, the Center for Educational Assessment at UMass Amherst has been the test publisher for the Massachusetts Adult Proficiency Test (MAPT).  This assessment program is commissioned by the Massachusetts Department of Elementary and Secondary Education’s Adult and Community Learning Services unit, and is used to assess the learning gains of adult students enrolled in education programs across the Commonwealth for federal and state accountability purposes, in mathematics and reading. The MAPT-CCR for Mathematics and for Reading are assessments that are directly targeted to the instruction that adult learners in Massachusetts receive, and learners' proficiency is evaluated against national proficiency standards developed by the U.S. Department of Education (DAEL, 2016).  It is a computerized multistage adaptive test, administered on computer via the internet to thousands of students each year. The faculty, staff, and students of the Research in Educational Measurement and Psychometrics program carry out all development and operational maintenance of the MAPT.  In 2018, the fourth-generation MAPT was released (the MAPT-College and Career Readiness, or MAPT-CCR). This iteration of the test was updated to reflect the most current curriculum standards (the College and Career Readiness Standards for Adult Education (CCRSAE; Pimentel, 2013).

Having an operational computerized test housed in the College of Education at UMass has provided students and faculty alike with rich opportunities for both theoretical and applied psychometric research, as well as the chance to contribute to the ongoing operational activities associated with a live testing program.  A listing of research reports involving MAPT data, completed by REMP students and faculty, can be found here (link to the listing from the MAPT-CCR TM).

In our Technical Manual, we have compiled psychometric evidence - drawn from a wide variety of sources and data analysis methods including surveys, studies involving expert judgment, simulations, and other qualitative and quantitative analyses - to make a convincing argument for the high technical quality of the MAPT-CCR as a measure of AE learners' knowledge and skills in mathematics and reading, relative to their progress in meeting educational goals. The data and processes presented here illustrate the ways in which the content of the MAPT-CCR is aligned to the College and Career Readiness Standards for Adult Education (CCRSAE; Pimentel, 2013) and the NRS EFLs, and that the utmost care is taken in test construction and maintenance procedures, from item development and review, through calibration and scaling, to standard setting and operational procedures and score interpretation.  The MAPT-CCR adheres to the very best practices in testing as defined by the Standards for Educational and Psychological Testing (AERA et al., 2014).

Citation: Zenisky, A. L., Sireci, S. G., Lewis, J., Lim, H., O'Donnell, F., Wells, C. S., Padellaro, F., Jung, H. J., Pham, D., Hong, S. E., Banda, E., Park, Y., Botha, S., Lee, M., & Garcia, A. (2018). Massachusetts Adult Proficiency Tests for College and Career Readiness: Technical Manual. Center for Educational Assessment research report No. 974. Amherst, MA: Center for Educational Assessment.

Faculty, staff, and students from the Research, Educational Measurement, and Psychometrics (REMP) degree programs / Center for Educational Assessment are very active in many international, national, and regional conferences. The information below offers a sampling of UMass participation in selected conferences over the past several years.

NERA 2018 - Trumbull, CT

 

Author(s)Title

Sandra BothaEvaluating the Dimensionality of Vocabulary

Maritza CasasEvaluation of the Full Information Maximum Likelihood Procedure to Handle Missing Data From Adaptive Tests

Amanda Gorham, Sandra Botha, Jennifer Lewis, Frank Padellaro, Lisa KellerReconceptualizing Test Content Representation Using a Generalizability Theory Framework: A Follow Up Study

Hwanggyu Lim, Duy PhamComparison of fixed item parameter calibration equating methods between flexMIRT and PARSCALE

Ella Banda, Stephen G. SireciAdapting the ATMI into Chichewa: Language Validity and Preliminary Psychometric Investigation

Seong Eun Hong, Scott MonroePower Analysis for Latent Growth Models in Educational Research Settings

Minhyeong Lee, Stephen G. SireciGauging Students' Progress on a Multistage-Adaptive Test

HyunJoo Jung, Jennifer RandallEffect of Advanced Placement Experience on College Readiness and College Completion

Sandra Botha, Jennifer Lewis, Amanda Gorham, Frank Padellaro, Lisa KellerGeneralizability Theory in Practice: A Review and Comparison of Methods and Software

Francis O’DonnellBuilding Validity Evidence Based on Consequences of Testing: The Massachusetts Adult Proficiency Test

Dukjae Lee, Lisa Keller, Siyu WanDo You See What I see? Investigating DIF in an Image-Based Assessment

Duy Pham, Uyen LeIs a new testing system in Vietnam harmful for equity?

Yooyoung Park, Hwanggyu LimDetecting DIF Items Using the Bayesian Hierarchical Logistic Regression Model

Darius Taylor, Lisa KellerUsing Differential Option Functioning to Examine Trends of Differential Post-Secondary Aspirations Across Demographic Groups

AERA 2018 - New York

Author(s)Title

Jennifer Lewis, April ZeniskyStandard Setting for Next-Generation Tests: An Inquiry into Theoretical and Methodological Considerations

Sandra BothaDeveloping and Validating a Scale of Social Educational Capital

Amanda Gorham, Sandra Botha, Jenn Lewis, Frank PadellaroEvaluation of Test Content Representation Using a Generalizability Theory Framework

Seong Eun Hong, Hwanggyu Lim, Scott MonroeDetecting DIF in IRT with Continuous Covariates Using flexMIRT

Frank PadellaroImpact of Item Administration in Writing Thought Process: Determining Group Differences in Keystroke Logging

Yooyoung ParkExploring the Utility of Profile Band Feedback Failing Examinees on a Repeat Attempt

NCME 2018 - New York

Author(s)Title

Hwanggyu Lim, Minjeong Shin, Ah-young ShinInvestigating the practical impact of model misfit in IRT

Michelle Boyer, Lisa Keller, Umass, Richard J. PatzEffects of Automated Rater Improvements on Test Equating Solutions

HyunJoo Jung, Pamela Kaliski, Lei WanAn Investigation of the Dimensionality of Large-Scale Hybrid Assessment

Francis O'DonnellUsing Public Data to Examine Potential Effects of an ACT-For-All Policy

Maritza CasasEvaluating the Dimensionality of Multistage-Adaptive Test Data

Duy Pham, Malcolm Bauer, Caroline Wylie, Craig WellsEvaluating a Learning Progression Theory: Comparative Results from Two Psychometric Models

Duy PhamConsidering Sampling Errors in Estimating Value-Added Ratios of Subscores: A Bootstrap Method

Darius Taylor, Frank PadellaroImproving Validity in Image-Based Assessment Using Simplified Line Drawings

Publications

Publications

Gandara, F. & Randall, J. (in press). Assessing mathematics proficiency of bilingual students: the case for translanguaging in the Democratic Republic of the Congo. Comparative Education Review.

Randall, J. & Jung, H. (2018). Confirmatory Factor Analysis in B. Frey (ed.) The SAGE Encyclopedia of Educational Research, Measurement, and Evaluation. Sage Publications, Thousand Oaks, CA.

Jung, H. & Randall, J. (2018). Covariance in B. Frey (ed.) The SAGE Encyclopedia of Educational Research, Measurement, and Evaluation. Sage Publications, Thousand Oaks, CA.

Fan, F. & Randall, J. (2018). Reliability in B. Frey (ed.) The SAGE Encyclopedia of Educational Research, Measurement, and Evaluation. Sage Publications, Thousand Oaks, CA.

Hansen, M. & Monroe, S. (2018). Linking not-quite-vertical scales through multidimensional item response theory. Measurement: Interdisciplinary Research and Perspectives, 16, 155–167.

Monroe, S. (2018). Contributions to estimation of polychoric correlations. Multivariate Behavioral Research, 53, 247–266.

Falk, C. F., & Monroe, S. (2018). On Lagrange Multiplier tests in multidimensional item response theory: Information matrices and model misspecification. Educational and Psychological Measurement, 78, 653–678.

Boyer, M., & Zenisky, A. L. (2018). Technology-enhanced items. In B. Frey (Ed.), The Sage Encyclopedia of Educational Research, Measurement, and Evaluation. Thousand Oaks, CA: Sage.

Banda, E., & Zenisky, A. L. (2018). Performance assessment. In B. Frey (Ed.), The Sage Encyclopedia of Educational Research, Measurement, and Evaluation. Thousand Oaks, CA: Sage.

Zenisky, A. L., Keller, L.A., & Park, Y. (2018). Reporting student growth: Challenges and opportunities. In D. Zapata-Rivera (Ed.), Score Reporting: Research and Applications.

Hambleton, R. K. & Zenisky, A. L. (2018). Score reporting and interpretation. In W. J. van der Linden (Ed.), Handbook of Modern Item Response Theory (2nd ed.).

Faulkner-Bond, M., Wolf, M. K., Wells, C. S., & Sireci, S. G. (2018). Exploring the factor structure of a K-12 English language proficiency assessment. Language Assessment Quarterly, 15(2), 130-149.

Marcotte, A. M., Rick, F., & Wells, C. S. (2018). Investigating the reliability of the sentence verification technique. International Journal of Testing. DOI: 10.1080/15305058.2018.1497636

Hintze, J. M., Wells, C. S., Marcotte, A. M., & Solomon, B. G. (2018). Deceision-making accuracy of CBM progress-monitoring data. Journal of Psychoeducational Assessment, 36(1), 74-81.

Sireci, S. G., Banda, E., & Wells, C. S. (2018). Providing and evaluating test accommodations. In S. N. Elliott, R. J. Kettler, P. A. Beddow, & A. Kurz. Handbook of Accessible Instruction and Testing. Springer, Cham.

Zenisky, A. L., Sireci, S. G., Lewis, J., Lim, H., O'Donnell, F., Wells, C. S., Padellaro, F., Jung, H. J., Pham, D., Hong, S. E., Park, Y., Botha, S., Lee, M., & Garcia, A. (2018). Massachusetts Adult Proficiency Tests for College and Career Readiness: Technical Manual. Center for Educational Assessment research report No. 974. Amherst, MA: Center for Educational Assessment.

Lewis, J., & Sireci, S. G. (2018). Analysis of the National Board for Professional Teaching Standards Candidate Survey. Center for Educational Research Report No. 964. Amherst, MA: Center for Educational Assessment.