REMP faculty, staff, and students, through the Center for Educational Assessment, partner with many national and international agencies to carry out psychometric research and operational testing activities. These collaborations not only provide students in the REMP program with funding as full-time research assistants in the Center, but also offer students and faculty opportunities to work with testing professionals and often lead to papers that are presented at national and international conferences and/or of publishable quality.
Below we provide a sampling of recent activities with details on selected projects. This list is updated as grants and contracts are awarded.
Standardized Assessment Training and Support for Massachusetts Adult Education. [4 years]
As of July 2018, REMP has contracted with the Massachusetts Department of Elementary and Secondary Education’s Adult and Community Learning Services unit to provide support and professional development to adult education providers with respect to all of the standardized assessments used in Massachusetts adult education programs. [In addition to the MAPT-CCR, programs have five other assessments that can be used to measure student gains, depending on student goals and content being assessed.] UMass coordinates both face to face and online training for administration and scoring of these assessments, as well as annual recertification for two of the assessments. We serve as the primary point of contact for the adult education field in Massachusetts with respect to assessment-related questions. In addition, REMP faculty, staff, and students are developing a library of short video resources for adult education instructors and the students themselves. Some of the videos are focused on general topics in assessment such as tips for taking tests, and understanding measurement error and score changes, while others are intended to be informational and somewhat more test-specific, on topics such as the MAPT-CCR score reports or recertification procedures for certain tests.
For more information about our work to support assessment in adult education in Massachusetts, please check out our resource blog.
A Complex Test, A Single Cut: Methodological Approaches to Setting and Validating Cut Scores on the Next Generation NCLEX. [1 year]
Over the past several years, REMP faculty and staff have had two projects funded by the National Council of State Boards of Nursing (NCSBN), as that agency embarks on the development of the next-generation NCLEX exam. Most recently, we completed a project to look into the emerging methods for standard setting in the context of complex assessments. Credentialing tests measure examinee knowledge, skills, and abilities (KSAs) in the context of professional employment and training. While credentialing tests have long focused on content knowledge and asking questions about processes, with the increasing prevalence of technology in many professions and advances in test delivery, more and more credentialing tests are investigating the feasibility of assessing more complex cognitive processes (i.e., clinical judgement and decision making) that job analyses define as necessary for success as an entry-level candidate. To accurately measure this wide range of skills, test developers are increasing the complexity of tests by including a range of item types, mixed scoring models, and adaptive test administration. Given the increasing complexity of next-generation credentialing tests, the process for determining the pass-fail cut-score that delineates a qualified candidate from an unqualified candidate should be informed by these kinds of test elements in order to ensure that the recommended passing score reflects the test and testing context appropriately. The goal of this exploratory study was to develop a deeper understanding of (1) the current and emerging practices in standard setting for complex assessments and (2) the nature of current and optimal approaches to gathering validity evidence for standard setting. We conducted seven in-person or phone interviews with nationally-renown standard setting experts. The analysis of the interviews revealed four general themes, along with several sub-themes. The findings of this study were used to inform recommendations for evaluation, or re-evaluation, of standard setting practices, including the construction of a validity argument related to standard setting for complex tests.
This research was presented at the 2018 AERA Research-in-Progress gala in New York City in 2018, and an extension of the project has been accepted on the 2018 NCME program in Toronto as part of a symposium on visualizations in standard setting.
Technical Assistance to Support the National Board of Professional Teaching Standards Examinations. [2 years]
Lewis, J., & Sireci, S. G. (2018). Analysis of the National Board for Professional Teaching Standards Candidate Survey. Center for Educational Research Report No. 964. Amherst, MA: Center for Educational Assessment.
Since 2003, the Center for Educational Assessment at UMass Amherst has been the test publisher for the Massachusetts Adult Proficiency Test (MAPT). This assessment program is commissioned by the Massachusetts Department of Elementary and Secondary Education’s Adult and Community Learning Services unit, and is used to assess the learning gains of adult students enrolled in education programs across the Commonwealth for federal and state accountability purposes, in mathematics and reading. The MAPT-CCR for Mathematics and for Reading are assessments that are directly targeted to the instruction that adult learners in Massachusetts receive, and learners' proficiency is evaluated against national proficiency standards developed by the U.S. Department of Education (DAEL, 2016). It is a computerized multistage adaptive test, administered on computer via the internet to thousands of students each year. The faculty, staff, and students of the Research in Educational Measurement and Psychometrics program carry out all development and operational maintenance of the MAPT. In 2018, the fourth-generation MAPT was released (the MAPT-College and Career Readiness, or MAPT-CCR). This iteration of the test was updated to reflect the most current curriculum standards (the College and Career Readiness Standards for Adult Education (CCRSAE; Pimentel, 2013).
Having an operational computerized test housed in the College of Education at UMass has provided students and faculty alike with rich opportunities for both theoretical and applied psychometric research, as well as the chance to contribute to the ongoing operational activities associated with a live testing program. A listing of research reports involving MAPT data, completed by REMP students and faculty, can be found here (link to the listing from the MAPT-CCR TM).
In our Technical Manual, we have compiled psychometric evidence - drawn from a wide variety of sources and data analysis methods including surveys, studies involving expert judgment, simulations, and other qualitative and quantitative analyses - to make a convincing argument for the high technical quality of the MAPT-CCR as a measure of AE learners' knowledge and skills in mathematics and reading, relative to their progress in meeting educational goals. The data and processes presented here illustrate the ways in which the content of the MAPT-CCR is aligned to the College and Career Readiness Standards for Adult Education (CCRSAE; Pimentel, 2013) and the NRS EFLs, and that the utmost care is taken in test construction and maintenance procedures, from item development and review, through calibration and scaling, to standard setting and operational procedures and score interpretation. The MAPT-CCR adheres to the very best practices in testing as defined by the Standards for Educational and Psychological Testing (AERA et al., 2014).
Citation: Zenisky, A. L., Sireci, S. G., Lewis, J., Lim, H., O'Donnell, F., Wells, C. S., Padellaro, F., Jung, H. J., Pham, D., Hong, S. E., Banda, E., Park, Y., Botha, S., Lee, M., & Garcia, A. (2018). Massachusetts Adult Proficiency Tests for College and Career Readiness: Technical Manual. Center for Educational Assessment research report No. 974. Amherst, MA: Center for Educational Assessment.
Faculty, staff, and students from the Research, Educational Measurement, and Psychometrics (REMP) degree programs / Center for Educational Assessment are very active in many international, national, and regional conferences. The information below offers a sampling of UMass participation in selected conferences over the past several years.
|Sandra Botha||Evaluating the Dimensionality of Vocabulary|
|Maritza Casas||Evaluation of the Full Information Maximum Likelihood Procedure to Handle Missing Data From Adaptive Tests|
|Amanda Gorham, Sandra Botha, Jennifer Lewis, Frank Padellaro, Lisa Keller||Reconceptualizing Test Content Representation Using a Generalizability Theory Framework: A Follow Up Study|
|Hwanggyu Lim, Duy Pham||Comparison of fixed item parameter calibration equating methods between flexMIRT and PARSCALE|
|Ella Banda, Stephen G. Sireci||Adapting the ATMI into Chichewa: Language Validity and Preliminary Psychometric Investigation|
|Seong Eun Hong, Scott Monroe||Power Analysis for Latent Growth Models in Educational Research Settings|
|Minhyeong Lee, Stephen G. Sireci||Gauging Students' Progress on a Multistage-Adaptive Test|
|HyunJoo Jung, Jennifer Randall||Effect of Advanced Placement Experience on College Readiness and College Completion|
|Sandra Botha, Jennifer Lewis, Amanda Gorham, Frank Padellaro, Lisa Keller||Generalizability Theory in Practice: A Review and Comparison of Methods and Software|
|Francis O’Donnell||Building Validity Evidence Based on Consequences of Testing: The Massachusetts Adult Proficiency Test|
|Dukjae Lee, Lisa Keller, Siyu Wan||Do You See What I see? Investigating DIF in an Image-Based Assessment|
|Duy Pham, Uyen Le||Is a new testing system in Vietnam harmful for equity?|
|Yooyoung Park, Hwanggyu Lim||Detecting DIF Items Using the Bayesian Hierarchical Logistic Regression Model|
|Darius Taylor, Lisa Keller||Using Differential Option Functioning to Examine Trends of Differential Post-Secondary Aspirations Across Demographic Groups|
|Hwanggyu Lim, Minjeong Shin, Ah-young Shin||Investigating the practical impact of model misfit in IRT|
|Michelle Boyer, Lisa Keller, Umass, Richard J. Patz||Effects of Automated Rater Improvements on Test Equating Solutions|
|HyunJoo Jung, Pamela Kaliski, Lei Wan||An Investigation of the Dimensionality of Large-Scale Hybrid Assessment|
|Francis O'Donnell||Using Public Data to Examine Potential Effects of an ACT-For-All Policy|
|Maritza Casas||Evaluating the Dimensionality of Multistage-Adaptive Test Data|
|Duy Pham, Malcolm Bauer, Caroline Wylie, Craig Wells||Evaluating a Learning Progression Theory: Comparative Results from Two Psychometric Models|
|Duy Pham||Considering Sampling Errors in Estimating Value-Added Ratios of Subscores: A Bootstrap Method|
|Darius Taylor, Frank Padellaro||Improving Validity in Image-Based Assessment Using Simplified Line Drawings|
|Jennifer Lewis, April Zenisky||Standard Setting for Next-Generation Tests: An Inquiry into Theoretical and Methodological Considerations|
|Sandra Botha||Developing and Validating a Scale of Social Educational Capital|
|Amanda Gorham, Sandra Botha, Jenn Lewis, Frank Padellaro||Evaluation of Test Content Representation Using a Generalizability Theory Framework|
|Seong Eun Hong, Hwanggyu Lim, Scott Monroe||Detecting DIF in IRT with Continuous Covariates Using flexMIRT|
|Frank Padellaro||Impact of Item Administration in Writing Thought Process: Determining Group Differences in Keystroke Logging|
|Yooyoung Park||Exploring the Utility of Profile Band Feedback Failing Examinees on a Repeat Attempt|
|Gandara, F. & Randall, J. (in press). Assessing mathematics proficiency of bilingual students: the case for translanguaging in the Democratic Republic of the Congo. Comparative Education Review.|
|Randall, J. & Jung, H. (2018). Confirmatory Factor Analysis in B. Frey (ed.) The SAGE Encyclopedia of Educational Research, Measurement, and Evaluation. Sage Publications, Thousand Oaks, CA.|
|Jung, H. & Randall, J. (2018). Covariance in B. Frey (ed.) The SAGE Encyclopedia of Educational Research, Measurement, and Evaluation. Sage Publications, Thousand Oaks, CA.|
|Fan, F. & Randall, J. (2018). Reliability in B. Frey (ed.) The SAGE Encyclopedia of Educational Research, Measurement, and Evaluation. Sage Publications, Thousand Oaks, CA.|
|Hansen, M. & Monroe, S. (2018). Linking not-quite-vertical scales through multidimensional item response theory. Measurement: Interdisciplinary Research and Perspectives, 16, 155–167.|
|Monroe, S. (2018). Contributions to estimation of polychoric correlations. Multivariate Behavioral Research, 53, 247–266.|
|Falk, C. F., & Monroe, S. (2018). On Lagrange Multiplier tests in multidimensional item response theory: Information matrices and model misspecification. Educational and Psychological Measurement, 78, 653–678.|
|Boyer, M., & Zenisky, A. L. (2018). Technology-enhanced items. In B. Frey (Ed.), The Sage Encyclopedia of Educational Research, Measurement, and Evaluation. Thousand Oaks, CA: Sage.|
|Banda, E., & Zenisky, A. L. (2018). Performance assessment. In B. Frey (Ed.), The Sage Encyclopedia of Educational Research, Measurement, and Evaluation. Thousand Oaks, CA: Sage.|
|Zenisky, A. L., Keller, L.A., & Park, Y. (2018). Reporting student growth: Challenges and opportunities. In D. Zapata-Rivera (Ed.), Score Reporting: Research and Applications.|
|Hambleton, R. K. & Zenisky, A. L. (2018). Score reporting and interpretation. In W. J. van der Linden (Ed.), Handbook of Modern Item Response Theory (2nd ed.).|
|Faulkner-Bond, M., Wolf, M. K., Wells, C. S., & Sireci, S. G. (2018). Exploring the factor structure of a K-12 English language proficiency assessment. Language Assessment Quarterly, 15(2), 130-149.|
|Marcotte, A. M., Rick, F., & Wells, C. S. (2018). Investigating the reliability of the sentence verification technique. International Journal of Testing. DOI: 10.1080/15305058.2018.1497636|
|Hintze, J. M., Wells, C. S., Marcotte, A. M., & Solomon, B. G. (2018). Deceision-making accuracy of CBM progress-monitoring data. Journal of Psychoeducational Assessment, 36(1), 74-81.|
|Sireci, S. G., Banda, E., & Wells, C. S. (2018). Providing and evaluating test accommodations. In S. N. Elliott, R. J. Kettler, P. A. Beddow, & A. Kurz. Handbook of Accessible Instruction and Testing. Springer, Cham.|
|Zenisky, A. L., Sireci, S. G., Lewis, J., Lim, H., O'Donnell, F., Wells, C. S., Padellaro, F., Jung, H. J., Pham, D., Hong, S. E., Park, Y., Botha, S., Lee, M., & Garcia, A. (2018). Massachusetts Adult Proficiency Tests for College and Career Readiness: Technical Manual. Center for Educational Assessment research report No. 974. Amherst, MA: Center for Educational Assessment.|
|Lewis, J., & Sireci, S. G. (2018). Analysis of the National Board for Professional Teaching Standards Candidate Survey. Center for Educational Research Report No. 964. Amherst, MA: Center for Educational Assessment.|