The coronavirus disease 2019 (COVID-19) pandemic has exacerbated concerns about whether differences in school-aggregate test scores can be interpreted to represent real and meaningful differences in school progress and performance. In this report, RAND researchers investigate one issue that may contaminate utilization of COVID-19–era school-aggregate scores: changes in school composition over time.

Cover: Changes in School Composition During the COVID-19 Pandemic
Read Online

Changes in School Composition During the COVID-19 Pandemic

Implications for School-Average Interim Test Score Use

by Jonathan Schweig, Megan Kuhfeld, Melissa Kay Diliberti, Andrew McEachin, Louis T. Mariano

Download eBook for Free

FormatFile SizeNotes
PDF file 0.3 MB

Use Adobe Acrobat Reader version 10 or higher for the best experience.

Research Questions

  1. What percentage of students taking MAP Growth assessments in spring 2019 were assessed again in spring 2021, and what percentage of these students attended the same schools across these two school years? How does this compare to pre-pandemic patterns?
  2. Is there variability in the percentage of students who attended the same schools in these two school years? To what extent does this percentage vary within and among districts?
  3. Among students who took MAP Growth in spring 2019, to what extent is spring 2021 test participation associated with the student, school, and community characteristics of gender, race/ethnicity, poverty, and pandemic vulnerability?

School officials regularly use school-aggregate test scores to monitor school performance and make policy decisions. After the U.S. Department of Education offered assessment waivers to all 50 states in 2019–2020, many educators and policymakers advocated for assessment programs to be restarted in the 2020–2021 school year to evaluate the state of teaching and learning and to inform policies for recovery from the coronavirus disease 2019 (COVID-19) pandemic. However, the use of school-aggregate test scores for these purposes relies on the assumption that differences in aggregate scores can be accurately interpreted as representing real and meaningful differences in school progress and performance. There are serious concerns about the accuracy of such interpretations even under routine schooling conditions, but the pandemic may exacerbate these issues and further compromise the comparability of these test scores. In this report, RAND researchers investigate one specific issue that may contaminate utilization of COVID-19–era school-aggregate scores and result in faulty comparisons with historical and other proximal aggregate scores: changes in school composition over time. To investigate this issue, they examine data from NWEA's Measures of Academic Progress (MAP) Growth assessments, interim assessments used by states and districts during the 2020–2021 school year.

Key Findings

The school composition of MAP Growth test-takers has shifted over time, and this could affect school-aggregate test scores

  • Participation in MAP Growth assessments was lower in 2020–2021 than in pre-pandemic years. Of students taking MAP assessments in spring 2019, 42 percent were assessed again in spring 2021, and 21 percent of these test-takers attended the same schools across these two school years.
  • Within and among districts, there was wide variability in the percentage of students who attended the same schools and participated in assessments over two academic years.
  • Participation in assessments was uneven in 2020–2021. Students of color were less likely than white students to have attended the same schools and participated in assessments over two academic years.
  • Historically higher-achieving students who participated in assessments in a given year were more likely than their peers to have attended the same schools and participated in assessments over two academic years.
  • Schools serving high-poverty and COVID-19–vulnerable communities had systematically fewer students attend the same school and participate in assessments over two academic years than other schools.

The following insights about test participation could help guide school systems as they report school-level test score information from spring 2021 and implement restart and recovery plans

  • Comparing spring 2021 school-aggregate test scores with those from spring 2019 might misrepresent school progress or pandemic impacts.
  • Comparing spring 2021 aggregates among schools might misrepresent the relative performance of schools.
  • Information about low and differential participation in spring 2021 testing should be transparently reported.

Research conducted by

The research reported here was sponsored by the Institute of Education Sciences, U.S. Department of Education and conducted by RAND Education and Labor.

This report is part of the RAND Corporation Research report series. RAND reports present research findings and objective analysis that address the challenges facing the public and private sectors. All RAND reports undergo rigorous peer review to ensure high standards for research quality and objectivity.

This document and trademark(s) contained herein are protected by law. This representation of RAND intellectual property is provided for noncommercial use only. Unauthorized posting of this publication online is prohibited; linking directly to this product page is encouraged. Permission is required from RAND to reproduce, or reuse in another form, any of its research documents for commercial purposes. For information on reprint and reuse permissions, please visit www.rand.org/pubs/permissions.

The RAND Corporation is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors.