June 2023: Test Score Patterns Across Three COVID-19-Impacted School Years

Overview

Concerns about the effect of COVID-19-related school disruptions on students’ academic progress have been raised since the beginning of the pandemic. In addition to overall worries about the impact of disrupted schooling on all students, particular concerns have been raised about deleterious effects on achievement gaps associated with income and with race/ethnicity. To examine these issues, Kuhfeld, Soland, and Lewis (2022) analyzed math and reading data from students tested in grades 3–8 in the fall of 2019, 2020, and 2021. Data were obtained from schools in 48 states that administered the Measures of Academic Progress (MAP) Growth assessment (NWEA, 2019) across all 3 years. Kuhfeld et al. sought to determine whether scores from fall 2020 and fall 2021 were lower than those from fall 2019 and whether achievement gaps between students based on socioeconomic status widened between fall 2019 and fall 2021.

Methodology

Many schools across the United States routinely administer the MAP Growth in the fall, winter, and spring of each year to monitor all students’ progress in reading and math. Kuhfeld et al. used a dataset that comprised fall administrations in approximately 12,000 schools that used the assessment in 2019, 2020, and 2021. Scores from a total of 5.4 million students in grades 3–8 were included in this analysis. Kuhfeld et al. did not attempt to track the same students over 3 years but included only schools that administered the MAP to at least 10 students in all 3 years. In this way, Kuhfeld et al. controlled for differences that might exist between schools that did and did not give the MAP each year. Their sample comprises approximately 12% to 15% of U.S. schools, depending on grade level.

Kuhfeld et al. reported that compared to all U.S. schools, the schools in the analysis sample comprised a similar percentage of urban, suburban, and rural locales. However, the percentage of white students in the schools in the sample was somewhat higher than the average for all schools (55% vs. 49%), and the dataset contained a lower percentage of Hispanic students (20% vs. 26%) and students receiving free or reduced-price lunch (FRPL; 53% vs. 56%), which is a proxy for low socioeconomic status. In their analysis of achievement gaps based on poverty, Kuhfeld et al. categorized schools as low poverty if less than 25% of students in the school qualified for FRPL and high poverty if 75% or more students qualified for FRPL in the 2019–2020 school year.

To quantify the learning growth each year between fall 2019 and fall 2022, Kuhfeld et al. calculated the difference in average reading and math scores between fall 2020 and fall 2019 and fall 2021 and fall 2019. Then they divided the difference in scores by the standard deviation (SD) of the fall 2019 scores. The resulting number represented the difference in SD units between the scores in fall 2019 and scores in fall 2020 and fall 2021. For the analysis of achievement gaps between high- and low-poverty schools, Kuhfeld et al. calculated the difference in mean scores between students in the two types of schools in the same way, standardizing the difference using the fall 2019 scores. Similarly, for the analysis comparing racial/ethnic groups, group differences were calculated for fall 2019 and differences in later years were standardized against fall 2019 scores. In all cases, the data were treated as cross-sectional, not longitudinal, because no attempt was made to track individual students over time. Rather, all scores from students who responded to the MAP Growth at each time point were analyzed.

Key Findings

Math and Reading Achievement During the Pandemic

• Compared to fall 2019, math scores decreased in fall 2020 by 0.11 to 0.18 SD (depending on grade level). In fall 2021, they decreased by 0.21 to 0.27 SD. Declines in scores were similar in magnitude across all grades.
• Compared to fall 2019, reading scores in fall 2020 were similar, changing by 0.05 SD or less. However, scores in fall 2021 showed decreases of 0.09 to 0.18 SD. Students in grades 3–5 had the largest decreases in scores, ranging from 0.15 to 0.18.

Achievement Gaps Based on School Poverty Level During the Pandemic

• In fall 2019, prior to the COVID-19 pandemic, the gap in MAP Growth scores between students in low-poverty schools and high-poverty schools was approximately 1 SD in both reading and math.
• In fall 2020, this gap between students in low- and high- poverty schools remained at approximately 1 SD in reading. In fall 2021, reading scores declined to a lesser degree in low-poverty schools than in high-poverty schools, resulting in the gap in scores increasing by 0.13 SD compared to fall 2019. The gap widened the most for students in grades 3–5.
• In math, the gap in scores between students in low- and high-poverty schools expanded in both years. By fall 2021, the difference in scores between students in low- and high-poverty schools increased by 0.20 SD. As with reading, the gap widened most for students in grades 3–5.

Achievement Gaps Based on Race/Ethnicity During the Pandemic

• The gap in MAP Growth scores for Hispanic students compared to the national average in fall 2019 was approximately one-third of an SD in both reading and math. By fall 2021, the gap widened to approximately half of an SD in math. In reading, the gap grew in both years and reached 0.53–0.63 SD in fall 2021. Gaps increased the most for students in grades 3–5.
• The gap in math MAP Growth scores for Black students compared to the national average increased from 0.44–0.52 SD in fall 2019 to 0.75–0.86 SD in fall 2021. In reading, the gap increased from 0.33–0.42 SD in fall 2019 to 0.50–0.58 in fall 2021. Similar gaps were observed across students in all grades.

Implications and Recommendations for Students With Learning Difficulties and Disabilities

The results of the Kuhfeld et al. provide useful information about the effects of school disruptions during the COVID-19 pandemic. The declines observed in reading and math highlight the need for intervention to help students master the academic skills that they did not learn during the 2019–2020 and 2020–2021 school years. The need for action is particularly acute for students who were in grades 3–5 in the 2020–2021 school year. These students will be in middle school in the fall of 2023 and many may present with learning difficulties. Schools will need to be prepared to screen students to determine whether they need remediation and to provide services to a larger group of students than in the past to help them catch up. This need is likely to be particularly great in schools with high percentages of students from low-socioeconomic-status families and those that serve predominantly Hispanic and Black communities. Although Kuhfeld et al.’s analysis was unable to include students in kindergarten to grade 2, it is reasonable to think that the impact on their academic growth would be similar if not greater than students in grades 3–5, given the overall pattern of seeing greater effects on younger vs. older students.

One of the limitations of Kuhfeld et al.’s dataset is that no information was available indicating which students were receiving special education services. Therefore, their results are unable to shed light on the effect of the pandemic school disruptions on the academic progress of students with learning disabilities. Undoubtedly, special education leaders and teachers will see effects on students with learning disabilities that are at least as large as those observed in the broader population of students. These students likely will need stronger interventions to help them improve their achievement than in the past. Kuhfeld et al. recommend high-dosage tutoring as a promising approach to accelerating learning.

Reference

NWEA. (2019). MAP® Growth™ technical report. https://www.nwea.org/uploads/2021/11/MAP-Growth-Technical-Report-2019_NWEA.pdf