Finding the learning loss data needed to drive learning recovery

The National Assessment of Education Progress (NAEP) Report Card on learning loss was a sobering but not unexpected reckoning for how deeply and broadly the pandemic impacted student learning and achievement. 

NAEP state-level findings of drops in math and reading scores were followed by the release of the Education Recovery Scorecard, which leveraged NAEP data to offer the first comparable view of district level learning loss during the pandemic. This one-two punch confirms that COVID-19 learning loss was extensive and, in some cases, worse than expected. Recommendations on how to move forward are not in short supply, and for many, data lies at the heart of transitioning from learning loss to learning recovery. 

Funding, policy, and learning decisions without data is a recipe for disaster – particularly given estimates that it will take hundreds of billions of dollars to offset the impact of learning loss. But we also need the right data and the right approach to interpreting this data, to initiate a successful learning recovery process.  

Holistic learning loss data only goes so far

During my tenure as South Dakota Secretary of Education, I witnessed firsthand the importance of data to support and enhance all aspects of student, teacher, and institutional performance. With roughly 150 school districts, statewide data held value, but the diversity of education experiences across urban and rural areas underscored the need for individual student data as well. In 2019, 40 percent of South Dakota students attended rural public schools, which meant different student-teacher ratios and access to digital learning. Despite assumptions, however, standardized test scores in rural areas often kept pace or outpaced those in more populated areas. 

State and district-level learning loss data is critical. But learning recovery requires analysis of individual student-level data. That’s why attention is being paid to a dozen states that have been specifically tracking COVID learning loss all the way down to individual students. All told, the data from these states represented approximately 15 million students who participated in state assessment programs.  The individual state analyses used students’ entire available testing histories in all tested grades and subjects. In this approach, students are compared to themselves. 

This statistical approach is used to predict how students would have scored on assessments absent the pandemic. By comparing those results to the expected scores and assessing how students performed versus how they were expected to perform, one can arrive at a student-specific measure of learning loss. The intended value is to reveal the strengths and struggles by school, grade, subject, student group, and individual students. That’s the level of information education leaders, and teachers need to make instructional decisions and allocate resources for learning recovery and acceleration. 

Student-level data will help guide learning recovery 

Regardless of how states responded to the pandemic, this look at student-level data revealed commonalities that mirror the national findings, as well as anomalies that should be heeded when making learning recovery investments.