July 2018: Data-Based Decision Making in Reading Interventions

Overview

There is near universal agreement among reading researchers that students at risk for reading difficulties should receive supplemental, evidence-based reading interventions. However, these interventions are not universally effective; some students will not adequately respond to these interventions. Although there is not widespread agreement about how to best assist students who show persistent reading difficulties when given evidence-based reading intervention support, researchers suggest reading further intensifying intervention support for students with persistent reading difficulties (Fuchs, Fuchs, & Vaughn, 2014). Intensifying reading interventions may take many forms (e.g., decreasing group size, increasing intervention time).

One prominent recommendation for designing effective interventions for students with intractable reading difficulties is data-based decision making. Data-based decision making is an umbrella term for an ongoing process in which educators use individual student data to tailor interventions to the student’s individual needs. Although data-based decision making is widely regarded as a recommended facet of intensive interventions, empirical evidence supporting the efficacy of this approach is limited. 

In their literature review and meta-analysis, Filderman and colleagues (2018) summarize research around data-based decision making and evaluate the effects of such interventions for struggling readers in kindergarten through 12th grade. Two research questions guided their study:

1. What are the features of data-based decision making in reading intervention research?
2. What are the effects of interventions that use data-based decision making on reading outcomes?

Methods

The research questions that Filderman and colleagues pose require distinct methodologies and data. The first question is descriptive and seeks to understand the components and features of data-based decision making as it has been described and evaluated in empirical research. This type of question is best answered through a systematic literature review that summarizes all relevant research on the topic. Systematic literature reviews are different from essays or summaries of research because they use transparent and replicable procedures defined before examining research to minimize bias and maximize the trustworthiness of the findings. Systematic literature reviews include (1) clear inclusion/exclusion criteria for studies, (2) a replicable search process, and (3) systematic coding procedures by which data/information is extracted from studies.

The second question addresses the effects of data-based decision making on the reading outcomes of school-aged struggling readers. Questions of effectiveness are best addressed through meta-analysis, a specific subset of systematic literature reviews that includes all features of a systematic literature review, but also incorporates an analysis that statistically aggregates findings across studies to determine whether there is more generalizable evidence for the effectiveness of the intervention or procedure.

Filderman and colleagues screened nearly 15,000 articles describing previous studies. Articles were included if they described the data source used to make decisions and the process for how data were used as a tool to inform instructional decisions. Fifteen studies met these criteria and were systematically coded to extract data.

Key Findings

Data-based decision making has been used for both code-based reading instruction (i.e., word reading, reading fluency) and meaning-based instruction (i.e., reading comprehension), but it is more commonly used to target code-based instruction. Studies used a variety of decision-making criteria, including mastery measures tied to instructional practices and slope methods, which involve evaluating the steepness of a student’s slope of progress on curriculum-based measures. A majority of studies collected data at least once a week; however, the studies did not frequently report how often decisions were made based on these data or what types of instructional adjustments were made.

The meta-analytic portion of this systematic literature review found that interventions that used data-based decision making outperformed typical practice (nonintervention) comparison groups. The weighted mean meta-analytic effect size was .24, but the confidence interval for this comparison was wide (.01 to .46). Additionally, interventions that incorporated data-based decision making outperformed interventions that did not incorporate data-based decision making (weighted mean meta-analytic effect size = .27). However, only six studies contributed to this analysis, so additional research may be required to have strong confidence in these findings.

Conclusions

The findings of this systematic literature review and meta-analysis suggest that data-based decision making may be an important feature of improved reading interventions. The literature review suggests that data-based decision making has been incorporated into studies in a variety of ways, targeting different component skills. The positive, statistically significant finding suggests that incorporating data-based decision making has a positive effect on student outcomes.

Despite these positive results, the authors state, “the results of this synthesis reveal that claiming [data-based decision making] to be an evidence-based practice is tenuous at best.” Although there are reasons to be optimistic about future research examining data-based decisions making, the authors’ recognition of the limitations of current research on data-based decision making is well-founded. One reason for caution is the fact that most studies (9 of the 15) in this systematic review compared the effects of a reading intervention with data-based decision making to a no-reading-intervention control condition. As a result, these studies cannot isolate the unique effects of adding a data-based decision making process to a reading intervention. Only six studies presented a research design that assessed the unique effects of data-based decision making and the results of these individual studies were mixed. Another reason for caution relates to the way in which data-based decision making was implemented in these studies. Of the six studies that contrasted reading interventions with and without data-based decision making, only two studies implemented a data-based decision making model that was considered by the authors to be highly aligned with National Center on Intensive Interventions recommendations. Thus, it may be that data-based decision making is a more effective practice when implemented in a manner that better aligns with expert recommendations. Well-designed experiments that implement recommended data-based decision making models are needed to further investigate the promise of data-based decision making.

Reference

Fuchs, D., Fuchs, L. S., & Vaughn, S. (2014). What is intensive instruction and why is it important? Teaching Exceptional Children, 46(4), 13–18.