April 2019: Evaluating the Effects of Supplemental Reading Intervention Using a Regression Discontinuity Design

Background

Response-to-instruction models, such as multitiered systems of support (MTSS) and response to intervention (RTI), are frequently the framework through which schools identify and address the needs of students in need of intensive and individualized instruction. Although there is a body of research supporting small group reading interventions for students with reading difficulties, there are only a couple of studies evaluating effects of these interventions under naturalistic conditions, such as within the context of a broader school-wide MTSS framework (Balu et al. 2015; Baker et al., 2015). Recent studies evaluating the effectiveness of MTSS frameworks have had conflicting results regarding efficacy of Tier 2 reading interventions (Balu et al. 2015; Baker et al., 2015). Coyne and colleagues (2018) suggest the incongruity of the findings of the studies may be explained by differences in how schools implement MTSS. For instance, such as identification criteria, the type of Tier 2 intervention, the dosage and fidelity of implementation, and the quality of instruction. Given these mixed findings, further research is needed to understand if and under what conditions Tier 2 interventions are most effective within MTSS frameworks.

Study Purpose and Methodology

The purpose of the Coyne et al. (2018) study was to evaluate the impact of a Tier 2 supplemental reading intervention implemented as part of a statewide MTSS initiative. The primary research question sought to determine the impact of small group reading intervention implemented within an MTSS framework on the reading outcomes of students in grades 1 through 3 identified as at risk through universal screening procedures. The researchers also explored whether there were any differences in outcomes by grade and outcome measure.

Coyne et al., identified four schools in four districts as part of a statewide initiative focused on MTSS in reading for studentes in kindergarten through third grade. These schools were selected based on persistently low reading achievement. All students (n = 678) in grades 1 to 3 were screened in the fall using DIBELS. Cut scores were applied to identify students at risk of reading difficulty. The actual cut score values varied by grade level, but were similar to established DIBELS scores indicating the need for strategic support. In total, 308 students were identified as at risk and participated in the Tier 2 reading intervention.

Students were assessed at fall, winter, and spring using DIBELS phonemic sound fluency (PSF), nonsense word fluency (NWF), and oral reading fluency (ORF) subtests. Finally, the research team administered the Woodcock Reading Mastery Test Passage Comprehension subtest at spring.

Intervention

The Tier 2 reading intervention was Proactive Early Interventions in Reading (P-EIR; Mathes, 2005). P-EIR was selected because it has a comprehensive focus, from alphabetic knowledge to decoding multisyllabic words, fluency, and comprehension strategies, and because it has demonstrated efficacy through intervention studies. Student groups were adjusted based on response to intervention. Students in Tier 2 received intervention 4 days a week in 30-40 minute sessions from November to June. The instruction was provided by trained reading interventionists. The schools also employed literacy coaches who supervised and supported the interventionists over the course of the school year.

Regression Discontinuity Design

A sharp regression discontinuity design was used to estimate the effects of receiving the Tier-2 intervention. Regression discontinuity designs vary from other experimental designs, such as a randomized controlled trial, in that they allow all students who meet a certain criteria to participate in the treatment. Thus, unlike experimental trials in which students randomized to receive and not receive treatment are directly compared, regression discontinuity designs compares students who score just below and above a predetermined score at pretest that determines placement in the treatment group. Students below the cut point at pretest receive treatment whereas students above the cut point are withheld treatment. After treatment students have received instruction, researcher plot students’ pretest and posttest scores using linear regression. If there is a “break” in the regression line at the cut off score (as shown in this study), then the treatment can be assumed to have had an effect, as students just below the score outperform the students just above it. If you are interested in learning more in seeing a visual representation or learning more about regression discontinuity design, you might check out this Social Research Methods web-based textbook entry on regression discontinuity design. A regression discontinuity design was an appropriate study design for Coyne et al. because (a) it allowed all students who qualified for Tier 2 services within a MTSS framework to receive services and (b) it allowed the research team to make causal inferences.

Study Findings

Primary Research Question: Across grades, the results show a positive and statistically significant treatment effect on the PSF and NWF subtests of DIBELS. Effect sizes were .39 for phonemic awareness and .36 for decoding. The intervention had no discernible effect on oral reading fluency or reading comprehension.

Secondary Research Question: Phonemic awareness and decoding effects were larger in grades 1 and 2. There were very small effects on reading fluency in grades 2 and 3, and positive effect on reading comprehension in grade 2.

Interpretation of Study Findings and Implications for Future Research

The findings of Coyne et al. (2018) differ from Balu et al. (2015), which found no discernable effects of Tier 2 interventions in RTI contexts and appear to align better with Baker et al. (2015), which found students who received Tier 2 interventions performed better than students who just received a Tier 1 intervention. Coyne et al. (2018) also builds on previous efficacy studies of P-EIR.

In Coyne et al. (2018) schools implemented a Tier 2 intervention with consistency, a high degree of fidelity, and instructional quality and support for interventionists. Additionally, the Tier 2 intervention increased dosage and exposure.  The findings show support for supplemental small group reading interventions as part of Tier 2 for MTSS frameworks in reading. Students in grades 1 to 3 showed improvements in phonemic awareness and decoding. Specifically, the findings show support for interventions that increase instructional intensity through dosage and exposure. Finally, the findings show support that Tier 2 reading interventions can be implemented with high fidelity and quality within MTSS frameworks.

The authors note the state-supported MTSS initiative provided training and resources to interventionists that helped enable the schools to implement the Tier 2 intervention with the high levels fidelity and intensity described above. Although this study did explore Tier 2 intervention in realistic settings, it is possible that other schools and districts may lack the personnel and resources to carry out MTSS frameworks with the high levels of intensity of fidelity described in the study.

The findings from Coyne et al. (2018) provide additional empirical support for the following:

• Identifying at-risk students through universal screening
• Using research validated program, such as P-EIR
• Ensuring high levels of instructional fidelity and quality
• Adjusting instruction on an ongoing basis based on students’ response to instruction
• Increasing instructional dosage and exposure by supplementing Tier 1 reading instruction  with reading inteventions
• Providing intervention in small groups