Styck, K. M., Beaujean, A. A., & Watkins, M. W. (2019). Profile reliability of cognitive ability subscores in a referred sample. Archives of Scientific Psychology, 7, 119–128.
Summary by Dr. Pat Taylor
Overview
The current article extends previous research on the use of cognitive profile analysis (CPA). The underlying premise of CPA is that patterns of scores for an individual provide more information than the actual scores. CPA has been applied to many types of tests including vocational, personality, and intelligence. In this forum, the primary interest is in the use of CPA for identifying and treating specific learning disabilities (SLD). In this context CPA is more commonly known as patterns of strengths and weaknesses (PSW). Variants of PSW have been proposed for the identification of SLD, including the Discrepancy/Consistency Method (D/CM; Naglieri, 1999), Concordance/Discordance Method (C/DM; Hale and Fiorello, 2004), and the Dual Discrepancy/Consistency Method (XBA; Flanagan, Ortiz, & Alfonso, 2013).
The use of PSW methods for SLD identification in U.S. public schools is quite popular. Based on a survey of 461 school psychologists, Maki & Adams (2019) report that 35% of US school districts dictate PSW as the method for SLD identification. This popularity exists despite growing evidence cautioning against the use of such methods (Kranzler, Floyd, Benson, Zabowski, and Thibodaux, 2017; Miciak, Fletcher, Stuebing, Vaughn, & Tolar, 2014; Stuebing, Fletcher, Branum-Martin & Francis, 2012; Taylor, Miciak, Fletcher, & Francis, 2016). The current article points out that most tests used for PSW methods were designed to maximize between-subject reliability while proper application of PSW methods requires within-subject reliability. Within-subject reliability for a PSW method would indicate the extent that the set of tests were able to provide stable patterns over time. The concern that within-subject reliability might be lacking in PSW methods led to research questions that focused on within-person reliability of cognitive profiles based on subscores and index scores.
Key Findings
The sample included 296 students who had been tested with the WISC-IV on two occasions. Of those students, 197 were classified with SLD. The tests occurred at an average of 2.84 years apart. Using a method from Bulut and colleagues (2013) the authors calculated the within-person reliability for cognitive subscore profiles of the WISC-IV. Within-person reliability of profile scores based on subtests was 0.37 for the total sample and 0.36 for the SLD subsample. Within-person reliability of profile scores based on index scores was higher at 0.53 for the total sample and 0.52 for the SLD subsample.
Implications
The findings in the current article extend a growing body of research that cautions against the use of PSW methods for the identification of SLD. Although the tests that are commonly used for PSW methods (e.g., WISC-IV, WJ-IV) report high reliability coefficients, these are for between-subjects reliability. The current article demonstrates that profiles constructed from subtests have lower reliabilities than suggested by the between-subjects values. Practitioners should consider the within-subjects reliability when deciding if a PSW method is to be considered for SLD identification.
Additional Reading
Miciak, J., Fletcher, J. M., Stuebing, K. K., Vaughn, S., & Tolar, T. D. (2014). Patterns of cognitive strengths and weaknesses: Identification rates, agreement, and validity for learning disabilities identification. School Psychology Quarterly, 29(1), 21. doi:10.1037/spq0000037
Stuebing, K. K., Fletcher, J. M., Branum-Martin, L., & Francis, D. J. (2012). Evaluation of the technical adequacy of three methods for identifying specific learning disabilities based on cognitive discrepancies. School Psychology Review, 41, 3–22. Retrieved June 23, 2015 from http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3466817/
Taylor, W., Miciak, J., Fletcher, J., & Francis, D. (2016). Cognitive discrepancy models for specific learning disabilities identification: Simulations of psychometric limitations. Psychological Assessment. Advance online publication. doi:10.1037/pas0000356.