«Istation's Indicators of Progress (ISIP) Advanced Reading Technical Report Version 4 Computer Adaptive Testing System for Continuous Progress ...»
Istation's Indicators of Progress
(ISIP) Advanced Reading
Computer Adaptive Testing System for Continuous Progress Monitoring
of Reading Growth for Students Grade 4 through Grade 8
Patricia Mathes, Ph.D.
2000 Campbell Centre II
8150 North Central Expressway
Dallas, Texas 75206
Copyright ©2016 Istation. All rights reserved.
Istation's Indicators of Progress
(ISIP) Advanced Reading
Technical Manual (2016)
by Patricia Mathes, Ph.D.
Reid Lyon, Ph.D.
Gale Roid, Ph.D.
With assistance from:
Kevin Kalinowski, Ph.D.
Dawn Levy, M.Ed.
Beverly Weiser, Ph.D.
Diane Gifford, M.Ed.
Jenny Lawton, M.Ed.
Olga Palchik, M.S.
Ratna Englehart, M.Ed.
Margaret Lamar Tracey Roden, M.Ed.
ISIP AR Technical Manual Table of Contents Chapter 1: Introduction
Background and Significance
The Need for Continuous Progress Monitoring
Computer Adaptive Testing
Continuous Monitoring of Advanced Reading Skills
ISIP Advanced Reading Assessment Domains and Subtests
ISIP Advanced Reading Item Design and Development
The ISIP Advanced Reading Link to Instructional Planning
Data Analysis and Results
Evidence of Validity
Instructional Tier Goals
Chapter 1: Introduction ISIP™, Istation’s Indicators of Progress, Advanced Reading (ISIP Advanced Reading) is a sophisticated, web-delivered Computer Adaptive Testing (CAT) system that provides Continuous Progress Monitoring (CPM) by frequently assessing and reporting student ability in critical domains of reading throughout, and across, academic years. ISIP Advanced Reading is the upward extension of a similar CAT reading assessment for Grades Pre-K to Grade 3, Istation’s Indicators of Progress, Early Reading (ISIP Early Reading). ISIP Advanced Reading is the culmination of many years of work begun by Patricia G. Mathes, Ph.D. on extending computerized CPM applications to middle grades while assisting teachers with information about student reading ability across the academic year.
Designed for students in Grades 4–8, ISIP Advanced Reading provides teachers and other school personnel with easy-to-interpret, web-based reports that detail student strengths and deficits and provide links to teaching resources. Use of this data allows teachers to more easily make informed decisions regarding each student’s response to targeted reading instruction and intervention strategies.
ISIP Advanced Reading provides growth information in four critical domains of reading, including Word Analysis, Text Fluency, Vocabulary, and Comprehension. It is designed to (a) identify specific reading needs of the older struggling reader, (b) provide automatic continuous progress monitoring of skills, and (c) provide immediate and automatic linkage of assessment data to student-learning needs, which facilitates differentiated instruction.
ISIP Advanced Reading has been designed to automatically provide continuous measurement of Grade 4–8 students throughout the school year, as well as across school years, in critical areas of reading, including word analysis, text fluency, vocabulary, and comprehension. Importantly, there are no other valid reading assessment tools for the middle grades that can precisely identify reading abilities and deficits in all four critical domains for the purposes of continuous and differentiated instruction. This is accomplished through short tests, or "probes," administered at least monthly, that target critical areas to inform instruction.
Assessments are computer-based, and teachers can arrange for entire classrooms to take assessments as part of scheduled computer lab time or individually as part of a workstation rotation conducted in the classroom. The entire assessment battery for any assessment period requires thirty minutes or less. It is feasible to administer ISIP Advanced Reading assessments to an entire classroom, an entire school, and even an entire district in a single day, given adequate computer resources. Classroom and individual student results are immediately available to teachers, illustrating each student’s past and present performance and skill growth. Teachers are alerted when a particular student is not making adequate progress so that the instructional program can be modified before a pattern of failure becomes established.
Background and Significance Perhaps the most important job of schools and teachers is to ensure that all children become competent readers, capable of fully processing the meaning of complicated texts from a variety of venues. Reading proficiency in our information-driven society largely determines a child’s academic, social, occupational, and health trajectory for the rest of his or her life. In a society that requires increasingly higher literacy skills of its citizenry, it cannot be stated strongly enough that teaching every child to read well is not an option, but a necessity. Every child who can read benefits society by being healthier, more fully employed, and better informed.
Sadly, teaching every child to read is a goal we are far from achieving. Large portions of our children continue to struggle to become competent readers (National Reading Panel, 2000; Lyon, 2005). By the middle grades (Grades 4-8), students are expected to demonstrate the ability to read and comprehend grade-level, content-area texts. Yet, for most middle grade students, this is not their reality. The 2007 National Assessment of Educational Progress (Lee, Grigg, & Donahue, 2007) indicates that 74% of 8th graders nationwide struggle to read and gain information from their textbooks, making success in school very difficult. Without adequate reading skills to comprehend and apply information from text, students frequently experience school failure. In fact, many students drop out of school as soon as they are able (Alliance for Excellent Education, 2006). Thus, the middle grades may be the last opportunity for older readers to "catch up" (Bryant et al., 2000).
Older struggling readers are often casualties of prior inadequate reading instruction that insufficiently taught the critical skills necessary for fluent reading and deep processing of text. Many of these students are able to "catch up" in critical reading areas with sufficient targeted instruction (Torgesen et al., 2007). However, many students in the middle grades have little access to effective reading instruction, simply because there
1-2 Chapter 1: IntroductionISIP AR Technical Manual are no reliable and valid assessments that can help their teachers to provide targeted instruction tailored to their needs. Without effective assessments to assist teachers in providing data-informed instruction, many students make little progress year-to-year. This lack of progress is particularly damaging during the middle grade years (Grades 4-8), where learning content-area subject matter becomes a priority. Put succinctly, children who have not learned to read cannot read to learn.
These are not new findings. Overall reading achievement in the United States has remained flat since 1971, when national data were first reported. Because of this alarming and persistent trend, the National Institutes of Health (NIH), through the National Institute of Child Health and Human Development (NICHD), initiated a comprehensive, multidisciplinary effort in 1983 to (1) map the cognitive, linguistic, perceptual, genetic, and neurobiological foundations of reading development; (2) determine the causes of reading failure; and (3) identify and/or develop effective interventions for struggling readers (Lyon, 1985, 1999, 2002; Lyon & Gray, 1992; Lyon & Moats, 1997). Beginning in 1997, and every year until 2005, NICHD program scientists testified on the status of this research in response to requests from Congressional House and Senate Education and Health Committees (Lyon, 1997, 2002b-2005).
These requests were based, in part, on Congressional concerns that the consequences of reading failure went far beyond difficulties in school. NICHD scientists continue to report replicated data showing that reading failure not only constitutes an educational problem, but also a social and public health problem.
Specifically, low reading performance is the strongest predictor of students dropping out of school.
Consequently, dropouts are more than eight times as likely to be in jail or prison as high school graduates, and nearly 70% of prison inmates score at the lowest two levels of literacy (below fourth grade), with 19% being completely illiterate (Lyon, 1997, 1998). Equally alarming is that poor reading portends adverse health disparities and outcomes, including increased incidence of chronic illness, drug and alcohol abuse, risky sexual behavior, less than optimal use of preventive health services, difficulties accessing medical care, and difficulties understanding health risks (Lyon, 2002a).
The Need for Continuous Progress Monitoring While the statistics for the long-term outcomes of reading failure are grim, the solution (i.e., reading success for all students) has thus far eluded our schools. While ultimately we want all children to leave the early grades reading, the fact that so many children leave the early grades without a firm foundation for reading suggests that teachers in the middle grades require help to better serve their students. Importantly, a number of efficacy studies have demonstrated that middle grade students are able to "catch up" in critical reading areas with sufficient differentiated instruction (Fletcher, Lyon, Fuchs, & Barnes, 2007; Torgesen et al., 2007). However, for students to receive such targeted instruction, their teachers must first have information about which areas and skills to target for which students.
Teaching that includes frequent monitoring of student progress has been shown to produce higher student outcomes in reading and mathematics than when monitoring is absent (Conte & Hintze, 2000; Mathes, Fuchs, Roberts, & Fuchs, 1998; Ysseldyke & Bolt, 2007). Also, teachers who use Continuous Progress
Monitoring (CPM) data to plan instruction have a more realistic conception of the capabilities of their students than teachers who do not regularly use student data to inform their decisions (Fuchs, Fuchs, Hamlett, & Stecker, 1991; Mathes et al., 1998). Thus, in order to differentiate, teachers must have reliable and valid CPM assessment tools to (a) determine the specific reading needs of individual children at all levels of the achievement continuum, (b) determine which instructional methods and strategies would be most effective, and (c) monitor children’s progress frequently (i.e., at least monthly) over time so that instructional changes can be made when necessary. Unfortunately, assessment tools for any grade level that meet all of these criteria are sorely lacking. Currently, the only CPM reading assessments available for students in the middle grades require one-to-one administration by a teacher to a student.
Computer Application The problem with most CPM systems is that they have been cumbersome for teachers to utilize (Stecker & Whinnery, 1991). Teachers have to physically administer the tests to each child individually and then graph data by hand. The introduction of handheld technology has allowed for graphing of student results, but information in this format is often not available on a timely basis. Even so, many teachers find administering the assessments onerous. The result has been that CPM has not been as widely embraced as would be hoped, especially within general education. Computerized CPM applications are a logical step to increasing the likelihood that continuous progress monitoring occurs more frequently, with monthly or even weekly assessments. Computerized CPM applications using parallel forms have been developed and used successfully in upper grades in mathematics and spelling (Fuchs et al., 1995). Computerized applications save time and money. They eliminate burdensome test administrations and scoring errors by calculating, compiling, and reporting scores. They provide immediate access to student results that can be used to affect instruction. They provide information organized in formats that automatically group students according to risk and recommended instructional levels. Student results are instantly plotted on progress charts with trend lines projecting year-end outcomes based upon growth patterns, eliminating the need for the teacher to manually create monitoring booklets or analyze results.
Computer Adaptive Testing With recent advances in Computer Adaptive Testing (CAT) and computer technology, it is now possible to create CPM assessments that adjust to the actual ability of each child. Thus, CAT replaces the need to create parallel forms. Assessments built on CAT are sometimes referred to as "tailored tests," because the computer selects items for students based on their performance, thus tailoring the assessment to match the performance abilities of the student. This also means that students who are achieving significantly aboveor below-grade expectations can be assessed to more accurately reflect their true abilities.
There are many advantages of using a CAT model rather than a more traditional parallel forms model, as is used in Dynamic Indicators of Basic Early Literacy Skills (DIBELS). First, it is virtually impossible to create alternate forms of any assessment that are truly parallel. Thus, reliability from form to form will always be