THE RELATION BETWEEN A MATHEMATICS CURRICULUM-BASED MEASURE AND MATHEMATICS PERFORMANCE ON EXPLORE by CAREY ABENDROTH KILLEN A DISSERTATION Presented to the Department of Educational Methodology, Policy, and Leadership and the Graduate School of the University of Oregon in partial fulfillment of the requirements for the degree of Doctor of Education June 2013 ii DISSERTATION APPROVAL PAGE Student: Carey Abendroth Killen Title: The Relation Between a Mathematics Curriculum-based Measure and Mathematics Performance on EXPLORE This dissertation has been accepted and approved in partial fulfillment of the requirements for the Doctor of Education degree in the Department of Educational Methodology, Policy, and Leadership by: Gerald Tindal, Ph.D. Chair Akihito Kamata, Ph.D. Member Ben Clarke, Ph.D. Member David Levin, Ph.D. Outside Member and Kimberly Andrews Espy Vice President for Research and Innovation Dean of the Graduate School Original approval signatures are on file with the University of Oregon Graduate School. Degree awarded June 2013 iii © 2013 Carey Abendroth Killen iv DISSERTATION ABSTRACT Carey Abendroth Killen Doctor of Education Department of Educational Methodology, Policy, and Leadership June 2013 Title: The Relation Between a Mathematics Curriculum-based Measure and Mathematics Performance on EXPLORE Educators need clear, actionable data to help them understand students’ current levels of performance and students’ probable trajectory toward college- and career- readiness in math if they are to make informed programmatic decisions to shape that trajectory. This study explored the relation between CBM-math in Grade 7 as a one- point, teacher accessible measure of student math skill and the students’ performance on the Grade 8 EXPLORE-math test, a large-scale achievement test linked to one set of college- and career-readiness benchmarks. Results indicated that a moderate positive correlation and predictive relation exist between CBM-math and EXPLORE-math. Information was disaggregated by gender and for subgroups, including students eligible for special education, free or reduced meals, and English language development services. No difference in means for male and female students on either measure was identified, but eligibility for special education or for free or reduced lunch was associated with lower performance on both measures. Insufficient numbers of ELD students hindered detailed analysis, but none of the ELD students included in the study achieved the EXPLORE benchmark or the CBM normalized cut score based on the 40th percentile. v An ROC analysis showed that easyCBM consistently predicted students who did not meet the EXPLORE benchmark, although results indicated that a higher cut score on easyCBM may be a more consistent predictor. The study adds to validity research on CBM and may be useful for educators seeking to identify students at risk of missing achievement benchmarks and make programmatic decisions to ensure students are on track to be college- and career-ready in math. vi CURRICULUM VITAE NAME OF AUTHOR: Carey Abendroth Killen GRADUATE AND UNDERGRADUATE SCHOOLS ATTENDED: University of Oregon, Eugene, Oregon University of Salamanca, Salamanca, Spain DEGREES AWARDED: Doctor of Education, Educational Methodology, Policy, and Leadership, 2013, University of Oregon Master of Arts, Spanish Language and Cultures, 2008, University of Salamanca Bachelor of Arts, Spanish and Music, 1990, University of Oregon AREAS OF SPECIAL INTEREST: Professional Development for Instructors in Classroom Engagement, Academic Vocabulary, Instructional Technologies, Second Language Instruction, and Assessment International Diploma Programme and International Education Educational Equity Gifted and Talented Education PROFESSIONAL EXPERIENCE: International Baccalaureate Diploma Programme Coordinator, Bend-La Pine Schools District, Bend, Oregon 2011-Present Advanced Placement and International Baccalaureate Programme Spanish Teacher, Bend Senior High School, Bend, Oregon, 2007-Present Spanish Instructor, Central Oregon Community College, Bend, Oregon, 2007-2008 Advanced Placement and International Baccalaureate Programme Spanish Teacher, Beaverton Senior High School, Beaverton, Oregon, 1995-2005 Fulbright Exchange Teacher, English Language Instructor and Head of Secondary English Program, Campvs College, Santiago, Chile, 1997 Middle School Spanish and Language Arts Teacher, Meadow Park Middle School, Beaverton, Oregon, 1991-1995 vii ACKNOWLEDGMENTS I owe a debt of gratitude to the outstanding educators who have served as true mentors in this journey. I would like to thank the members of my committee, Dr. Akihito Kamata, for the invaluable instruction he provided in statistics, for Dr. Ben Clarke for his detailed advice and edits, and Dr. David Levin for taking time during his sabbatical to serve as outside member on my committee. Heartfelt thanks go to Dr. Gerald Tindal whose gifts in instruction and encouragement inspire me to be a better researcher, writer, and educator. I am deeply indebted to our D.Ed. cohort whose camaraderie, professionalism, and passion for serving students made this a memorable and meaningful journey. Last, I would like to thank my students over the years whose resilience, talents, and energy continue to surprise and inspire me. viii For my amazing husband Robert and my incredible sons Ethan, Liam, and Rowan, whose love, encouragement, and great sense of humor are a continual source of inspiration, and to my parents, Terrell and Annis Abendroth who taught me the joy of learning. ix TABLE OF CONTENTS Chapter Page I. INTRODUCTION ................................................................................................ 1 Foundations for the Use of Curriculum-based Measures in Math ................... 3 The Technical Adequacy of Math CBMs ........................................................ 5 Content Domain of Math and CBM-math Development ............................. 5 Criterion Validity of Math CBMs ................................................................ 8 Predictive Validity of Math CBMs .............................................................. 10 Research on the Use of CBMs for Instruction, Programmatic Decisions, and Projections ................................................................................................. 12 The Importance of Understanding Middle School Students’ Math Achievement Trajectory ................................................................................... 15 College- and Career-readiness in Math........................................................ 16 The Pivotal Nature of Middle School Math Achievement as It Relates to College- and Career-Readiness ................................................................ 18 Middle School Math Assessments and College- and Career-readiness in Math ......................................................................................................... 22 Gaps in the CBM Research and Warrants for a Study on the Relation between CBM and College- and Career-readiness .......................................... 27 Summary of CBMs in Middle School Programmatic Decision-making to Shape Students’ Future Math Achievement .................................................... 30 II. METHODOLOGY ............................................................................................... 34 Setting and Participants.................................................................................... 34 Curriculum ....................................................................................................... 37 Procedures ........................................................................................................ 39 x Chapter Page Training and Administration for easyCBM-math ........................................ 40 Training and Administration for EXPLORE-math ...................................... 40 Measures .......................................................................................................... 40 Description of Grade 7 easyCBM-math ...................................................... 41 Reliability ................................................................................................ 42 Criterion-related Validity Evidence ........................................................ 42 Description of Grade 8 EXPLORE-math .................................................... 44 Reliability ................................................................................................ 44 Content-related Validity Evidence .......................................................... 44 Analyses .......................................................................................................... 46 III. RESULTS .......................................................................................................... 52 Cases Included and General Description ........................................................ 52 Research Question 1: Relation Between easyCBM-math and EXPLORE-math ............................................................................................. 55 Research Question 2: Relation Between Demographic Variables and Math Performance on CBM and EXPLORE .......................................... 56 Regression of easyCBM-math on Demographic Variables ......................... 58 Regression of EXPLORE-math on Demographic Variables ....................... 60 Research Question 3: Consistency of CBM Prediction of Achievement of Early College- and Career-readiness Benchmark on EXPLORE-math ..... 64 Summary ......................................................................................................... 68 xi Chapter Page IV. DISCUSSION ..................................................................................................... 69 Main Findings ................................................................................................. 69 Limitations ...................................................................................................... 70 Identification of F/RL, SpEd, and ELD ....................................................... 70 Differences in Math Preparation .................................................................. 70 Differences in the Assessments ................................................................... 71 Difference in Linguistic Load .................................................................. 71 Differences in Content ............................................................................. 71 Operationalization of Construct of College- and Career-readiness ............ 72 Statistical Validity ........................................................................................ 72 Interpretations .................................................................................................... 72 Eighth-grade Math Performance .................................................................. 73 Criterion-related Validity Evidence for Math CBMS .................................. 73 CBM-math as a Predictor............................................................................. 75 Demographic Characteristics as Predictors ................................................. 76 Consistency of CBMs in Predicting Later Performance .............................. 77 Conclusions and Implications ............................................................................ 78 Areas for Future Research ................................................................................. 82 CBMs and Measures of College- and Career-readiness ............................... 82 Consideration of Diverse Populations .......................................................... 83 Identification of Students at Risk .................................................................. 84 xii Chapter Page APPENDICES ............................................................................................................. 85 A. DISTRIBUTIONS OF EASYCBM AND EXPLORE WITH NORMAL CURVE ............................................................................................................. 85 B. SCATTERPLOT OF EASYCBM AND EXPLORE AND DISTRIBUTION OF RESIDUALS OF REGRESSION OF EXPLORE ON EASYCBM .......... 86 C. ROC CURVE INCLUDING EASYCBM-MATH AND EXPLORE-MATH AND COORDINATES OF THE CURVE ....................................................... 87 D. SAMPLE OF GRADE 7 EASYCBM-MATH AND GRADE 8 EXPLORE-MATH ........................................................................................... 90 REFERENCES CITED ................................................................................................ 91 xiii LIST OF FIGURES Figure Page 1. Distribution of Grade 7 easyCBM-math Results ........................................... 85 2. Distribution of Grade 8 EXPLORE-math Results ......................................... 85 3. Scatterplot of easyCBM-math and EXPLORE-math .................................... 86 4. Distribution of Residuals for Regression of EXPLORE on easyCBM and Demographic Variables ........................................................................... 86 5. ROC Curve Including easyCBM-math and EXPLORE-math ....................... 87 xiv LIST OF TABLES Table Page 1. College-Readiness Benchmarks for EXPLORE, PLAN, and ACT ............... 25 2. Grades 6-8 Student Race or Ethnicity ............................................................ 35 3. Seventh-grade Student Subgroups ................................................................. 36 4. Math Course Enrollment for Seventh-Graders .............................................. 38 5. Estimated Reliabilities for Grade 8 EXPLORE Math Tests .......................... 44 6. Non-performance Variables for Students in Participating Schools ............... 49 7. Demographic Data for easyCBM and EXPLORE Participants ..................... 50 8. Performance Variables’ Names, Descriptions, and Coding Definitions........ 50 9. Descriptive Statistics for easyCBM-math and EXPLORE-math................... 53 10. ANOVA for easyCBM and EXPLORE for F/RL, SpEd and ELD ............... 54 11. Regression of EXPLORE-math on easyCBM-math ...................................... 56 12. Subgroup Achievement of easyCBM and EXPLORE benchmarks .............. 57 13. Regression of easyCBM on Free or Reduced Lunch ..................................... 58 14. Regression of easyCBM on Special Education ............................................. 59 15. Regression of easyCBM on English Language Learners .............................. 59 16. Regression of EXPLORE on Free or Reduced Lunch ................................... 60 17. Regression of EXPLORE on Special Education ........................................... 61 18. Simultaneous Regression of easyCBM on Free or Reduced Lunch, Special Education, and English Language Learner ....................................... 62 19. Simultaneous Regression of EXPLORE on easyCBM, Free or Reduced Lunch, Special Education, and English Language Learner ........................... 63 xv Table Page 20. Case Processing Summary for easyCBM and EXPLORE ............................. 65 21. Area Under the Curve ..................................................................................... 66 22. Grade 7 easyCBM-math and Grade 8 EXPLORE-math Crosstabulation ...... 67 1A. Coordinates of the Curve ............................................................................... 88 1 CHAPTER I INTRODUCTION As schools focus greater attention on preparing students to meet Common Core State Standards (CCSS) with college- and career-readiness for all in mind, the need for meaningful and actionable data about students’ math performance during the pivotal middle school years is essential. To provide middle school students with the learning opportunities and support they need in math, educators must base their programmatic decisions on a clear understanding of students’ current math skills and students’ probable trajectory of achievement in math. Understanding whether or not students are on track to become college- and career-ready is vital if educators are to make timely, effective decisions to improve outcomes for all students. In particular, for populations that experience a persistent achievement gap in math, the types of actionable data available to schools may be of great consequence if students are to receive the educational opportunities and interventions that will help them succeed on par with other students. Despite past efforts, disparities in math performance persist between various student populations. Recent gains in math for female students have led to documented similarities in performance across gender (Lindberg, Hyde, Peterson, & Linn, 2010; Scafidi & Bui, 2010), yet scores for students eligible for English language development, free or reduced meals, or special education services consistently lag behind those of other students nationally (Hemphill & Vanneman, 2011; National Center for Education Statistics, 2011; Wagner, Newman, Carmeto, & Levine, 2006). 2 Although differences exist in how best to define and measure college- and career- readiness (ACT, 2011; Conley, 2007), large scale assessments such as the ACT used in college admissions decisions have served as one indicator of students’ preparedness for college. An understanding of how students’ current levels of achievement may indicate if they are on track to perform well on this type of measure may assist schools in making programmatic decisions to help students develop needed skills in math. Clearly, math preparedness is important for all students, but all the more so for student groups who may have additional challenges related to English language proficiency, poverty, or learning differences. If educational leaders are to understand how the current math skills of students in teachers’ classrooms may relate to the students’ future performance in math and use that information to improve programmatic decisions, they need readily available, technically adequate measures. Perhaps the most dominant form of teacher accessible measurement to appear in the past thirty years has been curriculum-based measurement, first described by Deno (1985). Many schools use CBMs as a formative assessment within a Response to Intervention (RTI) system (Fuchs, 2004; National Center on Response to Intervention, and others) to serve (a) as a universal screener to benchmark students’ performance and identify students at risk of low performance outcomes, and (b) as a progress monitoring tool to measure students’ growth over time. Within RTI, educators may use CBM data to help shape programmatic decisions based on students’ perceived need for support. Studies on the technical adequacy of CBMs have lent support to the measures’ use as one-point measures of student math performance and as a potential stimulus for changes in how math instruction is delivered to students. Evidence of connections 3 between CBMs and students’ later math performance may prove useful to schools as they work to track and to shape students’ achievement trajectory in math. The accessibility and technical adequacy of the measures may aid sound programmatic decision-making by allowing educators to make research-based inferences about students’ math skills and to target resources to students most in need of support. Furthermore, an understanding of the possible predictive nature of CBMs may serve to increase urgency among staff if they know that the students they are working with today may not be on track to be college- and career-ready unless greater action is taken on their behalf. Understanding students’ math performance levels and possible links between current performance and later math success may guide schools to bring resources to bear to alter the trajectory of students positively, particularly for students whose future performance may not be projected to meet college- and career-readiness benchmarks. The current study seeks to analyze the connection between CBM-math used as a universal screener and a large scale test used as one early indicator of college readiness, namely the EXPLORE test in Grade 8 that forms part of the ACT test series, EXPLORE- PLAN-ACT. The study can be understood as an extension of CBM validation research and as an additional source of information for educators striving to understand how students’ current level of math performance may relate to early indicators of students’ college- and career-readiness in math. Foundations for the Use of Curriculum-based Measures in Math Research in curriculum-based measures in math, though not nearly as extensive as that for curriculum-based measures in reading, provides a foundation for understanding the nature of math CBMs and their potential utility in schools. From early research on 4 math CBMs focused primarily on computation skills (Shinn & Marston, 1985; Tindal, Marston, & Deno, 1983) to later studies highlighting CBMs’ possible use to target instruction based on assessed needs (Deno, 1989; Fuchs & Fuchs, 1986; Shinn & Hubbard, 1992) to recent studies of CBMs as possible predictors of student performance (Anderson, Alonzo, & Tindal, 2010a; Helwig, Anderson, & Tindal, 2002), researchers have developed a base that informs CBM use in schools today. Indeed, Deno (2003) summarized over fourteen common uses of CBMs in schools, ranging from screening students to evaluating classroom interventions to predicting performance on high stakes assessments. Within RTI, schools use curriculum-based measures in math in two primary ways, (1) to benchmark student performance at designated points in the year and identify students at risk of low performance outcomes and (2) to monitor student progress over time. In recent decades, much has been written about CBM use in progress-monitoring, including schools’ use of CBMs to guide implementation of interventions (Calhoon & Fuchs, 2003; Phillips, Hamlett, Fuchs & Fuchs, 1993) and to help evaluate the effectiveness of interventions provided in RTI systems (Fuchs, Fuchs, & Compton, 2010; Fuchs, Mock, Morgan, & Young, 2003). The current study focuses on a different aspect of CBMs, namely CBMs as one-point measures to benchmark students’ performance and what those CBM results may indicate about students’ probable performance on later math assessments. Past research highlighting the technical adequacy of math CBMs provides a springboard for further investigations of the measures’ potential utility in schools. In addition, empirical evidence regarding the predictive relation between CBMs and other 5 standardized measures sets the stage for further research about the potential of CBMs to indicate students’ likely future performance. Understanding math CBMs’ capacity to measure students’ current performance, suggest future performance on later math measures, and help schools identify students most in need of assistance, can, along with an understanding of the possible limitations of CBMs, aid educators in making more informed decisions to shape student achievement. The Technical Adequacy of Math CBMs If measures of student performance are to serve as a basis for sound decision- making, they must be technically adequate. An understanding of the degree of validity associated with CBMs in math has emerged within the literature as researchers have worked to determine the accuracy with which CBMs reflect students’ competence in math and the utility of CBMs in instructional and programmatic decision-making. Before examining empirical evidence on the technical adequacy of math CBMs, it is useful to understand the connections between CBMs and the content domain of math to comprehend more fully the content aspect of construct validity (Messick, 1995) associated with math CBMs. Content domain of math and CBM-math development. The content domain of math includes a complex combination of related skills and content areas (Foegen & Deno, 2001). Numerous researchers acknowledge the inter-related nature of math skills and constructs (Christ, Scullin, Tolbize, & Jiban, 2008; Foegen & Deno, 2001; Fuchs & Fuchs et al., 2008; Keller, Shapiro, & Hintze, 2008; Kelley, Hosp, & Howell, 2008; Rutherford-Becker & Vanderwood, 2009, among others) and point out that math proficiency is composed of both conceptual understanding (i.e. the relations that underlie 6 math problems) and procedural knowledge (i.e. the rules and steps to solve the problems) (Hiebert & Lefevre, 1986; National Research Council, 2001). Because of the multi- faceted nature of the domain of math, researchers support assessment models that include multiple areas (Kelley et al., 2008), specifically, both computation and application (Thurber, Shinn, & Smolkowski, 2002). Indeed, as math skills increase in complexity and scope as students move into the middle grades, it is important that math instruction and math assessments, both for students in general education and those with mild disabilities, address the multi-faceted nature of the skills students are expected to demonstrate (Espin & Tindal, 1998). In recent years nationwide efforts to clarify the skills students are expected to acquire in the middle years have provided guidance to inform both instruction and assessment development. In 2006, the National Council of Teachers of Mathematics (NCTM) released the Curriculum Focal Points that provided curricular guidance for math instruction in pre- kindergarten through eighth-grade in the U.S. and served to define more clearly the skills that should be assessed within the domain of math in each grade level (NCTM, 2006). Subsequently, the Common Core State Standards in math, informed heavily by the NCTM Focal Points and adopted in almost all fifty states, have served to define the skills expected of students in each grade level in math with the goal of preparing all students for college or career after high school. For the middle grades, in particular, both the NCTM Focal Points and the CCSS that followed them have emphasized the importance of developing conceptual understanding, problem-solving, and algebraic concepts to prepare for algebra and the later complexities of high school math. For example, the Common Core State Standards in Mathematics emphasize that “students who have 7 completed 7th grade and mastered the content and skills through the 7th grade will be well-prepared for algebra in grade 8” (CCSSI, 2010a). Such guidance may help school leaders converse more clearly about what math skills today’s students may be expected to demonstrate in specific grade levels and may guide assessment developers as they seek to align measures with new standards. The degree to which CBMs reflect the content domain of math they are intended to measure has been an ongoing conversation in the literature, both before and after the emergence of NCTM Focal Points and the more recent Common Core State Standards. In brief, the development of curriculum-based measures in math over several decades has centered on two different methods, described by Fuchs (2004), that aim to reflect some of the math skills students of a given grade might typically be expected to demonstrate. In the curriculum-sampling method developers create CBMs by sampling from a given year’s math curriculum. Evidence exists that developing CBMs using material from other sources, other than those that reflect the specific curriculum taught within a school, can be used successfully to assess student skills (Deno, 2003), and many CBM developers use the robust-indicator method in which prompts are designed to represent overall proficiency in math (Foegen, Jiban, & Deno, 2007). CBMs serve as general outcomes measures (GOMs; Deno & Fuchs, 1991) that may allow inferences about students’ overall math content knowledge and skills at a particular grade level. Although the majority of CBMs has been designed for use in the elementary grades, middle school measures are more available now than in previous years. An examination of research on CBMs in the middle grades, the majority of which were generated using the robust indicator method, provides evidence of strong technical adequacy of math CBMs both in 8 the area of criterion validity and predictive validity and appears to strengthen, not weaken, the construct validity associated with the measures. Criterion validity of math CBMs. In their comprehensive review of math CBM studies prior to 2007, Foegen, et al. (2007) point out that, although reliability and validity coefficients for math CBMs are not as high as those for reading CBMs, the majority falls within the range of .50 to .70. Of particular interest in this study is research evidence in the recent decades regarding the criterion validity associated with math CBMs used as one-point measures of math performance in the middle grades. Several studies mentioned in the existing review by Foegen, et al. and information from more recent studies show that criterion validity of math CBMs used in the middle grades, in studies both with general education and with special education populations, appears relatively strong overall. Studies showing modest results include a study of a Grade 6 math CBM based on robust indicators that revealed criterion validity coefficients based on teacher ratings and rankings, student grade, student semester GPA and a standardized math achievement test (Iowa Test of Basic Skills, Hoover, Hieronymus, Frisbie, & Dunbar, 1993) between .45 and .66 (Foegen, 2000). Similarly, in a math CBM study involving 100 general education students in Grades 6-8 Foegen and Deno (2001) found that criterion validity coefficients varied from .29 to .63 based on teacher rating and a standardized math achievement test, with the majority within the .40-.50 range. Other studies have shown higher coefficients. For example, Fuchs, Fuchs, and Hamlett (1989), in a study of 40 students in grades 2-9 receiving special education services, found that a math CBM showed median criterion validity exceeding .80. A study of a math CBM for computation and for concepts and 9 applications taken by sixth grade students (N = 44) in general education showed criterion validity of .71 (Fuchs, Hamlett, & Fuchs, 1999, as cited in Foegen et al., 2007). Research involving a math CBM taken by 90 eighth-graders in general education and 81 eighth- graders identified for special education to measure understanding of math concepts revealed criterion validity of .80 and .61 respectively when student scores were compared to their performance on a computer adaptive state test (Helwig, Anderson, & Tindal, 2002). Similarly, an additional study involving an eighth-grade math CBM to assess math concepts and applications among general education students (N = 117) revealed high criterion validity of .81-.87 (Helwig & Tindal, 2002). Although studies included in the 2007 review by Foegen et al. showed a wide range of results, evidence on the whole indicates that student performance on math CBM showed moderate to strong associations with student performance on other standardized measures, results that appear to enhance the construct validity associated with CBMs. More recently, Foegen (2008) synthesized data on various CBMs designed to assess middle school students’ skill development in algebra. In a 2005-2006 study comparing the results of six CBMs with other assessments of student skill, including teacher ratings and math achievement tests, Foegen reported moderate validity coefficients for a computation CBM (n = 71-73; r = .35-.38) and for an estimation CBM (n = 120-126; r =.26-.51) administered to seventh-graders, but higher results for a CBM measure involving concepts and applications (n = 73-77; r = .73-.87). Subsequent work by Foegen to develop and analyze a CBM to assess student skill development in algebra showed that a CBM of basic skills revealed moderate criterion validity (n = 79-97; r = .55-.60) when compared with a state algebra aptitude test. A CBM developed as a robust 10 indicator around foundational skills in algebra and analyzed in relation to the same state algebra aptitude test showed slightly higher criterion validity of (n = 62-71; r = .57-.73), and a CBM developed from content analysis of algebra texts in use at the time showed similar criterion validity of .59-.73. Although Foegen’s study also investigated results for a subset of students with disabilities and showed slightly higher coefficients for that group, the small sample size (n = 9-15) makes it challenging to interpret the results or generalize to other settings. Using a much larger sample size, a study on a math CBMs for grades 3-8 designed around the NCTM Focal Points for each grade level showed strong concurrent validity in relation to state math achievement tests in Oregon and Washington (Nese et al., 2010). Results for the general population of seventh-graders involved in the study in Oregon were reported as N = 1,846 with r = .82 and, in Washington as N = 530 with r = .81. As researchers look beyond the elementary grades to understand more complex levels of student math performance, these studies in math CBMs provide an indication of moderate to strong criterion validity associated with math CBMs in middle grades, including CBMs that move beyond computation to address beginning algebra skills. Predictive validity of math CBMs. Empirical evidence provides an understanding of the predictive validity associated with math CBMs. Some prior studies have indicated support for computation CBMs in predicting math performance on applied math tests (Rutherford-Becker & Vanderwood, 2009; Fuchs, Fuchs, & Courey, 2005). Of particular interest in the current study is research that has delved into a possible predictive relation between CBM and large-scale math achievement tests. In a study of 171 eighth-grade students Helwig & Anderson et al. (2002) reported that a middle school 11 math CBM designed to measure understanding of math concepts (as opposed to procedural knowledge) was a predictor of student performance in math on a statewide achievement test both for general education students (r = .80; n = 90) and those receiving special education services (r = .61; n = 81). Several other researchers also have indicated that CBMs have potential to predict student performance on statewide tests (Anderson, Alonzo, & Tindal, 2010a; Jiban & Deno, 2007; Shapiro, Keller, Lutz, Santoro, & Hintze, 2006). Nese et al. (2010) reported that a math CBM in use in districts in Oregon and in Washington State was predictive of student results on the state assessment in math. In their three-district study in Oregon of students’ math performance on a tri-annual CBM and students’ performance on the Oregon Assessment of Knowledge and Skills (OAKS), scores of seventh-graders on the math CBM showed strong predictive relation with their results on the OAKS (R = .80; R2 and adjusted R2 = .64; n = 3,047). Anderson et al. (2010a) went a step further to examine the accuracy with which CBM math used as a screener in middle school was able to predict student performance on the state year-end assessment. Of particular interest to the current study are the findings in their ROC analysis for fall Grade 7 math CBM which showed Area Under the Curve (AUC) = .88, indicating that student performance on a grade 7 math CBM could be useful to schools as one possible predictor of student performance on the state math achievement test and as a means to help identify students at risk of low math performance. In summary, several studies indicate that the technical adequacy of math CBMs, although not as strong or as well researched as that of reading CBMs, is strong. The degree of criterion validity associated with math CBMs indicated in existing research 12 lends credence to the idea that math CBMs can provide solid information about aspects of student math achievement. In addition, evidence of the predictive nature of math CBMs, based on strong correlations with summative standards-based achievement tests and the documented accuracy of math CBM used as a screener to identify students at risk, indicates possible additional utility for schools. Research on the Use of CBMs for Instruction, Programmatic Decisions, and Projections Throughout the literature, research on CBMs has centered on aspects of the assessments that may enhance or detract from their utility in decision-making. Much of the prior research on CBMs’ utility in schools has focused on how data on students’ performance levels might inform school decisions about instruction, programs, and resource allocation (Deno, 2003). Initially designed to address the need for monitoring individual student performance within special education programs (Deno, 1989), CBMs long have been used to provide educators with actionable data to shape instruction to meet students’ perceived needs either at the individual or group level. For example, as CBM-based progress-monitoring within RTI systems aids in the identification of students in need of special education services, schools are obligated to respond by allocating resources (i.e. staff, materials, time, programs) to serve those students’ needs. However, schools have expanded the use of CBM data as educators seek to understand and influence student performance within classrooms, across a given grade level, or across the school or district (Gersten et al., 2009). Of particular interest in this study are CBMs as a one-point measure of student performance and the measures’ potential to provide 13 data to schools about students’ probable trajectory in math, information that may aid in sound programmatic decision-making. As schools use CBMs to benchmark student performance at designated points in the year, results may provide school leaders with a bird’s eye view of student math achievement levels across a school or district (Foegen & Morrison, 2010; Gersten et al., 2009). Such information, in combination with other data sources, may be useful in shaping resource allocation, program offerings, and program emphases as well as in planning short-term interventions. If CBM data also provide, as some of the literature indicates, an early indication of student performance on other, later measures of math achievement, that information may help schools understand students’ probable achievement trajectory in math and the need for timely action to assist all students. CBMs used as a screener early in the school year can help identify students who are not on track to meet grade level standards assessed by summative tests (National Council for Response to Intervention, 2010). For example, schools may use the predictive nature of CBMs to establish a CBM cut score (e.g. a score representing the 20th percentile) to help identify students considered at risk of low performance outcomes on year-end summative tests or may develop a series of cut scores to help identify categories of perceived risk (i.e. below 20th percentile = high risk, 20th-40th percentile = some risk, 40th-60th percentile = low risk). Following current recommendations for RTI (Fuchs, Fuchs, & Compton, 2012; Gersten et al., 2009), schools may follow up with additional screening, diagnostic assessments, or progress monitoring of students identified at risk to improve accurate identification of struggling students. Accurate early identification of students at risk of low achievement outcomes can help schools understand overall math 14 performance across a grade level, provide needed interventions, and allocate resources (i.e. time, funding, focused instruction) to assist those students most in need. In short, as Foegen and Morrison (2010) point out CBM data may be “used to identify students who may be at risk of not achieving proficiency on high-stakes achievement tests and who can then be targeted for intensive instruction” (p. 97). In prior years, those high-stakes achievement tests generally referred to state assessments used to determine students’ achievement of grade level standards and to measure schools’ performance. In the current era in which school accountability and funding is linked to their adoption of college- and career-readiness standards and assessments (U.S. Department of Education, 2010), understanding the relation between CBMs and assessments linked to college- and career-readiness standards may assist schools in understanding whether or not students are on track to meet those standards and allocating resources accordingly. For those students who appear at risk of not meeting college-and career-readiness standards, additional information that leads to action on their behalf may make an important difference in their future achievement. Further research on this aspect of CBMs is needed. As the nation’s focus shifts from state-generated summative assessments used in the No Child Left Behind era (NCLB, 2002) to shared, standardized measures of college- and career-readiness related to the Common Core State Standards, connections between student performance on CBMs and their performance on large-scale national measures linked to college- and career-readiness may be useful. Furthermore, as schools across the nation look at multiple sources of data to make well-informed choices about resource allocation for the benefit of students, additional studies on the relation between various measures of student math 15 skills can prove insightful for educators. Indeed, research along these lines may assist school leaders in building a sense of urgency around students’ math skills and in understanding what timely, programmatic decisions may best serve students. Information about the possible predictive nature of CBM from this and other studies may help educators to build a more complete picture of students’ current and future performance and to stimulate action on behalf of students, particularly those found to be at risk of low achievement in math. The Importance of Understanding Middle School Students’ Math Achievement Trajectory Math skills have served increasingly as a focal point for schools across the nation as they struggle to ensure that all students are on track to build the skills they need for college and career. Interest in raising math achievement across the nation continues unabated with the emergence of the Common Core State Standards aimed at raising expectations for students across the board and improving the percentage of students who are prepared for life after high school. Although slight improvements appeared in student performance on the 2011 National Assessment of Educational Progress (NAEP; National Center for Education Statistics, 2011) compared to previous years, only 35% of eighth- graders performed at a proficient level in math, and an even lower percentage, 26%, of students performed at a proficient level in twelfth grade (Aud et al., 2011). In addition, the data highlighted the persistent achievement gap present for specific subgroups at risk of low performance outcomes. The average score of 284 for eighth-graders on the 2011 NAEP math assessment fell below the proficient range, but still within the basic range for math performance. Students receiving free or reduced lunch scored 27 points lower than 16 their peers, placing them on the low end of the basic range. Students eligible for special education scored 38 points lower than their non-eligible peers, and students eligible for English Language development scored 42 points lower than non-ELD students. Neither group’s average score fell within the basic range. Low levels of math achievement across the nation have heightened the sense of urgency in getting students college- and career-ready, and low math achievement can limit students’ opportunities in education significantly (Hanich, Jordan, Kaplan, & Dick, 2001). Lack of readiness in math may affect (a) the amount of instructional time teachers spend re-teaching skills, (b) student access to more advanced coursework in math and science, and (c) student potential for being well-prepared for life after high school (ACT, 2007). For example, students proficient in math earn approximately 38% more than students who lack math proficiency (Riley, 1997, as cited in Clarke & Shinn, 2004). In the effort to improve students’ preparedness for life after high school, understanding early whether or not students are on track in math to be well prepared for college and career is essential if schools are to structure math programs effectively and target limited funds to provide needed assistance. College- and career-readiness in math. Exactly which skills students need if they are to be successful in college and careers after high school continues to be a source of debate. Although a variety of definitions for college- and career-readiness have been proposed, Conley (2007) defines the term college-readiness as “the level of preparation a student needs to enroll and succeed—without remediation—in a credit-bearing general education course at a postsecondary institution that offers a baccalaureate degree or transfer to a baccalaureate program.” (p. 5). Similarly, ACT indicates that college- 17 readiness is “the acquisition of the knowledge and skills a student needs to enroll and succeed in credit-bearing, first-year courses at a postsecondary institution (such as a two- or four-year college, trade school, or technical school) without the need for remediation.” (ACT, 2010, p. 1). Although Conley’s conception of college-readiness encompasses dispositions and skills beyond content knowledge alone, the focus of the present study rests specifically on quantifiable data about students’ mathematical skills and knowledge rather than the attitudes and dispositions that may also contribute to student success in later coursework. Consequently, college- and career-readiness mentioned herein is defined as achieving a research-based college-readiness benchmark score on the ACT, as described later in greater detail. Measures of student preparedness for life after high school are undergoing a profound shift across the nation. School accountability for student performance as mandated by No Child Left Behind (NCLB) once hinged on results of state achievement tests, often created by individual states, leading to varying degrees of reliability and validity across the board and complicating state-to-state comparisons. The adoption of the Common Core State Standards (CCSSI, 2010a) in math by almost all fifty states has stimulated the development of two major co-existing assessment systems designed to assess those standards, including (a) a CCSS math assessment created by the Smarter Balanced Assessment Consortium (SBAC, 2012) and (b) a CCSS math assessment created by the Partnership for Assessment of Readiness for College and Careers (PARCC, 2012). Those assessments promise to form the core of state accountability as schools across the nation shift from individual state achievement tests to assessments shared across multiple states and focused on a common set of more rigorous standards. 18 Schools that previously sought to understand links between grade level formative assessments and state summative achievement tests may find it increasingly important to understand links between grade level formative assessments and large-scale nationwide achievement tests, particularly assessments that have college- and career-readiness as their focus. One such test, the ACT, provides an existing source of empirical data on student math performance to inform educators’ understanding of college- and career- readiness, at least in the area of content knowledge and skill. Indeed, ACT and college- readiness research conducted by ACT, Inc. in recent decades informed the development of CCSS (ACT, 2010) and of the SBAC and PARCC assessments. Although the assessments developed by SBAC and PARCC will differ from the ACT and from each other, understanding the relation between CBM-math and existing national standardized instruments such as the ACT test series has the potential to provide useful data to schools at a time when the focus is shifting toward preparing students for college and career. With college- and career-readiness by the end of high school as a goal, schools are focusing increasingly on what data and programmatic decisions may be necessary to help shape students’ achievement trajectory in math. In recent years, research has highlighted the key role that middle school math preparation plays in preparing students for later success in math. By understanding the pivotal nature of middle school math for students’ later math achievement and the relation between students’ current math performance and their probable future performance, school leaders can comprehend the likely trajectory of student math achievement and take action to shape that trajectory. The pivotal nature of middle school math achievement as it relates to college- and career-readiness. By the time students move beyond elementary school, “academic 19 deficits are well-established” (Fuchs et al. 2010, p.24), making their skill development in middle school all the more important. Studies point to middle school’s key role in shaping students’ math achievement trajectory toward college- and career-readiness (Terry & Rosin, 2011). Skill development, coursework, and test results in middle school may all play a role in high school math experiences. For example, it appears that students who struggle in seventh grade often have continued difficulties in math through their high school years, and eighth-grade math often serves as a gateway to more advanced learning in math and may shape student access to certain science courses (Wang & Goldschmidt, 2003). Math skill development and assessment in the middle years pose ongoing challenges. Students’ skill development in math often is linked to their completion of specific math coursework. Quite simply, students gain access to the curricula they need for further skill development in math primarily through access to the coursework where teachers address those skills. Although taking a particular math course is not a guarantee that students will develop the skills taught, it is unlikely that most students will develop higher-level math skills without taking a course. For example, as would seem logical, students who took higher-level math courses performed better on the 2009 NAEP than did students who had not taken higher-level math courses (Aud et al., 2012). Schools gauge measure student math preparedness district wide not only on student performance on math achievement tests but also on student completion of specific coursework, with a focus on key courses that stand out as benchmarks along the way. Because completion of Algebra 2, in particular, appears to be key in developing the math skills helpful for success in college math coursework (ACT, 2010; National Mathematics 20 Advisory Panel, 2008), much of the focus on preparing students for college-level math has centered traditionally on ensuring completion of a core curriculum of Algebra 1, Geometry, and Algebra 2 at minimum. More recently, educational leaders point to completion of Calculus prior to high school graduation as a necessity for students if they are to be prepared for advanced study leading to careers in math and science (CCSSI, 2010a). Planning backward to determine what course pathways may lead to student preparedness in math for high stakes assessments (i.e. college admissions tests, PARCC, SBAC) and life after high school, many districts have pushed to implement Algebra 1 programs in eighth grade to ensure that all students are on track to develop advanced math skills in high school. Indeed, one third of the eighth-grade students who took the NAEP in 2011 were enrolled in Algebra 1 (NCES, 2011). Increasing numbers of Grade 8 students in Algebra I became a goal for schools in some states such as California with mixed results (Liang, Heckman, & Abedi, 2012), depending on students’ prior preparation. Students in that state placed in Algebra I without sufficient prior skill development performed poorly on the state assessment. Again, it is clear that taking the course, in and of itself, does not guarantee skill development, and that adequate pre- requisite skill development is paramount if students are to be successful in later algebra coursework. An understanding of middle school students’ math skills is essential for sound programmatic decision-making, both small- and large-scale, to avoid missteps and to benefit students. Students are unlikely to develop higher-level math skills without access to higher-level coursework, and students need sufficient preliminary skills to handle higher levels of math. Students identified early in the year as struggling within a 21 particular level of math may benefit from interventions (i.e. additional time, additional concurrent coursework, focused skill development, small group instruction) and potentially avoid repeating the course and falling further behind in their preparation for life after high school. For others, the additional skill development they may experience as a result of supports put in place on their behalf can strengthen their understanding of math and contribute to success within the course and on later assessments. Although the recent Common Core State Standards outline additional pathways and course configurations besides the traditional Algebra 1, Geometry, Algebra 2 path to prepare students for college and career in math, the common emphasis is on introducing algebraic concepts, problem-solving skills, and rational number understanding much earlier than eighth grade in order to prepare students well for skills in algebra and the conceptual understanding they are expected to acquire before high school if they are to be on track for college- and career-readiness (CCSSI, 2010a). Student skill development in middle school can be key to their access to and success in more advanced math coursework and eventual college- and career-readiness in math. This may be particularly true for students who are English learners and students with disabilities; those groups completed Algebra 2 by twelfth-grade at lower rates than their peers. 2009 NAEP results indicated that, whereas 76.2% of students not identified as English learners completed Algebra 2 in high school, only 58.1% of English learners did. Similarly, whereas 79.1% of nondisabled students completed Algebra 2 during high school, only 39.5% of students with disabilities completed that level of math course (Aud et al., 2012). For students at risk of not being on track to be college- and career-ready in math, tools that help schools 22 understand students’ current and likely future probable math achievement early enough to provide intensive interventions on their behalf are essential. Middle school math assessments and college- and career-readiness in math. The importance of middle school math achievement to students’ future math success has highlighted the need for schools to be able to assess not only students’ current level of performance but also their probable future performance on high stakes math measures. Indeed, as stated explicitly in the Common Core State Standards, “[b]ecause important standards for college and career readiness are distributed across grades and courses, systems for evaluating college and career readiness should reach as far back in the standards as Grades 6-8” (CCSSI, 2010b, p. 84). Increasingly, schools rely on both formative and summative math assessments to provide information about instructional and program effectiveness and to inform instructional changes and programmatic decisions. Many schools find value in seeking multiple data points to develop a more complete understanding of student achievement than what is available through a single year-end test. As schools seek to understand student progress toward college- and career- readiness, the links between formative and summative assessments that form a data pathway for understanding student progress toward college- and career-readiness may become increasingly important. Schools can document math performance via assessments in several ways. First, end of year summative state achievement tests provide educators a single data value in making decisions about student performance and math program effectiveness. These tests often have formed the core of accountability measures aimed at assessing school and district performance. Teachers often use in-course assessments; either generated by 23 publishing companies to accompany published math programs or created by teachers themselves. However, these in-course assessments quite often lack the technical adequacy, efficiency, or sensitivity to change in student performance over time (Kelley et al., 2008) needed to support their use for decisions about student performance and placement, and they may vary substantially from classroom to classroom, complicating comparisons of student performance. For sound programmatic decision-making, educational policy makers recommend that schools rely on research-based, technically adequate measures to provide both formative and summative information about student performance in math. Having multiple sources of data about student performance and understanding connections between formative and summative standardized measures widely in use in schools may assist educators in early identification of students at risk and inform their programmatic decisions. One national standardized test of math achievement commonly used in college admission requirements is the American College Test (ACT) developed by ACT, Inc. Approximately 52% of all 2012 high school graduates had taken the ACT in their high school years (ACT, 2012b). ACT has established college-readiness benchmarks that provide one possible indicator about whether students are on track to enter college without needing to take remedial coursework in specific subject areas. To develop those benchmarks ACT examined student levels of success in first year college coursework and determined that students who met the ACT college-readiness benchmark in math (22 out of a possible 36 points) had a 75% chance of achieving a C or higher in their first year College Algebra course and a 50% chance of achieving a B or higher (ACT, 2011). ACT 24 scores, then, may provide one possible measure of college-readiness, at least in the realm of subject area content and skills. Results of 1.67 million high school graduates in 2012 nationwide who had taken the ACT showed that only 46% met the ACT benchmark for college- and career- readiness in mathematics (ACT, 2012b). The information can be useful for schools in providing a general picture of graduates’ math preparedness and raising the sense of urgency among educators, but earlier research-based indications of student performance are vital for schools that hope to improve students’ preparedness in math. If schools are to take action to improve students’ skills, they need data on all students--not just known graduates--and they need it early enough to make a difference. The EXPLORE test, typically administered to students in Grade 8, represents the first in a series of three national standardized assessments developed by ACT, Inc. of student achievement in the areas of English, Math, Reading, and Science. Together, these assessments, the EXPLORE, PLAN, and ACT tests, form the Early Progress Toward Academic Success (EPAS) system, and may provide a data pathway for schools to track student college-readiness as students progress through grades 8-12. ACT has established college-readiness benchmarks, detailed in Table 1, using 150,000 records of students who had taken all three assessments. (ACT, 2011a). Schools have the option of administering the EXPLORE to Grade 9 students, but college-readiness benchmarks are established using Grade 8 data. 25 Table 1 College-Readiness Benchmarks for EXPLORE (Grade 8 and 9), PLAN (Grade 10), and the ACT (Grade 11 or 12) Test EXPLORE 8 (1-25) EXPLORE 9 (1-25) PLAN (1-32) ACT (1-36) English 13 14 15 18 Math 17 18 19 22 Reading 15 16 17 21 Science 20 20 21 24 Source: ACT, 2012c Results from a 2005-2006 study indicated that only 35% of students who took the EXPLORE in math achieved the college-readiness benchmark of 17. The implication is that only about 1 out of 3 middle school students were on track to be college-ready in math. (ACT, 2008). Awareness of these early indicators of student college-readiness may serve to increase the sense of urgency within schools as educators develop a better understanding of how their middle school students’ current levels of performance may relate to predicted levels of success on later assessments. Additional research indicates that students who do not meet the early indicator of college readiness in Grade 8 may continue to struggle significantly in high school and may not meet later indicators of college readiness (ACT, 2008). In the ACT study, being on track to perform well on the EXPLORE was a greater predictor of eventual college and career readiness than other factors addressed in the study taken separately, including, (a) maintaining a B average, (b) earning higher grades, (c) taking a core curriculum, (d) taking additional standard courses, or (e) taking advanced coursework. In short, students who were not ready to achieve high results on the math achievement test by Grade 8 were 26 less likely to meet later indicators of college readiness than their more prepared peers, despite a variety of efforts at the high school level. It is important to note that the study did not address the effects of efforts at remediation, such as specific programs of intervention in math, and therefore may present only a partial picture of students’ likely outcomes. It appears clear, however, that middle school continues to be a time of critical importance in in shaping students’ possible trajectory of math achievement. Schools’ understanding of the importance of middle school math achievement, students’ current math skill level, and students’ probable math achievement trajectory may shape programmatic decision-making as schools strive to improve students’ math achievement in these key years of skill development. However, results from many large scale math achievement tests available to schools often appear at a time far removed from the actual instruction that preceded the test and may be provided to instructors and school officials who have no direct relationship with the students who completed the assessment. The information from those summative tests, albeit useful, provides one piece of the constellation of data that schools need to be able to work effectively with students. A predictive relation between technically sound, teacher-accessible measures of math performance, such as CBM-math, given during the school year and larger standardized math achievement tests such as EXPLORE may prove useful for schools trying to get an early indication of students’ math achievement trajectory. Helwig et al. (2002) point to a number of reasons why identifying predictors of student achievement on large scale math assessments is important, including the perspective that CBMs: (a) may provide more frequent information about student achievement than is available through annual statewide tests, (b) may help provide 27 instructors with a more complete picture of student math achievement than is available from topic-based classroom assignments and assessments alone, (c) may help identify students receiving special education services who can participate in general education settings, (d) may help identify general education students in need of greater support, (e) may provide early notice of students who are not on track to meet state math requirements and allow for intervention, and (f) may supply evidence schools need of adequate yearly progress. The current study seeks to ascertain if a connection of this type exists that may help inform school leaders’ understanding and decision-making in math. Gaps in the CBM Research and Warrants for a Study on the Relation between CBM and College- and Career-Readiness Clearly, although much is known about math CBMs, the area remains fertile ground for continuing research. In particular, the relatively low number of studies on math CBMs at the middle school level and the challenges of inferring math performance from a single measure remain problematic areas in the research. In their 2007 review of the literature, Foegen et al. pointed to the dearth of research on CBM math in the middle school and the sheer absence of CBM-math research at the high school level. A year later, Calhoon (2008) pointed out that only one study (Calhoon & Fuchs, 2003) addressed math CBM for grades beyond middle school. Although focused primarily on CBMs in progress-monitoring, Foegen’s (2008) call for greater research into middle school math measures involving studies (a) with larger sample sizes, (b) of more diverse populations, (c) about the effectiveness of teachers’ use of CBM data (d) involving the use of CBMs with students with disabilities, pointed to the overall need for greater breadth in CBM research at the middle level. Since, then some additional studies and recommendations 28 have emerged (Foegen & Morrison, 2010; Gersten et al., 2009, Nese et al., 2010) on middle school CBMs, adding to the research base. Nese et al. (2010), in particular, included grades 3-8 in their examination of a CBM that included both computation and application and investigated the predictive nature of CBM for different racial and ethnic subgroups. However, there is a continuing need for research on math CBMs in the pivotal seventh-grade year, particularly with measurements that extend beyond computation alone and include algebraic concepts, conceptual understanding, and problem-solving skills that students are expected to develop more fully in the middle years. Similarly, although some empirical evidence exists to indicate a relation between student math performance demonstrated on CBMs and their later performance on other math measures, additional empirical evidence on correlations and a possible predictive relation between student performance on math CBMs and their performance on standardized assessments of math achievement may help educators draw early inferences about students’ potential to perform well on high stakes math assessments. As attention turns nationwide to college- and career-readiness, information about the relation between math CBMs and research-based, national measures of college- and career-readiness can prove of particular interest to schools. Yearly standardized tests may provide information useful to schools, but one advantage that CBMs may hold for moving schools forward is the immediacy of the information provided to teachers and the fact that repeated administrations allow instructors to see the result of their actions more readily than other measures might. It stands to reason that the information may have a greater impact on teacher and school behavior if the results are understood in terms of their relation to a large-scale assessment 29 such as EXPLORE. If schools are aware of a predictive relation between student results on CBM and students’ probable college- and career-readiness in math it may support their efforts to identify students early who need intervention and to provide for their needs. A possible predictive relation between CBM and EXPLORE, then, may increase school understanding of student preparedness, may increase the sense of urgency within schools regarding the need for changes, and may inform programmatic decision-making. Accurate information and the impetus to act on that information is particularly important for schools as they work to meet the needs of student subgroups who may be further from being college- and career-ready than some of their peers. Further research involving diverse student populations and large sample sizes is needed. Research on the possible relation between math CBMs and performance of various student subgroups on later standardized math assessments, such as EXPLORE, may help school leaders understand what inferences, if any, educators can draw from CBM results about students’ potential future performance. Such information could aid schools in making programmatic decisions in math and allocating resources wisely, particularly for students who appear to be at risk of underachieving in math. Clearly, there is an ongoing need for research-based, technically adequate measurements with high teacher-accessibility if schools are to take timely action to improve student performance. Considering the complex domain of math, few would argue for the use of math CBMs as the only data source for understanding student math proficiency, and it must be acknowledged that the design of CBM as an instrument for benchmarking and progress-monitoring may limit its utility as a predictive assessment linked to EXPLORE since such was not its original purpose. The sample of items within 30 a given CBM may not provide as close an alignment with other math achievement assessments than might be the case if CBMs were designed initially as predictive instruments for those assessments. Results from math CBMs, then, may provide a partial picture of students’ overall math competence (Christ et al., 2008; Helwig et al., 2002), but an important piece of the picture nonetheless, and prior research already has established CBMs’ potential predictive nature (Helwig et al., 2002; Shapiro et al., 2006 and others). Delving further into the possible predictive nature of CBMs already suggested in the research may strengthen the understanding of the measure’s potential utility for decision-makers or may highlight possible limitations. The present study extends the current understanding of CBM by examining its relation to a different standardized measure of math achievement and explores the possible utility of CBM results to screen for students at risk of not achieving on an early college- and career- readiness indicator. Summary of CBMs in Middle School Programmatic Decision-making to Shape Students’ Future Math Achievement Tracking math achievement in middle schools is important to ensure students become college- and career-ready in later years. In order to make sound decisions about program offerings, program effectiveness, and resource allocation, middle school educators must have accurate information about students’ current math performance. The use of research-based formative measures such as CBMs can assist educators in making these decisions about students’ needs with greater ease and accuracy than might be possible with less frequent or less technically sound measures. 31 Furthermore, an understanding of connections between student performance on math CBMs and their performance on large scale assessments linked to college- and career-readiness may help educational practitioners understand how students’ math performance in middle schools relates to their probable achievement of college- and career-readiness in math. Such data, obtained early in the school year through CBM used as a universal screener, along with other information about student math achievement, may allow educational leaders to identify students at risk of low performance outcomes, to make sound programmatic decisions, and to allocate resources wisely in the middle school years. Although many schools use CBM to measure school wide student performance at fall, winter, and spring benchmarking periods during the year, data obtained in the fall rather than later in the year may have the greatest utility. By obtaining data early educators have more months in the given year to implement interventions to influence student performance and schools may still have flexibility in the allocation of financial and personnel resources. Considering the pivotal nature of middle school math and the crucial role that algebraic concepts play in students’ preparation for later algebra curriculum, understanding student performance early in Grade 7 may be of particular importance if schools are to help students prepare for eighth grade and the challenges beyond. Information from a research-based, technically adequate measure such as CBM may provide schools with the data they need to provide intensive interventions where those are most needed and shape students’ trajectory in math. Effective decision-making using math achievement data is particularly critical for students who historically have struggled to achieve in math on par with their peers. Those students may be at risk of not 32 being well prepared for college and career unless more is done to provide adequately for their needs. In particular, students learning English, students receiving special education services, and students from socioeconomically disadvantaged backgrounds may benefit from the availability and wise use of actionable data. Previous research documents sufficient technical adequacy of math CBM, particularly through criterion-related evidence indicating medium to high correlations between CBMs and other measures of math proficiency. CBM has an established research base supporting its use in monitoring student progress and in instructional change. In addition, research is beginning to provide a rationale for the use of middle school CBMs not only in monitoring progress but also in developing an understanding of students’ likely future performance and in making programmatic decisions to improve students’ achievement trajectory in math. CBMs have been used previously as universal screeners to identify students at risk of not meeting established standards in math. However, as math standards shift nationwide, more research is needed on the use of math CBMs in grades beyond elementary school and possible connections between CBM results and those of other math measures. The current study represents an extension of the CBM validation research with new applications in middle schools and addresses a criterion measure related to college- and career-readiness not used previously in studies on CBM-math. The aim of this study is to address three questions: 1. What is the relation between Grade 7 fall CBM in math and student performance on the math section of EXPLORE in Grade 8? 33 2. How do student demographic characteristics of gender, free or reduced lunch, special education, and English Language Learner status interact with student performance on CBM-math and EXPLORE-math? 3. Does Grade 7 fall math CBM consistently identify the same students who are on track for college- and career-readiness as does the EXPLORE-math test in Grade 8? 34 CHAPTER II METHODOLOGY This study involved data analyses on extant datasets from two district-wide standardized assessments: (a) easyCBM-math (Alonzo, Tindal, Ulmer, & Glasgow, 2006), and (b) EXPLORE-math (ACT, 2011a). The study included separate analyses of each assessment and examined the relation of results of students who completed both of the assessments. The following sections describe (a) setting and participants, (b) curriculum, (c) procedures, (d) measures, and (e) data analyses. Setting and Participants The study took place in a school district in the Northwest covering a community of over 80,000 residents. The district serves over 16,300 students and has 28 schools, including 17 elementary schools, two K-8 schools, one charter school for Grades 6-8, five middle schools for Grades 6-8, and five high schools for Grades 9-12. In recent years, most of the eight schools that include middle grades (6-8) conducted the easyCBM and EXPLORE assessments in math. This study focused on student math performance within the six schools with middle grades that administered both the easyCBM and EXPLORE. More specifically, the study examined results of students who took the seventh-grade easyCBM-math in the fall of 2011 and who took the eighth-grade EXPLORE in the fall of 2012. The 965 students in the six schools included in the study represented 78% of the 1,235 eighth-graders enrolled in the district in 2012-2013. Although the district student population was much less ethnically diverse than the student population in many areas of the state and nation, the district had experienced a fairly dramatic increase in students whose families self-identified as Latino or Hispanic. 35 Table 2 outlines information on race and ethnicity of students in the six schools in the study. The total number of students (N = 965) included in the study from the six schools is somewhat lower than the 1,021 seventh-grade students represented in the table because some students moved out of the district between Grade 7 and 8 some students were excluded from the study, as is detailed later. Table 2 Grades 6-8 Student Race or Ethnicity at Participating Schools in 2011-2012 School Grades 6-8 Grade 7 White Hispanic Asian Black Multi- ethnic Alder -- 36 93.3% 3.3% 1.3% 0% .8% Birch 588 198 79.1% 16.3% .9% .7% 1.0% Cedar 877 292 90.6% 4.9% 1.5% .8% 1.9% Dogwood 728 238 84.8% 11.0% 1.5% 1.0% 1.0% Elm 168 22 84.5% 7.7% 0% 1.8% 1.8% Fir 724 235 85.8% 8.8% 1.9% .4% 1.4% Total 1,021 Source: Annual Measureable Objectives reports from state Department of Education and District Quick Facts In recent decades, the district also had seen an increase in students qualifying for English language development (ELD), although percentages remained significantly below state and national averages. Students in middle grades in the participating schools were from diverse socio-economically backgrounds. The proportion of students qualifying for the federal free or reduced meal program (F/RL) ranged from 25% to 70% 36 of the students across the schools included in the study. Qualification for free or reduced meals is used within the district and in this study as a proxy for students from families that are economically disadvantaged, but the number includes only those families who have applied and qualified for the federal program. Because some families with low income may elect not to apply, for various reasons, students from economically disadvantaged families may be underrepresented in the data. The percentage of students identified for special education services in middle grades across the district was approximately 14%, with percentages varying in the schools in the study from a low of 8.3% to a high of 26.2%. Table 3 reports percentages for specific student subgroups in each of the participating schools during 2011-2012. Table 3 Seventh-grade Student Subgroups at Participating Schools in 2011-2012 School Grade 7 F/RL SpEd ELD Alder 36 59.8% 19.7% .8% Birch 198 69.9% 18.9% 9.0% Cedar 292 24.5% 8.3% 1.8% Dogwood 238 47.9% 13.3% 4.8% Elm 22 54.2% 26.2% 3.6% Fir 235 43.5% 15.6% 2.3% Total 1,021 Source: Annual Measureable Objectives reports from state Department of Education and District Quick Facts from http://www.ode.state.or.us/data/reportcard/reports.aspx 37 Curriculum The district was in the process of transitioning to Common Core State Standards and recently had undergone an adoption of the Connected Mathematics Project (CMP) curricula and accompanying resources produced by Michigan State University that focused on four strands across Grades 6-8: 1) number and operation, 2) geometry and measurement, 3) data analysis and probability, and 4) algebra (Connected Mathematics Project, 2013). In addition, increased attention on math instruction had focused district efforts on intensive professional development activities such as math studio workshops for math instructors K-12 inspired by the Lesson Study model promulgated in Japan (Yoshida, 1998 as cited in Fernandez & Yoshida, 2004). In 2011-2012, middle school students used the district-adopted Connected Mathematics Project curriculum in Grades 6-8. The math program included a yearlong pacing calendar, which specified the number of days for each unit of math instruction and designated formative and summative assessments. Although the impact of specific coursework on student results is not the focus of this study, it is helpful to understand the type of coursework and curricula available to students during Grade 7 in order to provide a full context for this research. Table 4 reports coursework that seventh-grade students took in 2011-2012 across the district, using data drawn only from the six participating schools. Not all courses were offered at all schools, and the data do not include the relatively small number of students who may have taken a math course through an alternative learning option such as online coursework offered by the district or an alternative school. The majority, 90.2%, of seventh-grade students in the participating schools took the 7th-grade Math course 38 focused on the district wide adopted curriculum with a pacing guide. Of the roughly 10% of students who took a different course, 4.3% of students took a more rudimentary math course than 7th-grade Math and 5.4% took a more advanced math course. Table 4 Math Course Enrollment for Seventh-graders in the Participating Schools Math Courses Total 7th-graders % of 7th-graders Male Female Geometry 1 1 Algebra 1 8th-grade Math 55 5 31 24 Pre-algebra 7th-grade Math 913 90 465 448 General Math 34 3 17 17 Functional Math 9 1 6 3 All Math Courses 1,012 100 520 492 Note: Algebra I and Pre-algebra are courses available in the two schools not included in the study. Source: 2011-2012 District Middle School Math Enrollment Counts In two schools in the study a total of 40 students also were enrolled in a Math Lab course to provide support for their math studies in addition to their regular math course. On the whole, seventh-graders performed better in math in 2011-2012 than did their peers throughout the state. Across the district, 73% of seventh-grade students met or exceeded on the state summative achievement test in math, the Oregon Assessment of Knowledge and Skills (OAKS), compared with 63% of students across the state (District 39 Report Card 2011-2012). In addition to the state assessment, the district began using easyCBM in math and reading in all district middle schools in 2011-2012 to gather formative standardized assessment data in the fall and winter. School leaders were beginning to put RTI processes in place at the middle level, and the district had established a district wide cut score recommendation in 2011-2012 based on the 40th percentile on easyCBM to help determine which students were at risk of not meeting standards on the OAKS. Additionally, the district provided EXPLORE (Grades 8 and 9), PLAN (Grade 10), and ACT (Grade 11) testing free of charge to students in Grades 8-11 in an effort to gather data about students’ college-and career-readiness, to make programmatic decisions based on apparent student trajectories and program effectiveness, and to help students and parents understand more clearly whether students were on track to be ready for entry-level college coursework. Procedures Student scores were obtained from the fall Grade 7 math CBM used to benchmark student performance and from the EXPLORE-math assessment administered to Grade 8 students in October of the following year from the six schools included in the study. The assessments were administered to all students in the specified grade, with the exception of (a) students receiving special education services in the Life Skills program, (b) students in ELD levels 1 and 2, and (c) students who were absent the day of the assessment. easyCBM-math was administered by instructors, either in a computer lab or in classrooms via a mobile computer lab. The EXPLORE test was administered in the school cafeteria or in classrooms, depending on the decision of school administrators. 40 Training and administration for easyCBM-math. Minimal teacher training was required for administration of easyCBM-math because the assessment is computer-based and group-administered. A handbook provided protocols for administration of the measure and multimedia training modules were available online. Teachers needed to ensure that students understood the login procedure and could select the correct assessment from a drop-down list. Teachers were expected to provide students access to a pencil and scratch paper or other accommodations, make certain that students were focused on their assessment, and ensure that the test environment is quiet. Teachers had the option of printing out the assessment and having students complete it in a group setting, in which case the teacher was responsible for inputting the students’ answers into the computer for automated scoring (Anderson et al., 2010a). Training and administration for EXPLORE-math. Test administrators for the EXPLORE test were required to read the EXPLORE Supervisor’s Manual and adhere to detailed, standardized test administration instructions. For example, supervisors were expected to read instructions aloud verbatim to students and adhere to established time limits to help create a standardized testing environment. Middle schools modified the school schedule as needed to allow sufficient time for students to complete the EXPLORE test. Measures The two measures in this study included a computer-based easyCBM in math, used to establish a benchmark of each student’s math performance in fall of seventh grade, and the math assessment on the EXPLORE test in fall of eighth grade, used to 41 provide information about student math performance linked to later measures of college- readiness in math. Description of Grade 7 easyCBM-math. easyCBM is a computer-based assessment system designed to benchmark and monitor student progress. The assessments include universal screeners in fall, winter, and spring, and a series of progress-monitoring tests throughout the year that schools may use within RTI. Launched in 2006, online easyCBM was designed around the National Council of Teachers of Mathematics (NCTM) Focal Point Standards (Numbers and Operations, Measurement and Data Analysis, and Numbers and Operations and Algebra) (NCTM, 2006). With 45 multiple-choice items, the screening tool covers all of the Focal Point Standards addressed within three subtests: (a) Numbers Operations Algebra and Geometry, (b) Measurement Geometry and Algebra, and (c) Numbers Operations and Algebra. For each item, students choose the best answer from among three possible responses provided (see Appendix D). The 16-item progress-monitoring assessments, not used in this study, have 10 alternate forms focused on each of the three focal point areas (Anderson et al., 2010a). Only the fall CBM benchmarking tool, used as a universal screener, was used in the current study. Prior studies had established a possible benchmark score of 27 for fall 7th-grade easyCBM-math (Anderson et al., 2010a), and schools might use that research-based benchmark to infer which students were on track to achieve grade level standards on the state summative assessment. Alternatively, as was the case for the district in the current study, school districts may establish a district grade level benchmark each year based on locally normed data. In 2011-2012, the participating district had established a Grade 7 42 fall easyCBM-math benchmark of 29 to represent performance above the 40th percentile. Students who did not meet the benchmark were considered at risk of not achieving grade level standards as assessed by the state achievement test. Seventh-graders who scored 29 or above met benchmark and were considered to be less at risk and less in need of targeted intervention. The CBM assessment represented a sample of items, developed from the NCTM focal points, that incorporated algebraic and geometry concepts including both the computation and problem-solving skills that students may be expected to develop throughout the seventh-grade year in courses that are, themselves, focused on the NCTM focal points. In other words, rather than being designed specifically around curriculum taught in the district or around state summative assessment tools, it was designed around areas of national focus in middle level math. Reliability. Prior studies of fall easyCBM-math for Grade 7 have reported high reliability. For the overall student population, Cronbach’s alpha estimates for all three measures ranged from .89 to .90 and split half measures were .85 to .87. For ELD students, Cronbach’s alpha estimates were .60 to .80. Split-half reliability was .62 to .71. For students receiving special education services, Cronbach’s alpha was .86 to .87 and split-half reliabity estimates were .81 to .88 (Nese et al., 2010). Criterion-related validity evidence. The alignment of easyCBM with national and state standards during development (Alonzo et al., 2006) adds to the content validity associated with the measure. Fall easyCBM-math for Grade 7 appears highly correlated with the state standardized math achievement tests in Oregon and Washington, with correlations in the .80s. Nese et al. (2010) reported that easyCBM spring math assessment 43 in seventh grade in Oregon was predictive of seventh-grade Oregon state math test scores (R2 = .67; n = 1846; β = .82) and that results of easyCBM spring math assessment in seventh grade in Washington was predictive of Washington state math test scores (R2 = .66; n = 530; β = .81), information that strengthens the construct validity associated with easyCBM (Nese et al., 2010). easyCBM has certain design features that may improve inferences about student performance. Efforts to reduce construct irrelevant variance through adherence to principles of universal design during easyCBM development (Alonzo et al., 2006) may help ensure that students can access the assessment and that results provide a more accurate picture of student performance. For example, efforts to eliminate culturally biased language, reduce language complexity, and create a simple and accessible computer interface for students may make easyCBM more accessible for all students, particularly for those who are consistently low performing (Nese et al., 2010). The reduction of linguistic complexity in the measure may make it more accessible for students, yet also may make it distinctly different from some other measures of math, including EXPLORE, that present substantial linguistic demands on students. Of particular interest in this study is the potential predictive nature of easyCBM- math when analyzed in relation to later measures of math achievement. Predictive validity of seventh-grade fall easyCBM-math in relation to results of the same students on the seventh-grade OAKS was high (R2 = .64; n = 3,057; β = .80). Results of an analysis in Washington for easyCBM fall seventh-grade and the state summative assessment were similarly high (R2 = .65; n = 548; β = .81) (Nese et al., 2010). 44 Description of Grade 8 EXPLORE-math. EXPLORE is a standardized test in a three-test sequence in the EPAS system consisting of EXPLORE for Grades 8 and 9, PLAN for Grade 10, and ACT for Grade 11 or 12. Each test includes sections to assess English, math, reading, and science. EXPLORE-math is designed to measure students’ knowledge and cognitive skills related to math. The assessment is intended to provide information to schools and students that may assist them in planning further skill building opportunities for high school academic success and for life after high school (ACT, 2011a). Schools may administer the EXPLORE test any time between August and May. Reliability. Studies conducted by ACT (2011a) indicate high alternate form reliability of EXPLORE-math for Grade 8 as outlined in Table 5. Table 5 Estimated Reliabilities and Standard Errors of Measurement for Grade 8 EXPLORE Math Tests (2010) Reliability Form A Form B Form C Raw Scores Reliability .83 .80 .82 Scale Scores Reliability .76 .72 .74 SEM of Scale Scores 1.71 1.74 1.70 Content-related validity evidence. ACT test developers indicate that the content validity associated with EXPLORE is enhanced by the test’s focus on (a) problem- solving skills, (b) the knowledge and skills that students will need in further education and careers, and (c) significant areas of middle school and junior high school programs. According to the EXPLORE technical manual (ACT, 2011a), the 30-item, 30 minute EXPLORE-math test includes four cognitive areas: (a) knowledge and skills, (b) direct 45 application, (c) understanding concepts, and (d) integrating conceptual understanding. The development of the math items in the EXPLORE test was based on an analysis of published state objectives for instruction in Grades 6 through 9 at the time of test development, and consultation with educators in Grades 7 through the postsecondary level regarding the prerequisite skills and knowledge in Grades 6 through 9 considered to be essential for success in high school. Item developers are expected to adhere to guidelines to ensure fair portrayal of groups and the use of nonsexist language and to avoid subject matter that may be unfamiliar to some groups. ACT classifies the thirty test items into the four content areas of Pre-Algebra (10 items), Elementary Algebra (9 items), Geometry (7 items), and Statistics/Probability (4 items). The following descriptions drawn directly from the technical manual provide additional detail about the test content: Pre-Algebra. Items in this category are based on operations with whole numbers, integers, decimals, and fractions. The topics covered include place value, square roots, scientific notation, factors, ratio, and proportion and percent. Formal variables are not used. Elementary Algebra. The items in this category are based on operations with algebraic expressions. The operations include evaluation of algebraic expressions by substitution; use of variables to express functional relationships, solution of linear equations in one variable, use of real number lines to represent numbers, and graphing of points in the standard coordinate plane. Geometry. Items in this category cover such topics as the use of scales and measurement systems, plane and solid geometric figures and associated relationships and concepts, the concept of angles and their measures, parallelism, relationships of triangles, properties of a circle, and the Pythagorean theorem. All of these topics are addressed at a level preceding formal geometry. Statistics/Probability. Items in this category cover such topics as elementary counting and rudimentary probability; data collection, representation, and interpretation; and reading and relating graphs, charts, and other representations of data. These topics are addressed at a level preceding formal statistics. (p. 8) 46 For each item, students choose the best answer from among five possible responses provided (see Appendix D). The sample of items included in EXPLORE-math appears to provide a broader span of math skills than is present in the easyCBM-math. In addition, test items appear to incorporate significant language demands. Many of the items include a paragraph of text describing the math problem to be solved. The EXPLORE-math benchmark of 17 used in this study was developed by ACT based on the documented first-year college performance of 150,000 students who took EXPLORE in fall of Grade 8, PLAN in fall of Grade 10, and ACT in Grade 11 or 12 (ACT, 2011a). The aim of the study was to develop a series of benchmarks that reflect levels of achievement of students who are on track to meet standards of college- and career-readiness as established by the ACT, with the assumption that students at each level will continue to work hard and take challenging courses in the years leading up to high school graduation. Analyses The data initially gathered at the district level included a total of 1,235 students from all eight schools with middle grades, but was restructured to include data only from the six schools that had conducted both seventh-grade fall easyCBM in 2011-2012 and eighth-grade EXPLORE in 2012-2013. Because only those six district schools conducted both assessments in the years of the study, all students in the data file who had scores from both easyCBM and EXPLORE reported were included in the restructured data. These included thirty students for whom no school was indicated and five students who appeared to have an incorrect school (i.e. an elementary school) indicated, for a resulting N = 985. In summary, the data were restructured to include all students from the six 47 schools that conducted both assessments. Within those six schools, the data included all students for whom both scores, one score, and no scores were reported. Within the original data file, an additional 124 students did not have a school indicated. Ninety of those had only an EXPLORE score, but no easyCBM score. District personnel indicated that students with those characteristics in the data file generally were those who had moved into the district after the fall easyCBM benchmark was conducted. Because it is possible that a portion of the 124 students with no school reported were from the six schools in the study, I reviewed the data and determined that no pattern was apparent in gender, socioeconomic status, or race and ethnicity for students excluded. However, the excluded students appeared not to have ELD or SPED status reported, making it difficult to discern whether SPED or ELD students were over or underrepresented in the excluded results. Because students in levels 1 and 2 of ELD and students in Life Skills did not take the assessments, they do not appear in this study, and it is most likely that they were among the excluded students with no school and no scores reported. An a priori decision was made to exclude test scores of ‘0’ from the analyses. Only one student had a score of ‘0’ on easyCBM. The student was excluded from the analyses including easyCBM but was included in the analyses of EXPLORE. Within the group of 985 students from the six schools that conducted both assessments, 20 students did not have a score for either assessment and were excluded. Of the 965 remaining, 936 had scores for EXPLORE and 913 had scores for easyCBM (after removal of the single score of ‘0’). Eighty-nine percent (n = 885) of students in the study schools had scores for both EXPLORE and easyCBM. 48 Non-performance variables in the study included student number, gender, free or reduced lunch (F/RL), special education (SpEd), and English Language Development (ELD). Because the easyCBM and EXPLORE assessments used in the study center primarily on the students’ seventh-grade experience, functioning, respectively, as a formative assessment of student skills at the start of Grade 7 and a summative measure of student skills following Grade 7, the students’ F/RL, SpEd, and ELD status was drawn from the students’ seventh-grade year. Thirty students in the study had no F/RL status reported for 2011-2102, and their F/RL status for 2012-2013 was used instead. Twenty- nine participants (3.1%) who took CBM in Grade 7 did not have EXPLORE results reported. Of those, 9 were eligible for special education, 23 were eligible for free or reduced lunch, and none were eligible for ELD, resulting in a somewhat different demographic profile for the two assessments. easyCBM results included a slightly higher percentage of scores of students who qualified for F/RL than did EXPLORE results. Only students in English Language Learner levels 3, 4, or 5 participated in the assessments, and the ELD variable was recoded (ELD eligible = 1 and ELD not eligible = 0). Test results from the six participating schools included students identified for special education due to intellectual ability (n = 2), visual impairment (n = 1), communication disorder (n = 14), emotional disturbance (n = 6), other health impairment (n = 13), autism spectrum disorder (n = 10) and specific learning disability (n = 89). Although students receiving special education services vary greatly in their needs and their abilities, they all may face greater challenges in learning within a school setting and may warrant additional resources within schools to meet their needs. Because the study addressed differences in score associated with special education identification in general rather than 49 specific special education needs, the SpEd variable was recoded to indicate SpEd = 1 and non-SpEd = 0. Table 6 provides additional detail on coding of variables. Table 6 Non-performance Variables for Students in the Participating Schools Variable Name Description Coding Number Randomly assigned student number School School included in the study Alder Middle Birch Middle Cedar Middle Dogwood Middle Elm Middle Fir Middle Gender Male or Female 0 = Female 1 = Male F/RL Free or Reduced Lunch Eligibility based on family application and qualification for program 0 = Not eligible 1 = Eligible SpEd Special Education Eligibility 0 = Not identified 1 = Identified ELD English Language Development Eligibility (Levels 3, 4, or 5) based on student performance on the English Language Performance Assessment 0 = Not eligible 1 = Eligible 50 Table 7 provides demographic data of students in the study with scores reported. Percentages are rounded to the nearest whole number. Table 7 Demographic Data for easyCBM and EXPLORE Participants Assessment Total Gender F/RL SpEd ELD Male Female Yes No Yes No Yes No easyCBM 913 470 443 420 493 128 785 14 899 % of participants 51 49 46 54 14 86 2 98 EXPLORE 936 486 450 424 511 127 809 15 921 % of participants 52 48 45 55 14 86 2 98 Performance variables in the data file included Grade 7 fall easyCBM-math from 2011-2012 and Grade 8 math EXPLORE from 2012-2013. easyCBM was recoded into a new variable with two values (0 = meets; 1 = does not meet) to designate results that met the easyCBM fall district benchmark of 29. Similarly, EXPLORE was recoded into a new variable with two values (0 = meets; 1 = does not meet) to designate results that met the EXPLORE benchmark of 17. Table 8 details variable names, descriptions and coding. Table 8 Performance Variables’ Names, Descriptions, and Coding Definitions Variable Name Variable Description Coding easyCBM Result of 7th grade Fall easyCBM-math 1-45 continuous EXPLORE Result 8th grade EXPLORE-math 4-25 continuous easyCBM_BM Achievement of CBM benchmark of 29 0 = Meets 1 = Does not meet EXPLORE_BM Achievement of EXPLORE benchmark of 17 0 = Meets 1 = Does not meet 51 The statistical analyses conducted using the restructured, cleaned data file provided mean, median, and standard deviation of each measure and information about student performance in relation to established benchmarks on both assessments. I conducted a correlational analysis to determine the correlation coefficients between 7th grade fall easyCBM-math and 8th grade EXPLORE-math. Next, a simple linear regression was carried out using the two measures to look at a possible predictive relation between the earlier and later measure. After I conducted means testing to examine differences in performance of male and female students and of identified and non- identified students within various subgroups, each of the measures was regressed on demographic variables of gender, free or reduced meals, special education status, and English language learner separately. I conducted a simultaneous regression of results of each measure on the same demographic variables (a) gender, (b) free or reduced meals, (c) special education status, and (d) English language learner to determine which variable appeared to account for the greatest variance in scores. For EXPLORE, I repeated the analysis with easyCBM included in the regression and again with easyCBM removed from the model. Finally, to assess diagnostic efficiency of easyCBM, I conducted a Receiver Operator Characteristic (ROC) Analysis using easyCBM results with the 8th- grade EXPLORE benchmark as the outcome variable and calculated Area Under the Curve (AUC). The analyses provided information about the relation between the easyCBM-math and EXPLORE-math, the possible predictive nature of different demographic characteristics on student results, and the consistency with which easyCBM-math in seventh grade predicted student achievement of the EXPLORE-math college- and career-readiness benchmark in eighth grade. 52 CHAPTER III RESULTS Results of the statistical analyses detailed here provide information in answer to the three research questions in this study. First, descriptive statistics are provided for the performance variables in the analyses. Means testing provided information about overall student achievement of easyCBM and EXPLORE benchmarks. A correlational analysis and a simple regression analysis of the two performance variables, easyCBM and EXPLORE, yielded information about the association between easyCBM and EXPLORE as well as the possible predictive nature of easyCBM. Crosstabulations provided details about subgroup achievement of the district’s easyCBM benchmark used to identify students at risk of low performance outcomes on the state assessment and subgroup achievement of the EXPLORE benchmark established by ACT, Inc. Regressions of demographic variables on easyCBM and of demographic variables on EXPLORE were conducted separately and simultaneously to examine the possible relation between different demographic characteristics and student performance. Finally, a Receiver- Operator Characteristic (ROC) including Area Under the Curve (AUC) revealed the consistency with which easyCBM predicted achievement of EXPLORE benchmark in math. Cases Included and General Description Data in the study included results of all students with scores reported from the six schools that administered easyCBM and EXPLORE (N = 965). Both the easyCBM scores and the EXPLORE scores were normally distributed (See Appendix A), with a low 53 skewness and kurtosis. The distribution of EXPLORE scores showed a cluster of 12 participants with the maximum score of 25, indicating a possible ceiling effect on the exam. No such effect was observed in the easyCBM assessment. Descriptive statistics for easyCBM and EXPLORE are detailed in Table 9. Table 9 Descriptive Statistics for Grade 7 easyCBM-Math and Grade 8 EXPLORE-Math Measure n Min Max M SEM SD easyCBM 913 1 45 29.64 .26 7.90 EXPLORE 936 4 25 15.77 .11 3.23 The average score for participants was above the district easyCBM benchmark of 29 but below the EXPLORE benchmark of 17. Further examination of student scores showed that, among participating schools, only 37% (n = 346) of students who took EXPLORE met the EXPLORE benchmark of 17. I conducted means testing to examine results for males and females and for students eligible for F/RL, SpEd, or ELD, as reported in Table 10. For gender, the assumption of homogeneity of variance was violated for EXPLORE, but post hoc testing indicated that there was no statistically significant difference in means between males and females. There was no statistically significant difference in means for males and females on easyCBM either, indicating that gender had no discernible effect on student math performance on the two assessments. For both measures, differences in means for students identified for SpEd and their non-identified peers were statistically significant, 54 as were those for ELD students and non-ELD students. Difference in means between students eligible for F/RL and their non-eligible peers was statistically significant for easyCBM. Assumption of homogeneity of variances was violated in means testing within F/RL on EXPLORE, but post hoc testing showed significant results. Table 10 ANOVA for easyCBM and EXPLORE for F/RL, SpEd, and ELD Subgroup 95% CI n M Min Max SEM SD p LL UL easyCBM 913 F/RL Yes 420 27.15 2 44 .38 7.84 <.001 26.40 27.90 No 493 31.76 1 45 .33 7.32 <.001 31.12 32.41 SpEd Yes 128 23.70 2 43 .66 7.41 <.001 22.41 25.00 No 785 30.61 1 45 .27 7.55 <.001 30.08 31.14 ELD Yes 14 17.29 3 27 1.93 7.76 <.001 13.11 21.46 No 899 29.83 1 45 .26 7.76 <.001 29.33 30.34 EXPLORE 936 F/RL Yes 425 14.84 4 23 .15 3.02 <.001 14.55 15.13 No 511 16.53 7 25 .14 3.21 <.001 16.26 16.81 SpEd Yes 127 13.40 5 25 .27 3.04 <.001 12.87 13.94 No 809 16.14 4 22 .11 3.10 <.001 15.92 16.35 ELD Yes 15 11.73 9 16 .65 2.52 <.001 10.34 13.13 No 921 15.83 4 25 .11 3.20 <.001 15.62 16.04 In general, students eligible for F/RL, SpEd, or ELD services scored significantly lower than their non-eligible peers. It appeared that ELD students had the lowest scores 55 of any subgroup and scored, on average, approximately 12 points lower on easyCBM and 4 points lower on EXPLORE than the overall population of students in the study. Research Question 1: Relation Between easyCBM-math and EXPLORE-math The first research question addressed the relation between student results on easyCBM and EXPLORE. Because both variables were approximately normally distributed and a scatterplot (see Appendix B) indicated a likely positive linear correlation between the two measures, Pearson correlation was obtained to determine if a statistically significant association existed between the two measures. The results r(882) = .64, p < .001 indicated, as might be expected, that students who have higher results on easyCBM tended to have higher results on EXPLORE. Of interest is the strength of the correlation, which at .64 indicates a moderate positive association between the measures based on guidelines from Cohen (1988). Next, EXPLORE was regressed on easyCBM to determine a possible predictive relation between the two and the strength of that relation. Results reported in Table 11 were statistically significant, F(1, 882) = 624.34, p < .001. As anticipated from the Pearson coefficient obtained earlier, the R value was .64. The R2 value of .41 indicated that 41% of the variance in EXPLORE could be accounted for by variance in easyCBM. This medium effect (Cohen, 1988) indicated a moderate association between easyCBM and EXPLORE. Additionally,  =.264 indicated that for every increase of four points on easyCBM, a student might reasonably be expected to score one point higher on EXPLORE the following year. Results of the regression indicated that a moderate predictive relation existed between fall Grade 7 easyCBM-Math and Grade 8 EXPLORE-Math administered the 56 Table 11 Regression of EXPLORE-math on easyCBM-math Model Unstandardized Coefficients Standardized Coefficients 95% CI Β Std. Error Beta t p LL UL (Constant) 7.912 .325 24.31 <.001 7.27 8.55 easyCBM .264 .011 .644 24.99 <.001 .24 .29 following fall. The next research question examined whether certain demographic factors may be associated with variance in student math performance on the assessments and the size of that effect. Research Question 2: Relation Between Demographic Variables and Math Performance on CBM and EXPLORE Non-performance variables included in this study were gender, special education, free or reduced lunch, and English language learner. The second research question was designed to determine if specific variables accounted for some of the variance in EXPLORE results and easyCBM results and to examine the possible cumulative contribution of specific variables on the variance of easyCBM and EXPLORE results. To begin, I examined percentages of males, females, and students identified for F/RL, SpEd, or ELD who had achieved the district’s easyCBM benchmark of 29 out of 45 (i.e. students who scored at or above the 40th percentile that year and were not considered at risk) and percentages of those who had achieved the EXPLORE benchmark of 17 out of 25 as detailed in Table 12. Percentages are rounded to the nearest whole number. 57 Table 12 Subgroup Achievement of District easyCBM Benchmark and EXPLORE Benchmark Benchmark Gender Total Female Male F/RL SpEd ELD N % n % of females n % of males N % of F/RL N % of SpEd N % of ELD easyCBM 913 Met 559 61 269 61 290 62 202 48 35 27 0 0 Not Met 354 39 174 39 180 38 218 52 93 73 14 100 EXPLORE 936 Met 346 37 176 39 170 35 110 26 15 12 0 0 Not met 590 63 274 61 316 65 315 74 112 88 15 100 Although 61% of students overall met the easyCBM cut score, only 48% of F/RL, and 27% of SpEd met the cut score. The low number of ELD students in the study (n =15) did not provide sufficient power for a sound interpretation of some of the planned analyses because of the possibility of a Type II error. However, a brief review of the data showed that the 14 ELD students with easyCBM scores had an average score of 17.29 (SD = 7.23, SEM = 1.93), and all ELD results were below the easyCBM cut score. Of the 15 ELD students with an EXPLORE score, the mean score was 11.73 (SD = 2.53, SEM = .65), and none of the students achieved the EXPLORE benchmark. Although the small sample size precluded an accurate interpretation of the difference in means between students identified as ELD and those not identified as ELD, it seemed apparent from the data that ELD students in this study scored lower than their peers and struggled to meet benchmark on either measure. It seems unlikely that the perceived effect associated with 58 ELD was false. Students qualifying for F/RL or SpEd also met the EXPLORE benchmark at lower rates than their non-eligible peers. Regression of easyCBM on demographic variables. To understand a possible predictive relation between demographic factors and the 7th-grade math assessment, easyCBM was regressed on each of the demographic variables separately. Results of easyCBM regressed on gender were not statistically significant, indicating that the student’s gender did not appear to account for any difference in student math performance on easyCBM. Results of easyCBM regressed on F/RL were statistically significant. ANOVA results, F(1, 911) = 84.43, p < .001, indicated F/RL status contributed to variance in easyCBM results as detailed in Table 13. Table 13 Regression of easyCBM on Free or Reduced Lunch Unstandardized Coefficients Standardized Coefficients 95% CI Model Β Std. Error Beta t p LL UL (Constant) 31.77 .34 93.25 <.001 31.10 32.43 F/RL -4.62 .50 -.29 -9.19 <.001 -5.60 -3.63 It appeared that eligibility for free or reduced lunch was associated with an easyCBM score 4.6 points lower than that for non-eligible students. However, R value was .29 and the R2 value was .085, indicating that the effect attributable to F/RL eligibility was small, accounting for only 8.5% of the variance in easyCBM. 59 Regressing easyCBM on SpEd revealed similar results. Although the contribution of special education was statistically significant F(1, 911) = 92.53, p < .001, it accounted for only 9% of the variance in easyCBM score (R2 = .09, SEM = 7.53). Identification for special education services was associated with an easyCBM score almost 7 points lower on average than that of general education students as reported in Table 14. Table 14 Regression of easyCBM on Special Education Model Unstandardized Coefficients Standardized Coefficients 95% CI Β Std. Error Beta t p LL UL (Constant) 30.61 .27 113.85 <.001 30.08 31.14 SpEd -6.91 .72 -.30 -9.62 <.001 -8.32 -5.50 Finally, regression of easyCBM on ELD status appeared statistically significant, F(1, 911) = 68.07, p <.001, although results reported in Table 15 should be interpreted with caution due to low numbers in the analysis. Table 15 Regression of easyCBM on English Language Learners Model Unstandardized Coefficients Standardized Coefficients 95% CI Β Std. Error Beta t p LL UL (Constant) 29.83 .26 115.36 <.001 29.33 30.34 ELD -12.55 2.09 -.195 -6.01 <.001 -16.65 -8.45 60 ELD status was associated with easyCBM scores 12.55 points lower than non-ELD, yet the R2 value was .038, indicating that only 3.8% of variance in easyCBM was accounted for by ELD status. Regression of EXPLORE-math on demographic variables. Results of EXPLORE regressed on gender were not statistically significant, indicating that, as seen previously for easyCBM, students’ gender had little bearing on their math performance on the measure. EXPLORE regressed on free or reduced lunch was statistically significant, F(1, 934) = 68.16, p< .001. The R2 value of .068 provided evidence that free or reduced lunch eligibility may account for 6.8% of the variance in EXPLORE. Table 16 provides further detail. Table 16 Regression of EXPLORE on Free or Reduced Lunch Model Unstandardized Coefficients Standardized Coefficients 95% CI Β Std. Error Beta t p LL UL 1. (Constant) 16.53 .14 119.73 <.001 16.26 16.81 F/RL -1.70 .21 -.26 -8.26 <.001 -2.09 -1.29 Regression of EXPLORE on special education was also statistically significant, F(1, 934) = 85.78, p < .001. Results in Table 17 show that students in special education scored an average of 2.74 points lower on EXPLORE than their peers. The R2 value of .08 indicated only a small percentage of variance in EXPLORE could be attributed to SpEd eligibility. 61 Table 17 Regression of EXPLORE on Special Education Model Unstandardized Coefficients Standardized Coefficients 95% CI Β Std. Error Beta t p LL UL 1. (Constant) 16.14 .11 148.32 <.001 15.92 16.35 SpEd -2.74 .30 -.29 -9.26 <.001 -3.32 -2.16 EXPLORE regressed on ELD appeared statistically significant, F(1, 934) = 24.33, p < .001. Although students qualifying for ELD had an average score of 4 points lower on EXPLORE than their peers, R2 = .025 indicated that ELD status accounted for only 2.5% of the variance in EXPLORE. To understand further the way various demographic in combination might predict math performance, all were regressed on easyCBM and on EXPLORE to examine the possible cumulative effect. To understand the possible influence of demographic factors on student math performance, I conducted a simultaneous multiple regression of easyCBM on the demographic variables. Low correlations among the demographic variables indicated that collinearity was not an issue in the analysis. In the regression, results for gender were not statistically significant, and that variable was removed from the model with negligible difference in results. R2 for F/RL was .085 and the addition of SpEd to the model yielded R2 change of .072. The combined adjusted R2 for F/RL and SpED was .157, F(2, 910) = 84.77, p < .001, and the adjusted R2 was .18 for F/RL, SpEd, and ELD combined, indicating that 18% of the variance in easyCBM was accounted for by the three factors together, as seen in Table 18. 62 Clearly, identification for both free or reduced lunch and for special education attributed to more of the variance in easyCBM than did either factor alone. Although the Table 18 Simultaneous Regression of easyCBM on Free or Reduced Lunch, Special Education, and English Language Learner Model Unstandardized Coefficients Standardized Coefficients 95% CI for B Β Std. Error Beta t p LL UL 1. (Constant) 31.77 .34 92.25 <.001 31.10 32.43 F/RL -4.62 .50 -.29 -9.19 <.001 -5.60 -3.63 2. (Constant) F/RL, SpEd 32.38 -4.07 -6.16 .33 .49 .70 -.26 -.27 96.83 -8.37 -8.83 <.001 <.001 <.001 31.72 -5.02 -7.53 33.03 -3.11 -4.79 3. (Constant) F/RL SpEd ELD 32.38 -3.13 -6.22 -10.50 .33 .48 .69 1.95 -.23 -.27 -.16 98.33 -7.68 -9.05 -5.40 <.001 <.001 <.001 <.001 31.74 -4.66 -7.57 -14.31 33.03 -2.76 -4.88 -6.68 effect was small (Cohen, 1988), the results corroborate ongoing concerns in schools about students who may have multiple factors that place them at greater risk of not achieving than their peers. Although numbers of ELD in the analysis should lead to a cautious interpretation of the results, the combination of F/RL, SpEd, and ELD also appeared statistically significant, with a cumulative negative effect on student scores. Only three participants had all three characteristics. EXPLORE were regressed on easyCBM, F/RL, SpEd, and ELD simultaneously with cases excluded listwise to examine the possible predictive relation of those variables in combination and the cumulative contribution of non-performance variables on 63 EXPLORE. Again, in this analysis none of the predictor variables was highly correlated with another, indicating that collinearity was not a concern in the analysis. Residuals were normally distributed (see Appendix B). Gender was not included in the regression because prior analyses had shown it did not have a statistically significant effect. Table 19 reports results of the regression including easyCBM and the remaining demographic variables. Table 19 Simultaneous Regression of EXPLORE on easyCBM, Free or Reduced Lunch, Special Education, and English Language Learner Model Unstandardized Coefficients Standardized Coefficients 95% CI for B Β Std. Error Beta t p LL UL 1. (Constant) 7.91 .33 24.31 <.001 7.27 8.55 easyCBM .26 .01 .64 24.99 <.001 .24 .29 2. (Constant) easyCBM F/RL 8.47 .25 -.57 .37 .01 .17 .62 -.09 23.23 22.21 -3.33 <.001 <.001 .001 7.76 .23 -.91 9.19 .28 -.24 3. (Constant) easyCBM F/RL SpEd 8.88 .24 -.55 -.83 .38 .01 .17 .25 .59 -.09 -.09 23.17 21.50 -3.19 -3.31 <.001 <.001 .001 .001 8.13 .22 -.88 -1.32 9.64 .27 -.21 -.34 4. (Constant) 8.94 .39 22.94 <.001 8.17 9.70 easyCBM .24 .01 .59 20.98 <.001 .22 .26 F/RL -.53 .17 -.08 -3.10 .002 -.87 -.20 SpEd -.84 .25 -.09 -3.35 .001 -1.34 -.35 ELD -.53 .67 -.02 -.79 .43 -1.85 .79 R2 for easyCBM was .41 indicating that variance in easyCBM attributed to 41% of the variance on EXPLORE. R2 change for SpEd and for F/RL, although significant, were quite minimal at .007 each, indicating that only a very small amount of the variance 64 (.7%) in EXPLORE could be attributed to those demographic factors with easyCBM included in the analysis. The addition of ELD to the model had no effect. The same simultaneous regression with easyCBM removed showed R2 of .149, for F/RL, SpEd, and ELD combined, indicating that approximately 15% of the variance in EXPLORE scores might be attributed to the combined influence of F/RL, SpEd, and ELD. Without easyCBM in the model, ELD appeared statistically significant and contributed slightly to overall results, R2 change = .026. The final research question explored in greater detail the predictive relation between the easyCBM and EXPLORE, with an eye toward the measures’ utility in schools. How consistently did easyCBM scores predict student achievement of the EXPLORE benchmark in math? Research Question 3: Consistency of CBM Prediction of Achievement of Early College- and Career-readiness Benchmark on EXPLORE-math The participating district had used CBM results to establish categories of risk for students based on the predictive relation between easyCBM and the state math assessment. For example, students scoring below the normative benchmark of 29 in 2011-2012 were those who fell within the lowest 40th percentile on easyCBM and were considered at risk of low performance outcomes in math. The current study focused on the relation between easyCBM and the EXPLORE, an assessment generally considered to be more rigorous than the state assessment. How consistently did the district’s 7th grade fall easyCBM-math results predict student achievement of the early college- and career- readiness benchmark on the Grade 8 EXPLORE? 65 I conducted an ROC analysis (see Appendix C) to determine whether easyCBM results consistently predicted which students would meet or not meet the EXPLORE benchmark. Because the benchmark score for 8th-grade EXPLORE-math is 17, EXPLORE data were recoded so that values between 0 and 16 were coded as 1 = at risk and values between 17 and 25 were coded as 0 = not at risk. The ROC analysis was designed so smaller test values indicated stronger evidence for a positive actual state of 1. In other words, lower easyCBM scores were considered more likely to lead to a designation of at risk (i.e. not achieving the EXPLORE benchmark). Results in Table 20 indicated that 558 cases resulted in an accurate prediction of at risk (i.e. not meeting the EXPLORE benchmark) or not at risk (i.e. meeting the EXPLORE benchmark) whereas 326 cases resulted in an inaccurate prediction of at risk or not at risk. Table 20 Case Processing Summary for Grade 7 Fall easyCBM and Grade 8 EXPLORE EXPLORE at risk Valid N (listwise) Positive 558 Negative 326 Missing 101 To calculate the consistency with which easyCBM results predicted accurately that a student was at risk for not meeting benchmark on EXPLORE, I calculated Area Under the Curve (AUC) with a nonparametric assumption. A diagonal line with slope of .5 in the ROC analysis would indicate that the likelihood that easyCBM scores would indicate correctly that students would meet the EXPLORE benchmark was no better than 66 chance. A convex curve above the diagonal would indicate that the probability was better than chance. Consistency of prediction improves as AUC approaches 1.0, a statistic that would indicate a perfect prediction of meeting or not meeting on the outcome measure (EXPLORE). As reported in Table 21, the AUC was .86 with standard error of .013, 95% CI[.84, .89] and p < .001, results that contradicted the null hypothesis AUC = .5. The consistency with which easyCBM predicted correctly students who were at risk of not meeting the EXPLORE benchmark was notably strong. Table 21 Area Under the Curve Test Result Variable Area Std. Error Asymptotic Sig. Asymptotic 95% CI LL UL 7th Grade Fall easyCBM .86 .013 .000 .84 .89 The ROC analysis provided details on sensitivity and specificity related to easyCBM results in relation to EXPLORE results. Sensitivity indicates the true positive rate (TPR), which is the number of true positives (i.e. students not meeting EXPLORE benchmark who were identified as at risk based on easyCBM results) among all of the positives indicated within the analysis. In other words, sensitivity indicates the likelihood that a student predicted to not meet the EXPLORE benchmark based on his or her easyCBM result would fail to succeed on EXPLORE. Specificity indicates the true negative rate, or the proportion of students classified as not at risk who did indeed meet 67 the EXPLORE benchmark. Because schools often make use of an established cut score on formative assessments such as easyCBM to identify students at risk or not at risk for meeting math grade level standards on subsequent standardized assessments, I examined possible cut scores for easyCBM and the sensitivity and specificity statistics for each score to understand the consistency of the district’s easyCBM cut score and whether a different cut score might be a more consistent predictor of performance on EXPLORE. The 7th grade easyCBM-Math cut score used in 2011-2012 as a possible predictor of students’ preparedness or lack of preparedness for meeting standards on the state summative math assessment was 29. However, in the Coordinates of the Curve shown in this analysis (see Appendix C) the value of 28.5, which would translate to a cut score of 29 for practical purposes, showed sensitivity of .55 and specificity of .91 (1-specificity = .086). Based on that information, if the district were to use a cut score of 29 (representing those below the 40th percentile), only 55 out of every 100 students who did not meet the EXPLORE benchmark would have been identified accurately as being at risk. Table 22 Grade 7 Fall easyCBM-math and Grade 8 EXPLORE-math Crosstabulation EXPLORE-Math Test variable Does not meet Meets or exceeds Total easyCBM-math Does not meet 307 28 335 Meets or exceeds 251 298 549 Total 558 326 68 Very few (n = 28) students who did not meet the easyCBM benchmark went on to meet on EXPLORE the following year. Almost half (n = 251) of the students who met or exceeded the easyCBM cut score did not meet the EXPLORE benchmark. Summary To respond to the three research questions in this study, descriptive statistics were examined to provide an overview of results on easyCBM-math and EXPLORE math. A correlational analysis of easyCBM-math and EXPLORE-math results was conducted and a regression of EXPLORE on easyCBM revealed the possible predictive relation of easyCBM on EXPLORE. Each of the performance measures was regressed separately on each of the demographic variables of gender, F/RL, SpEd, and ELD and then in combination to determine how much of the variance in performance results might be attributed to different demographic factors. Finally, a ROC analysis was conducted, including Area Under the Curve to determine the consistency with which easyCBM-math results might predict achievement of the early college- and career-readiness benchmark in EXPLORE-math. 69 CHAPTER IV DISCUSSION This study investigated the relation between CBM-math and EXPLORE-math. With the intent of contributing to the research on CBM in light of an emerging focus nationwide on college- and career-readiness, I sought to (1) determine the association between student math performance on a Grade 7 math CBM and student performance on Grade 8 EXPLORE-math, (2) examine the predictive relation between the two assessments and the possible effect of different demographic variables on student performance, and (3) investigate the consistency with which CBM predicts which students may not attain the EXPLORE benchmark. In this section I present findings and limitations, interpret study results, and outline implications and areas for future research. Main Findings In terms of overall performance, slightly more than one third of eighth-graders met the Grade 8 EXPLORE benchmark of 17 in math, but almost two thirds did not. There was no statistically significant difference between male and female performance on either EXPLORE or easyCBM. However, students eligible for special education, for free or reduced lunch, or for English language development performed significantly lower on both measures than did their non-eligible peers. easyCBM showed a moderate positive association with EXPLORE, and a moderate predictive relation exists between the two measures. Special education status and free or reduced lunch status showed a slight predictive relation with easyCBM and with EXPLORE. ELD status appeared also to be slightly predictive of easyCBM and EXPLORE results, although low numbers prevent a 70 solid interpretation. Overall, easyCBM consistently predicted which students would or would not achieve the EXPLORE benchmark, but a higher CBM cut score than the one used in the district to identify students at risk of not meeting state standards appeared to improve identification of students who were at risk of not meeting the EXPLORE benchmark. Limitations I have controlled for various factors, but several limitations should be taken into account when interpreting the data from the study. These include, (a) subgroup identification, (b) differences in curriculum and instruction, (c) differences in the assessments, (d) operationalization of college- and career-readiness, and (e) statistical validity. Identification of F/RL, SpEd, and ELD. Because not all students with economic disadvantage apply for free and reduced lunch, it is likely that their true numbers are underreported in the study. Special education included only students formally identified for special education services, and did not examine differences between students with different types of disabilities. Students in ELD Levels 1 and 2 did not participate in the assessments district wide, and some students who qualify for ELD opt not to participate in ELD programs or choose to exit early from the program. Among participants in ELD levels 3-5, almost all were from families that self-identified as Hispanic or Latino. The study, then, provides only a partial picture of students with limited English proficiency and their math performance. Any interpretation of results is limited accordingly. Differences in math preparation. Although 95% of seventh-grade students took 7th-Grade Math, a course available district wide with specific curricular materials and 71 pacing guide, instruction still may vary from classroom to classroom. It is unknown whether instructors responded to the assessments by providing specific practice beforehand or by taking specific action after easyCBM results were available. In addition, 5% of students took a course other than 7th-Grade Math. Some students identified as needing additional math support participated in a Math Lab course in addition to their regular math course. Some students may have had access to ongoing supports other than Math Lab, including after school math help, regular help during lunch, online practice, peer tutoring, or tutoring at home. Differences in the assessments. Students take easyCBM online during a single class session in untimed conditions. Students take the timed EXPLORE-math test on paper as the second of four subject area assessments. An understanding of these differences and additional dissimilarities in the assessments should inform any interpretation of student math performance or the relation between the measures. Difference in linguistic load. EXPLORE requires substantial reading whereas easyCBM does not. Whether reading skill is considered an integral part of math competence or is viewed as construct irrelevant variance in the context of math assessments, it seems clear that differences between the linguistic load of easyCBM and EXPLORE may shape student performance and therefore require a cautious interpretation of results. Differences in content. Both easyCBM and EXPLORE focus on broad math knowledge and skills that correspond with nationally held standards of math performance rather than a specific curriculum taught in particular schools or districts. Both include basic algebra concepts, measurement, problem-solving, and computation, for example, 72 but the content of the assessments is not identical. EXPLORE includes sections on pre- algebra, elementary algebra, geometry, and probability and statistics. easyCBM groups concepts into (a) Numbers Operations Algebra and Geometry, (b) Measurement Geometry and Algebra, and (c) Numbers Operations and Algebra, content that appears to overlap, but not mirror exactly, the content of EXPLORE. Operationalization of construct of college- and career-readiness. EXPLORE is designed to reflect student content knowledge and skills in the area of math, but does not encompass the other skills and dispositions present in the literature on college- and career-readiness. EXPLORE, then, represents only one aspect of emerging college- and career-readiness and results should be interpreted accordingly. Statistical validity. Low numbers of ELD students in the study warrant a cautious interpretation of the results because power was insufficient in the analyses to avoid a Type II error. However, considering that all participating ELD students scored under benchmark on both measures, there appears to be little likelihood of accepting the null hypothesis (i.e. no effect or difference associated with ELD status) in error, but more research is warranted before firm conclusions can be drawn. Interpretations My study extended information provided by previous research regarding (a) validity associated with CBMs, (b) the predictive relation between CBMs and standardized summative math assessments, (c) the potential contribution of specific demographic variables to variance in math results, and (d) the consistency with which CBM predicted students at risk of poor performance outcomes in math. A brief review of 73 students’ performance on the outcome measure provides a context for understanding the study results regarding the relation between CBM and EXPLORE. Eighth-grade math performance. Results of the 2011 National Assessment of Educational Progress (NAEP) showed that only 35% of students performed at a proficient level in math (Aud et al., 2011), yet results on the statewide assessment for the district in the current study in recent years had shown that over 70% of students met grade level expectations in math in recent years (Oregon Department of Education). My results indicate that only 36% of the study participants met the Grade 8 EXPLORE benchmark. These appear to corroborate the national statistics on students’ math performance and contradict the impression that the majority of students are on track in math. The study results appear to indicate what many already presume, that although district wide skill development in math in Grade 8 may have appeared promising in light of state assessments, overall student performance on an early measure of college- and career- readiness showed the majority not to be on track to be well-prepared in math if higher standards are taken into consideration. However, because students took EXPLORE in October of Grade 8, as was standing practice in the district, rather than later in the year, it is likely that their results were lower than they would have been if students had taken the assessment later in the year after additional skill development opportunities. Criterion-related validity evidence for math CBMs. Results of this study add to the existing body of research on the technical adequacy of CBMs by showing a moderately strong association between easyCBM and a standardized math assessment linked to college- and career-readiness. My finding (r = .64) fits well with prior research on math CBMs summarized by Foegen et al. (2007), which showed that criterion validity 74 of CBMs in middle grades fell primarily in the range of r =.50-.70. Results in the current study were lower than those of Anderson et al. (2010a) who examined the relation between math CBMs and state assessments in Oregon and Washington (r = .81-.82) in middle grades, again, possibly due to the difference in rigor of the summative assessment used. Numerous researchers (Christ et al., 2008; Foegen & Deno, 2001; Fuchs et al., 2008 among others) have pointed out that math constructs are complex and inter-related, characteristics that makes them challenging to assess and to research, particularly as researchers move beyond examining simple computation in the elementary grades. The study adds to the body of research on CBMs and math assessments that include, as recommended by Thurber et al. (2002), both computation and application (Fuchs et al., 1999 as cited in Foegen et al., 2008; Helwig & Tindal, 2002; Nese et al., 2010). Additionally, although not focused solely on algebraic concepts, it complements existing research on CBMs involving algebra skills (Foegen, 2008) in the middle grades by examining links between a CBM and EXPLORE, both of which incorporate simple algebra skills. The moderately strong association shown in my study was not as strong as those found in studies within elementary grades or in studies of easyCBM and state generated standardized assessments such as those in Oregon and Washington, but it does imply that the easyCBM and EXPLORE may assess similar math skills. Results lend credence to the idea that educators may be able to be able to use easyCBM results to draw useful inferences about students’ math skills, even in light of the muti-faceted nature of the math skills being assessed in the middle grades and increasing standards nationwide in math. 75 Understanding the statistical relation between the measures may allow educators to build a more complete picture of students’ current math skills as they relate to emerging standards and may help them comprehend the potential utility of easyCBM to inform decision-making within schools focused on college- and career-readiness. CBM-math as a predictor. Many researchers have pointed to the potential of CBMs to predict student performance on summative tests (Anderson et al., 2010a; Jiban & Deno, 2007; Shapiro et al., 2006). Prior studies indicated support for computation CBMs as predictors of student results on applied math tests (Fuchs et al., 2005; Rutherford-Becker & Vanderwood, 2009) and CBMs’ predictive relation with large-scale math achievement tests (Helwig, et al., 2002; Anderson et al., 2010b). My study showed more modest results, but shed light on the predictive relation between easyCBM and one early indicator of students’ progress toward college- and career-readiness. It extends work done by prior researchers into CBMs in the middle grades by (a) including assessments that contain multi-faceted math skills that middle grade students are expected to master, beyond computation alone and (b) looking beyond state-generated assessments based on state standards to link CBM and EXPLORE, an assessment associated with college- and career-readiness standards. Furthermore, because the study indicated a predictive relation between an assessment administered at the beginning of students’ seventh-grade year and one administered a full year later, it expands the understanding of the potential predictive nature of CBM across grade levels. Understanding the relation between student performance on the two measures may help teachers comprehend the probable math achievement trajectory of their current Grade 7 76 students in subsequent grades and increase awareness of the need for targeted support during the students’ pivotal middle years. Demographic characteristics as predictors. Recent research had allayed some concerns about gender differences in math performances by showing similar levels of performance on standardized tests (Lindberg et al., 2010; Scafidi & Bui, 2010), but had highlighted concerns about low math performance among students qualifying for free or reduced lunch or special education (Hemphill & Vanneman, 2011; NCES, 2011; Wagner et al., 2006). My study reaffirmed those findings, showing that no statistical difference in means existed for males and females on either easyCBM or EXPLORE. However, students eligible for free or reduced lunch or special education scored significantly lower on both measures than did their non-identified peers. The current study provides evidence that various demographic variables, viewed both separately and in combination, may have a small but significant effect on math performance. Furthermore, being eligible for more than one category may put students at increased risk compared to their non-eligible peers. Results highlight existing concerns within schools about issues of equity and the underachievement of students within those subgroups. Although results for ELD students were inconclusive due to low numbers of ELD participants, the fact that none of the ELD students in the study achieved benchmark on either easyCBM or EXPLORE points to the ongoing challenge of closing the achievement gap in math for students whose dominant language is other than English. Students qualifying for SpEd and those qualifying for F/RL had a significantly lower average CBM score than that of their peers and scored well below the CBM cut score in fall of Grade 7. As Fuchs et al. (2010) pointed out, students moving beyond 77 elementary school may already have well-established academic deficits. My study affirms that students in certain subgroups come into seventh grade with pre-existing gaps in knowledge and skills, and that schools aiming to increase student math achievement must focus increased attention on skill development in early grades. Identification within the F/RL, SpEd, or ELD subgroups was associated with a negative effect on fall easyCBM scores in Grade 7. In turn, low math performance on the Grade 7 fall easyCBM was a significant predictor of low math performance on EXPLORE. Clearly, the largest portion of the variance in EXPLORE was attributable to easyCBM results, and demographic variables appeared to contribute very little additional change to EXPLORE. Rather than interpreting this to mean that student demographic characteristics had no influence on students’ performance on EXPLORE, it seems reasonable to interpret results to indicate that the easyCBM results already incorporated the effect of demographic variables on math performance. In other words, students did not perform lower than their peers on EXPLORE due to any particular impact of demographic characteristics on EXPLORE itself, but rather, demographic characteristics may have influenced their math performance overall, a phenomenon already evident in their easyCBM scores from the fall. Consistency of CBMs in predicting later performance. Helwig et al. (2002) and others have indicated that understanding predictors of student achievement on large- scale math assessments may help provide early notice of students who are not on track to meet state math standards and allow for intervention. With regard to CBMs specifically, a ROC analysis by Anderson et al. (2010a) had shown that easyCBM consistently predicted student achievement of seventh-grade math standards on the state test (AUC = 78 .88). Results of my study (AUC = .86) indicate that fall seventh-grade easyCBM also can predict consistently students who are at risk of not achieving the EXPLORE benchmark in eighth-grade. However, not surprisingly, the existing district cut score based on the 40th percentile was not optimal for predicting students who were at risk of missing the EXPLORE benchmark. Considering that a much smaller percentage of students in the participating schools met the EXPLORE benchmark (36%) than typically had met the state standards (73%) in previous years, it stands to reason that the cut score used on easyCBM based on prediction of state assessment results would be too low to capture accurately those students at risk of low performance on EXPLORE. Using decision- making rules set forth by Silberglitt and Hintze (2005) for establishing cut scores, I determined the optimal sensitivity (.79) and specificity (.76) provided within the coordinates of the curve in the ROC analysis was associated with a cut score of 32.5 (or 33 for practical purposes). A cut score of 33, rather than 29, would be a more consistent predictor of students at risk of not achieving the EXPLORE benchmark. Conclusions and Implications Results from this study may help shape educators’ understanding about students’ progress toward college- and career-readiness in math, assist schools in understanding students’ level of risk in relation to the EXPLORE benchmark, deepen schools’ understanding of the interplay between specific demographic factors and math assessment, and aid schools in their work to identify students in need of additional support. Such information may help school leaders make informed programmatic decisions to shape students’ achievement in math. 79 Classroom teachers may use easyCBM with some degree of confidence to infer information about students’ developing math skills and their likely future performance on EXPLORE. Results from the study may serve to extend the data pathway provided by the EXPLORE-PASS-ACT series by creating a link between CBM and the ACT test series. Used with other measures, easyCBM may help provide an early indication of whether students are on track to be college- and career-ready in math and can add to school leaders’ research-based understanding of seventh-graders’ preparedness in math at the classroom level, grade level, and district level in relation to more rigorous math standards. Because a large percentage of students (64%) did not achieve the EXPLORE benchmark, it seems clear that schools’ conversations about middle school students’ progress toward college- and career-readiness must change. Almost two thirds of seventh-graders are not on track to meet EXPLORE benchmark in math and might be considered at risk. Based on guidelines of the National Center on Response to Intervention, data indicating that over 20% of students are at risk may signal that schools need to make improvements in core instruction to ensure students achieve desired outcomes. Study results can aid leaders in building a sense of urgency to promote change and can guide educators in making sound programmatic decisions to improve students’ opportunities to build the math knowledge and skills they may need to demonstrate on later assessments. Results showing that CBM can predict student achievement of the EXPLORE benchmark with consistency indicate that fall easyCBM may continue to prove useful as a screener in an RTI system to help identify students at high risk, at some risk, and at low risk of missing the EXPLORE benchmark. During the current period of transition as 80 schools nationwide align curriculum, instruction, and assessment toward more rigorous standards, percentiles used to benchmark students will change over time. Currently, considering the increasing rigor of standards and assessments nationwide, educators will need to be aware that students who score above the 40th percentile are still at risk and may not be predicted to meet new standards. Districts will need to revise the range of cut scores used in CBM benchmarking in order to identify students at high risk, some risk, and low risk of not achieving standards on more rigorous assessments and may want to use CBM in combination with other measures. Study results that showed lower performance of special education, free or reduced lunch, and ELD students on both measures may serve to heighten concern in schools about the current achievement and probable future achievement of those students in math. In particular, the fact that no ELD students, even those in the upper levels of ELD, met benchmark on either measure may motivate school leaders to take action to address ELD students’ needs more than they have in the past. Because information on specific students’ socioeconomic status is confidential and not used by teachers to guide instruction for specific students, information garnered from the study about the possible impact of students’ free or reduced lunch status on math results may be useful primarily for leaders considering school wide supports for economically disadvantaged students and professional development for instructors about how to mitigate the effects of poverty. Understanding that gender had no discernible effect on math performance on easyCBM and EXPLORE may help further erode the misconception that female students will perform lower than their male counterparts in math. Information about the 81 similarities in math performance for boys and girls, shared with students, may help to diminish stereotype threat in schools. On a broad scale, study results indicating that only 36% of eighth-graders met the EXPLORE benchmark should prompt school districts to act to ensure that more students have the opportunity to be on track for college and career. Schools nationwide already are redesigning math programs K-12 in light of the Common Core State Standards and new expectations regarding college- and career-readiness. The results of the study may help school leaders build a sense of urgency for the changes that need to occur in the areas of curriculum, instruction, alignment with new standards, and both formative and summative assessment. New work on CBM benchmarking and new understandings of students’ level of risk for not achieving college- and career-readiness can assist school leaders in making decisions about core instruction in math and math interventions for students with identified needs. Using easyCBM to identify students at various levels of risk may assist schools in allocating resources to ensure that those who need the most intensive interventions receive them. Because school resources have not increased in tandem with the increase in rigor of the assessments, schools are still in need of research-based assessments to assist them in identifying the students who may require different levels of intervention and support so that they can make wise use of the resources that they have. Again, because schools do not share information about students’ socioeconomic status, it is not reasonable for schools to provide specific math interventions to students who qualify for free or reduced lunch based on their F/RL status alone. Instead, schools may use the information about students’ performance to provide supports school wide 82 that may benefit students with low income. Professional development about challenges facing low-income students and teaching strategies that may be effective with low- income students could improve teachers’ overall effectiveness. Areas for Future Research The association between easyCBM-Math and EXPLORE-math, the predictive relation between easyCBM and EXPLORE, and the consistency with which easyCBM identified students at risk of not meeting the EXPLORE benchmark provide some evidence that a teacher accessible CBMs can offer useful information about students’ likely math performance on a larger national assessment linked to college- and career- readiness. The study results point to areas for future research, including studies involving (a) CBMs and measures of college- and career-readiness, (b) math assessments and diverse student populations, and (c) identification of students at risk in math. CBMs and measures of college- and career-readiness. The ACT series including EXPLORE, PLAN, and ACT have a research base that connects them to standards of college- and career-readiness (ACT, 2010), and this study’s results document a moderately strong association between CBM and the first assessment in that series, the EXPLORE. Because Common Core State Standards and the upcoming assessments of those standards (i.e. Smarter Balanced, PARCC) promise to provide schools with new goals and new accountability measures, researchers should study the association between CBMs and the emerging CCSS assessments once those are made available to schools beyond the current pilot phase. In addition, as test developers create other early measures of college- and career-readiness, such as the upcoming ASPIRE authored by ACT and designed for use prior to Grade 8, researchers should seek to 83 understand the relation between CBMs and those new assessments. Future studies can provide schools with information about the utility of CBM as a predictor of student performance on a number of measures that promise to become part of the national conversation about college- and career-readiness. Consideration of diverse populations. As Foegen (2008) indicated, little prior CBM research has included students in the middle grades from diverse populations. The current study examined results from students from specific subgroups including SpEd, F/RL, and ELD, but much more information about diverse populations is needed. Participants in the current study were drawn from a district with little racial or ethnic diversity and included very few students studying English as an additional language. Future research on CBMs in the middle grades should include more ethnically and racially diverse student populations in order to provide schools with information about interactions between CBM and EXPLORE results and specific demographic characteristics. Studies should include larger samples of ELD students in order to provide schools with information about those students’ current math performance and likely achievement trajectory. Researchers should design studies to delve further into the association between assessment results and specific categories within student subgroups. The current study examined results of all participants eligible for special education services. Future research should examine whether students with specific disabilities score differently from other special education students and from their peers in general education. That information can assist educational practitioners in understanding math performance of specific populations, and, potentially, which students may be in greatest need of support. 84 Identification of students at risk. More research is needed on links between CBMs and measures of college- and career-readiness and should focus on the consistency with which CBMs predict student performance on those measures. Research on the CBM scores and percentiles that best identify students at risk will assist districts in establishing levels of prevention within RTI processes as they understand the level of risk that different students may have for not achieving new standards in math. That information can assist districts in sound decision making as they seek to allocate resources wisely to support students in preparing for college and career. In this study, CBM showed a moderate association and predictive relation with EXPLORE, and it appeared that CBM results predicted consistently which students would meet or not meet the EXPLORE benchmark. An awareness of those aspects of CBM may be useful to school leaders during what, clearly, is a period of transition as schools nationwide strive to realign curriculum, instruction, and assessments with new expectations in math. 85 APPENDIX A DISTRIBUTIONS OF EASYCBM AND EXPLORE WITH NORMAL CURVE Figure 1 Distribution of Grade 7 Fall easyCBM Math Results Figure 2 Distribution of Grade 8 Fall EXPLORE Math Results 86 APPENDIX B SCATTERPLOT OF EASYCBM AND EXPLORE AND DISTRIBUTION OF RESIDUALS OF REGRESSION OF EXPLORE ON EASYCBM Figure 3 Scatterplot of Grade 7 fall easyCBM-math and Grade 8 EXPLORE-math Figure 4 Distribution of Residuals in Regression of EXPLORE on easyCBM and Demographic Variables 87 APPENDIX C ROC CURVE INCLUDING EASYCBM-MATH AND EXPLORE-MATH AND COORDINATES OF THE CURVE Figure 5 ROC Curve Including easyCBM-Math and EXPLORE-Math 88 Table A1 Coordinates of the Curve with Test Result Variable CBM-Math Positive if Less Than or Equal Toa Sensitivity 1 – Specificity .00 .000 .000 1.50 .002 .000 2.50 .004 .000 4.00 .005 .000 5.50 .009 .003 6.50 .011 .003 7.50 .013 .003 8.50 .014 .003 9.50 .022 .003 10.50 .025 .006 11.50 .032 .006 12.50 .039 .006 13.50 .052 .006 14.50 .059 .006 15.50 .072 .006 16.50 .093 .009 17.50 .109 .009 18.50 .131 .012 19.50 .159 .012 20.50 .195 .018 21.50 .233 .021 22.50 .262 .021 23.50 .301 .028 24.50 .357 .034 25.50 .405 .037 26.50 .457 .049 27.50 .500 .061 28.50 .550 .086 29.50 .609 .117 30.50 .685 .147 31.50 .733 .190 32.50 .789 .239 33.50 .855 .288 89 Table A1 (continued) Positive if Less Than or Equal Toa Sensitivity 1-Specificity 34.50 .900 .362 35.50 .935 .457 36.50 .964 .552 37.50 .975 .647 38.50 .995 .721 39.50 .998 .791 40.50 .998 .819 41.50 1.000 .865 42.50 1.000 .883 43.50 1.000 .945 44.50 1.000 .985 46.00 1.000 1.000 The test result variable(s): CBM_FALL_2011_12 has at least one tie between the positive actual state group and the negative actual state group. a. The smallest cutoff value is the minimum observed test value minus 1, and the largest cutoff value is the maximum observed test value plus 1. All the other cutoff values are the averages of two consecutive ordered observed test values. (SPSS) 90 APPENDIX D SAMPLE OF GRADE 7 EASYCBM-MATH AND GRADE 8 EXPLORE-MATH Figure 7 Sample of Grade 7 easyCBM-Math Test Item Online © University of Oregon Figure 8 Sample of Grade 8 EXPLORE-Math Test Item © ACT, Inc. 91 REFERENCES CITED ACT. (2007). Rigor at risk: Reaffirming quality in the high school core curriculum. Iowa City, IA: Author. ACT. (2008). The forgotten middle: Ensuring that all students are on target for college and career readiness before high school. Iowa City, IA: Author. ACT. (2010). A First Look at the Common Core and College and Career Readiness. Iowa City, IA: Author. ACT. (2011). 2010/2011 Technical Manual: EXPLORE. Iowa City, IA: Author. ACT. (2012a). ACT Profile Report-National: Graduating Class 2012. Retrieved from www.act.org/newsroom/data/2012/profilereports.html ACT. (2012b). The Condition of College and Career Readiness 2012 National Report. Iowa City, IA: Author. ACT. (2012c). EXPLORE interpretive guide for student and school reports 2012-2013. Iowa City, IA: Author. Alonzo, J., Ulmer, K., Tindal, G., & Glasgow, A. (2006). easyCBM online assessment system. http://easycbm.com. Eugene, OR: Behavioral Research and Teaching, University of Oregon. Anderson, D., Alonzo, J., & Tindal, G. (2010a). Diagnostic Efficiency of easyCBM Math: Oregon (Technical Report No. 1009). Eugene, OR: Behavioral Research and Teaching, University of Oregon Anderson, D., Alonzo, J., & Tindal, G. (2010b). easyCBM Mathematics Criterion Related Validity Evidence: Oregon State Test (Technical Report No. 1011). Eugene, OR: Behavioral Research and Teaching, University of Oregon. Aud, S., Hussar, W., Johnson, F., Kena, G., Roth, E., Manning, E., … Zhang, J. (2012). The Condition of Education 2012 (NCES 2012-045). U.S. Department of Education, National Center for Education Statistics. Washington, DC. Retrieved from http://nces.ed.gov/pubsearch. Calhoon, M. B. (2008). Curriculum-based measurement for mathematics at the high school level: What we do not know…what we need to know. Assessment for Effective Intervention, 33(4), 234-239. doi:10.1177/1534508407313488 Calhoon, M. B., & Fuchs, L. S. (2003). The effects of peer-assisted learning strategies and curriculum-based measurement on the mathematics performance of secondary students with disabilities. Remedial and Special Education. 24(4), 235-45. 92 Christ, T. J., Scullin, S., Tolbize, A., & Jiban, C. L. (2008). Implications of recent research: Curriculum-based measurement of math computation. Assessment for Effective Intervention, 33(4), 198-205. doi:10.1177/1534508407313480 Clarke, B., & Shinn, M. R. (2004). A preliminary investigation into the identification and development of early mathematics curriculum-based measurement. School Psychology Review, 33, 234–248. Cohen, J. (1988). Statistical power and analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum Associates. Common Core State Standards Initiative. (2010a). Common core state standards for mathematics. Washington, D.C.: National Governors Association Center for Best Practices and Council of Chief State School Officers. Retrieved from http://www.corestandards.org/assets/CCSSI_Math%20Standards.pdf Common Core State Standards Initiative. (2010b). Common core state standards for mathematics: Appendix A: Designing high school mathematics courses based on the common core state standards. Washington, D.C.: National Governors Association Center for Best Practices and Council of Chief State School Officers. Retrieved from http://www.corestandards.org/assets/CCSSI_Mathematics_Appendix_A.pdf Conley, D. T. (2007). Redefining College Readiness, Vol. 3. Eugene, OR: Educational Policy Improvement Center. Connected Mathematics Project. (2013). Mathematics learning goals. Michigan State University. Retrieved from http://connectedmath.msu.edu/mathcontent/goals.shtml Deno, S. L. (1985). Curriculum-based measurement: The emerging alternative. Exceptional Children, 52(3), 219–232. Deno, S. L. (1989). Curriculum-based measurement and alternative special education services: A fundamental and direct relationship. In M.R. Shinn (Ed.), Curriculum- based measurement: Assessing special children (pp. 1-17). New York: Guilford Press. Deno, S. L. (2003). Developments in curriculum-based measurement. Journal of Special Education, 37(3), 184-192. Deno, S. L. & Fuchs, L. S. (1991). Paradigmatic distinctions between instructionally relevant measurement models. Exceptional Children, 57(6), 488-99. 93 Espin, C. & Tindal, G. (1998). Curriculum-based measurement for secondary students (pp. 214–53). In M.R. Shinn (Ed.), Advanced applications of curriculum-based measurement. New York: Guilford Press. Fernandez, C. & Yoshida, M. (2004). Lesson study: A Japanese approach to improving mathematics teaching and learning. Mahwah, NJ: Lawrence Erlbaum Associates. Foegen, A. (2000). Technical adequacy of general outcome measures for middle school mathematics. Diagnostique, 25(3), 175–203. Foegen, A. (2008). Algebra progress monitoring and interventions for students with learning disabilities. Learning Disability Quarterly, 31(2), 65-78. Foegen, A., & Deno, S. L. (2001). Identifying growth indicators for low achieving students in middle school mathematics. The Journal of Special Education, 35, 4- 16. Foegen, A., Jiban, C., & Deno, S. L. (2007). Progress-monitoring measures in mathematics: A review of the literature. The Journal of Special Education, 41, 121-139. doi:10.1177/00224669070410020101 Foegen, A., & Morrison, C. (2010). Putting algebra progress monitoring into practice: Insights from the field. Intervention In School & Clinic, 46(2), 95-103. doi:10.1177/1053451210375302 Fuchs, D., Fuchs, L.S., & Compton, D. L. (2012). Smart RTI: A next-generation approach to multilevel prevention. Council for Exceptional Children, 78(3), 263- 279. Fuchs, D., Mock, D., Morgan, P. L., Young, C. L. (2003). Responsiveness-to- intervention: definitions, evidence, and implications for the learning disabilities construct. Learning Disabilities Research & Practice, 18(3), 157-171. doi: 10.1111/1540-5826.00072. Fuchs, L. S. (2004). The past, present, and future of curriculum-based measurement research. School Psychology Review, 33,188-192. Fuchs, L. S., & Fuchs, D. (1986). Effects of systematic formative evaluation: A metaanalysis. Exceptional Children, 53, 199-208. Fuchs, L. S., Fuchs, D., & Compton, D. L. (2010). Rethinking response to intervention at middle and high school. School Psychology Review, 39(1), 22–28. Fuchs, L. S., Fuchs, D., & Courey, S. J. (2005). Curriculum-based measurement of mathematics competence: From computation to concepts and applications to real- life problem solving. Assessment for Effective Intervention. 30(2), 33-46. 94 Fuchs, L. S., Fuchs, D., & Hamlett, C. L. (1989). Effects of alternative goal structures within curriculum-based measurement. Exceptional Children, 55, 429–438. Fuchs, L. S., Fuchs, D., Hamlett, C. L., Stuebing, K., Fletcher, J.M., Lambert, W. (2008). Problem-solving and computational skill: Are they shared or distinct aspects of mathematical cognition? Journal of Educational Psychology, 100, 30-47. doi: 10.1037/0022-0663.100.1.30 Fuchs, L. S., Hamlett, C. L., & Fuchs, D. (1999). Monitoring basic skills progress: Basic math concepts and applications [computer program manual]. Austin, TX: PRO- ED. Gersten, R., Beckmann, S., Clarke, B., Foegen, A., Marsh, L., Star, J. R., & Witzel, B. (2009). Assisting students struggling with mathematics: Response to Intervention (RtI) for elementary and middle schools (NCEE 2009-4060). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education. Retrieved from http://ies.ed.gov/ncee/wwc/publications/practiceguides/ Hanich, L. B., Jordan, N. C., Kaplan, D., & Dick, J. (2001). Performance across different areas of mathematical cognition in children with learning disabilities. Journal of Educational Psychology, 93(3), 615-626. doi:10.1037/0022-0663.93.3.615 Helwig, R., Anderson, L., & Tindal, G. (2002). Using a concept-grounded, curriculum- based measure in mathematics to predict statewide test scores for middle school students with LD. The Journal of Special Education, 36, 102–112. Helwig, R., & Tindal, G. (2002). Using general outcome measures in mathematics to measure adequate yearly progress as mandated by Title I. Assessment for Effective Intervention, 28(1), 9–18. Hemphill, F.C., and Vanneman, A. (2011). Achievement gaps: How hispanic and white students in public schools perform in mathematics and reading on the National Assessment of Educational Progress (NCES 2011-459). National Center for Education Statistics, Institute of Education Sciences, U.S. Department of Education. Washington, DC. Hiebert, J., & Lefevre, P. (1986). Conceptual and procedural knowledge in mathematics: An introductory analysis. From Conceptual and procedural knowledge: The case of mathematics (Ed. Hiebert, J.). pp.1-27. Hillsdale, N.J. England: Lawrence Erlbaum Associates, Inc. Hoover, H. D., Hieronymus, A. N., Frisbie, D. A., & Dunbar, S. B. (1993). Iowa Tests of Basic Skills. Chicago: Riverside. 95 Jiban, C. L., & Deno, S. L. (2007). Using math and reading curriculum-based measurements to predict state mathematics test performance: Are simple one- minute measures technically adequate? Assessment for Effective Intervention, 32, 78–89. Keller, M. A., & Shapiro, E. S., & Hintze, J. M. (2008). Long-term diagnostic accuracy of curriculum-based measures in reading and mathematics. School Psychology Review, 37(3) 374-390. Kelley, B., Hosp, J. L., Howell, K. W. (2008). Curriculum-based evaluation and math: An overview. Assessment for Intervention, 33(4), 250-256. doi: 10.1177/1534508407313490 Liang, J. H., Heckman, P. E., Abedi, J. (2012). What do the California Standards Test results reveal about the movement toward eighth-grade algebra for all? Educational Evaluation and Policy Analysis, 34(3) 328-343. doi: 10.3102/0162373712443307 Lindberg, S. M., Hyde, J. S., Peterson, J. L., Linn, M. C. (2010). New trends in gender and mathematics performance: A meta-analysis. Psychological Bulletin, 136(6), 1123-1135. doi:10.1037/a0021276. Messick, S. (1995). The validity of psychological assessment: Validation of inferences from persons’ responses and performances of scientific inquiry into score meaning. American Psychologist, 20(9), 741-749. National Center for Education Statistics. (2011). The Nation’s Report Card: Mathematics 2011 National Assessment of Educational Progress at Grades 4 (NCES 2012- 458). National Center for Education Statistics [serial online]. U.S. Department of Education, Washington, D.C. National Center on Response to Intervention. (2010). Essential components of RTI: A closer look at Response to Intervention. Retrieved from http://www.rti4success.org/images/stories/pdfs/rtiessentialcomponents_051310.pdf National Council of Teachers of Mathematics. (2006). Curriculum focal points for prekindergarten through grade 8 mathematics: A quest for coherence. Reston, VA: Author. National Mathematics Advisory Panel (2008). Foundations for Success: The Final Report of the National Mathematics Advisory Panel. U.S. Department of Education: Washington, DC, 2008. 96 National Research Council. (2001). Adding it up: Helping children learn mathematics. J. Kilpatrick, J. Swafford, and B. Findell (Eds.). Mathematics Learning Study Committee, Center for Education, Division of Behavioral and Social Sciences and Education. Washington, DC: National Academy Press. Nese, J. F. T., Lai, C. F., Anderson, D., Jamgochian, E. M., Kamata, A., Saez, L., … Tindal, G. (2010). Technical Adequacy of the easyCBM Mathematics Measures: Grades 3-8, 2009-2010 Version (Technical Report No. 1007). Eugene, OR: Behavioral Research and Teaching, University of Oregon. No Child Left Behind Act of 2001. (2002). Public Law #107-110, 115 Stat. 1425. Oregon Department of Education. (2010). Oregon’s statewide assessment system technical report: Vol. 1. Annual report. Salem, Oregon: Oregon Department of Education, Office of Assessment and Information Services. Partnership for Assessment of Readiness for College and Careers. (2012). PARCC assessment design. Retrieved October 15, 2012 from http://www.parcconline.org/parcc-assessment-design Phillips, N. B., Hamlett, C. L., Fuchs, L. S., & Fuchs, D. (1993). Combining classwide curriculum-based measurement and peer tutoring to help general educators provide adaptive education. Learning Disabilities Research and Practice, 8, 148– 156. Riverside. (1995). Performance assessments for the Iowa Test of Basic Skills. Chicago: Author. Rutherford-Becker, K. J., & Vanderwood, M. L. (2009). Evaluation of the relationship between literacy and mathematics skills as assessed by curriculum-based measures. California School Psychologist. 14, 23-34. Scafidi, T., & Bui, K. (2010). Gender similarities in math performance from middle school through high school. Journal of Instructional Psychology, 37(3), 252-255. Shapiro, E. S., Keller, M. A., Lutz, G., Santoro, L. E., & Hintze, J. M. (2006). Curriculum-based measures and performance on state assessment and standardized tests: Reading and math performance in Pennsylvania. Journal of Psychoeducational Assessment, 24(19), 19-35. doi:10.1177/0734282905285237 Shinn, M. R. (Ed.). (1989). Curriculum-based measurement: Assessing special children. New York: Guilford. 97 Shinn, M. R., & Hubbard, D. D. (1992). Curriculum-based measurement and problem- solving assessment: Basic procedures and outcomes. Focus on Exceptional Children, 24(5), 1-20. Shinn, M. R., & Marston, D. (1985). Differentiating mildly handicapped, low-achieving, and regular education students: A curriculum-based approach. Remedial and Special Education, 6, 31–38. Silberglitt, B. & Hintze, J. (2005). Formative assessment using Cbm-R cut scores to track progress toward success on state-mandated achievement tests: A comparison of methods. Journal of Psychoeducational Assessment, 23, 304-325, doi: 10.1177/073428290502300402 Smarter Balanced Assessment Consortium. (2012, June). Quarterly Report. Retrieved from http://www.smarterbalanced.org/resources-events/publications-resources/ Terry, L., & Rosin, M. (2011, September). California’s math pipeline: The grade 7 pivot point. Issue Brief. EdSource. Thurber, R. S., Shinn, M. R., & Smolkowski, K. (2002). What is measured in mathematics tests? Construct validity of curriculum-based mathematics measures. School Psychology Review, 31, 498–513. Tindal, G., Marston, D., & Deno, S. L. (1983). The reliability of direct and repeated measurement (Research Report No. 109). Institute for Research on Learning Disabilities. Minneapolis: University of Minnesota. U.S. Department of Education. (2010, March). A Blueprint for Reform: The Reauthorization of the Elementary and Secondary Education Act. Retrieved from http://www2.ed.gov/policy/elsec/leg/blueprint/index.html Wagner, M., Newman, L., Cameto, R., and Levine, P. (2006). The academic achievement and functional performance of youth with disabilities. A report from the National Longitudinal Transition Study-2 (NLTS2). (NCSER 2006-3000). Menlo Park, CA: SRI International. Wang, J. & Goldschmidt, P. (2003). Importance of middle school mathematics on high school students’ mathematics achievement. Journal of Educational Research, 97, 3-19. Yoshida, M. 1999. Lesson study: A case study of a Japanese approach to improving instruction through school-based teacher development. Doctoral Dissertation, University of Chicago.