POSTER TEMPLATE BY: www.PosterPresentations.com easyCBM® Formative Assessment in Mathematics: A Comparison of CCSS-Aligned Item Writing Strategies P. Shawn Irvin Behavioral Research and Teaching (BRT) – University of Oregon – EDLD 655 – Fall 2012 Design Option Resource Considerations References The Importance of Aligned Formative Assessments Introduction Design Option Conceptual Models For further information, please contact Shawn Irvin, pirvin@uoregon.edu. Thanks to contributing researchers Dr. Gerald Tindal, Dr. Julie Alonzo and Daniel Anderson. University of Oregon The Common Core State Standards and easyCBM® In 2010, the Common Core State Standards (CCSS) for Mathematics were released to provide a unified set of expectations for developing mathematics knowledge and skills for grades K-12 (NGA and CCSSO, 2010). Currently, 47 states and the District of Columbia have formally adopted the CCSS through one or both of two assessment consortia working with states to implement the standards, with the impending release of the Common Assessments. Math items comprising the easyCBM® formative assessment system were originally written to align with the National Council of Teachers of Mathematics Curriculum Focal Points (Nese et al., 2010). Recently, researchers found a reasonable pattern of alignment between K-5 math items and the CCSS, though not without areas of concern. For example, Irvin, Park, Alonzo and Tindal (2012) suggested some K-2 CCSS were underrepresented by easyCBM® assessments – finding similar patterns for grades 3-8. Given the movement of educators toward the CCSS, assessment development within the easyCBM® system must focus on writing new items to address those standards currently underrepresented within existing K-5 measures. GOAL: In what follows, two alternative study design options for writing, reviewing and scaling 3,000 new K-5 easyCBM® CCSS-aligned math items by fall 2013 are explored. Based largely on budget and control advantages a streamlined Trainee-Trainee option is recommended. Within the current U.S. standards-based accountability system, academic standards inform both instructional and assessment practices. While researchers have studied the alignment of summative state tests to adopted standards (Webb, 1999), research around the alignment of formative assessments to such standards is lacking, though perhaps not justifiably. Within a response to intervention (RTI) framework, formative assessments (i.e., interim benchmarking and progress monitoring measures) are administered over the course of an academic year to track student progress toward grade-level expectations, guiding instruction and aiding in the identification of students in need of instructional intervention and special education services. In this light, formative assessment can be thought of as an integrated and ongoing process within the standards-based instructional cycle (pictured above) of gathering student-level information as a basis for valid instructional decision-making. Thus, academic standards and formative assessments must be strongly aligned for teachers to make valid test-based inferences and appropriate instructional decisions tied to student needs. Trainer-Trainee Design Option (Model for Grade K shown below): Trainer-Trainee Design Option Key Strengths: ²  In-person training allows for: Ø  Group discussion/brain-storming Ø  Practice writing and review/editing actual items Ø  Verbal/non-verbal cues and checks for understanding Ø  Greater buy-in due to personal relationship and teamwork •  Fewer recruiting/communication/enforcement demands Key Weaknesses: ²  Far greater budget costs due to more and deeper tasks for Item Reviewers, need for more Item Writers •  Expertise of participant pool geographically limited ²  Writer expertise out of lead researcher control (i.e., Reviewer conception of expertise, access to quality Writers) •  Communication gap b/t lead researcher and Writers •  In-person training places greater time/energy load on Reviewers Trainee-Trainee Design Option Key Strengths: ²  Fewer budget costs due to fewer and less deep Reviewer tasks (i.e., no recruiting/training duties), need for fewer Writers •  Nationwide geographic pool for expert participants •  Previous study results allow for “quality” of more participants to be known •  Item Writer expertise under lead researcher control •  Recruitment of Item Writers by Reviewers still possible Key Weaknesses: ²  Greater time commitment for lead researcher (i.e., more recruiting demands, two webinar trainings, communicating with 18 individuals) ²  Online training does not allow for advantages of in-person training (i.e., “group thinking and doing”, understanding checks) Alonzo, J., & Tindal, G. (2009). The development of K-8 progress monitoring measures in mathematics for use with the 2% and general education populations: Kindergarten (Technical Report No. 0921). Eugene, OR: Behavioral Research and Teaching, University of Oregon. Anderson, D., Irvin, P. S., Alonzo, J., & Tindal, G. (2012). The Alignment of the easyCBM middle school mathematics CCSS measures to the Common Core State Standards (Technical Report No. 1228). Eugene, OR: Behavioral Research and Teaching, University of Oregon. Anderson, D., Irvin, P. S., Patarapichayatham, C., Alonzo, J., & Tindal, G. (2012). The development and scaling of the easyCBM CCSS middle school mathematics measures (Technical Report No. 1207). Eugene, OR: Behavioral Research and Teaching, University of Oregon. Irvin, P. S., Park, B. J., Alonzo, J., Tindal, G. (2012). The alignment of the easyCBM grades K-2 math measures to the Common Core State Standards (Technical Report No. 1228). Eugene, OR: Behavioral Research and Teaching, University of Oregon. National Governors Association and Council of Chief State School Officers. (2010). Common Core State Standards. National Governors Association Center for Best Practices and Council of Chief State School Officers, Washington D.C. Webb, N. L. (1999). Alignment of science and mathematics standards and assessments in four states (Research Monograph No. 18). Madison, WI: University of Wisconsin-Madison, National Institute for Science Education. Contact Information and Acknowledgements Lead Item Reviewer/Editor Item Writer 1 CC (n = 25) OA (n = 25) NBT (n = 25) MD (n = 25) G (n = 25) Item Writer 2 Item Writer 3 Item Writer 4 In the Trainer-Trainee design option Lead Reviewers are recruited and trained by the lead researcher in person. The Lead Reviewer recruits and trains four grade-level Item Writers. Item Writers each produce a total of 125 grade level items that are reviewed and edited by the Lead Reviewer, prior to whole-group reviews by the lead researcher’s in- house review team. This item writing and review design option is adapted from the work of Anderson, Irvin, Patarapichayatham, Alonzo and Tindal (2012), whose research resulted in easyCBM® CCSS- aligned middle school math items in fall 2012. Trainee-Trainee Design Option (Model for Grade K shown below): Lead Researcher Item Writer 1 CC (n = 25) OA (n = 25) NBT (n = 25) MD (n = 25) K.G (n = 25) Item Writer 2 Item Writer 3 Item Writer 4 Lead Item Reviewer/Editor In the Trainee-Trainee design option Lead Reviewers and Item Writers are recruited and then trained by the lead researcher in separate online webinars. Each Item Writer produces 250 new items, 125 for each of two grades (e.g., K-1) that are reviewed for completeness by the lead researcher, and then passed to a grade-level Lead Reviewer for review and editing, prior to a series of whole-group reviews by the lead researcher’s in-house review team. This item writing and review design option is adapted from the work of BRT researchers, for example, see Alonzo and Tindal (2009), whose research resulted in the original easyCBM® NCTM-aligned math items for grades K-8. CCSS Domains: CC = Counting and Cardinality; OA = Operations and Algebraic Thinking; NBT = Numbers and Operations in Base Ten; MD = Measurement and Data; G = Geometry Lead Researcher (Formative) Assessment Instruction Evaluation Standards CCSS Domains: CC = Counting and Cardinality; OA = Operations and Algebraic Thinking; NBT = Numbers and Operations in Base Ten; MD = Measurement and Data; G = Geometry Math item movement Training Math item movement Training Trainer-Trainee Design Option (Grades K-5) Participant and Material Budget Requirements: •  6 Lead Item Reviewer/Editors – one for each of grades K-5 Ø  $3,000 each, $18,000 total •  24 Item writers – four for each of grades K-5 Ø  $875 each, $21,000 total •  Graphics Designer Ø  Dependent on # of graphics – $15/hr, 160 hours, $2400 •  Lead Item Reviewer and Item Writer Study Guide Packets (hard copies) Ø  $20 each, $600 total •  Lead Item Reviewer training (in-person) Ø  $125 continental breakfast and lunch Outcome projections: •  500 new CCSS-aligned math items for each of grades K-5, 3,000 new math items overall by fall 2013 ²  Overall budget ≈ $42,125 Trainee-Trainee Design Option (Grades K-5): Participant and Material Budget Requirements: •  6 Lead Item Reviewer/Editors – one for each of grades K-5 Ø  $1,500 each, $9,000 total •  12 Item writers – four for each of three grade bands, K-1, 2-3 and 4-5 Ø  $800 each, $9,600 total •  Graphics designer Ø  Dependent on # of graphics – $15/hr, 160 hours, $2400 •  Lead Item Reviewer and Item Writer Study Guide Packets (electronic-hardcopy hybrid) Ø  $5 each, $90 total •  Lead Item Reviewer training (webinar) Ø  2-hour webinar – $250 •  Item Writer training (webinar) Ø  2-hour webinar – $250 Outcome projections: •  500 new CCSS-aligned math items for each of grades K-5, 3,000 new math items overall by fall 2013 ²  Overall budget ≈ $21,590 Design Options: Relative Strength and Weaknesses Study Task Timeline Recruitment November 2012 Training and materials planning November – December 2012 Webinar training December 2012 Item writing and review December 2012 – February 2013 Graphics design January – March 2013 In-house item review/editing March – April 2013 Online item-scaling May – June 2013 Assessment forms and release June – September 2013 Projected Study Timeline The implementation timeline is the same for both study design options – 3,000 new easyCBM® math items for grades K-5 available by the fall 2013 academic year. An anticipated study timeline is shown: The two study design options have relative advantages and disadvantages when compared to each other. An overarching benefit is that both designs have produced quality math items for the easyCBM® formative assessment (Alonzo and Tindal, 2009; Anderson, Irvin, Alonzo and Tindal, 2012); thus, both designs likely allow the researcher achieve study goals. Despite greater demands around time and training placed on the researcher, adoption of the Trainee-Trainee Design Option is projected to save over $20,000, a fact that cannot be discounted with ever-tightening budgets. Also, the greater degree of researcher control over participant recruitment and monitoring are added benefits that may yield higher quality items at the advent of in-house reviews. Justification for Design Option Choice