AN EVALUATION OF THE EFFECTS OF TEACHING STUDENTS IN A RESOURCE CLASSROOM A SELF-REGULATED ASSIGNMENT ATTACK STRATEGY by BRYAN NESS A DISSERTATION Presented to the Department of Special Education and Clinical Sciences and the Graduate School of the University of Oregon in partial fulfillment of the requirements for the degree of Doctor of Philosophy June 2009 11 University of Oregon Graduate School Confirmation of Approval and Acceptance of Dissertation prepared by: Bryan Ness Title: "An Evaluation of the Effects of Teaching Students in a Resource Classroom a Self-Regulated Assignment Attack Strategy" This dissertation has been accepted and approved in partial fulfillment of the requirements for the Doctor of Philosophy degree in the Department of Special Education and Clinical Sciences by: McKay Sohlberg, Chairperson, Special Education and Clinical Sciences Robert Horner, Member, Special Education and Clinical Sciences Richard Albin, Member, Special Education and Clinical Sciences Bonnie Todis, Member, Not from U of 0 Thomas Dishion, Outside Member, Psychology and Richard Linton, Vice President for Research and Graduate Studies/Dean of the Graduate School for the University of Oregon. June 13, 2009 Original approval signatures are on file with the Graduate School and the University of Oregon Libraries. © 2009 Bryan Michael Ness iii Bryan Michael Ness An Abstract of the Dissertation of for the degree of IV Doctor of Philosophy in the Department of Special Education and Clinical Sciences to be taken June 2009 Title: AN EVALUATION OF THE EFFECTS OF TEACHING STUDENTS IN A RESOURCE CLASSROOM A SELF-REGULATED ASSIGNMENT ATTACK STRATEGY Approved: McKay Moore Sohlberg, PhD Students who struggle academically frequently lack or are unable to apply cognitive-motivational processes imperative for self-regulated learning. It is increasingly evident that deficits in self-regulation are a shared characteristic across students who qualify for special education. For example, impaired executive functions, or the cognitive processes responsible for managing and directing goal-directed activity, is a prevalent symptom domain across students with diverse special education identifications. Higher- order cognitive deficits become increasingly relevant as students progress to the secondary level as this transition necessitates the use of more complex organizational schemes to manage increased academic workload. Assignment management is a particularly challenging task for these students as poor organizational skills and lack of strategic approach translate to excessive time spent on assignments, lost materials, and negative attitude toward academic work. There is a growing body of research suggesting interventions targeting self- regulation have potential to improve performance on complex academic tasks such as assignment completion. Broadly speaking, these interventions are referred to as "self- management" strategies and are intended to help students actively reflect on their own cognitive and learning processes while engaged in academic tasks. The purpose of this study was to evaluate the impact of self-management strategy instruction on student "Assignment Attack" and related academic, behavioral, and cognitive variables. This study extended a line of inquiry examining implementation and efficacy of interventions targeting organized, independent student completion of assignments in resource contexts for adolescents who qualify for special education. Utilizing a multiple baseline, across participants research design, this study revealed positive effects of a Self-Regulated Assignment Attack Strategy (SAAS) on assignment attack and teacher-reported student behavior during assignment completion. The results indicated the effect was domain specific with minimal generalized improvement to the other academic, behavioral, or cognitive outcome measures. A discussion of the results is provided focused on the implications of improved assignment attack, generalization, and measurement challenges. Suggestions for further research in this area are provided. v CURRICULUM VITAE NAME OF AUTHOR: Bry~n Ness PLACE OF BIRTH: Anaheim, California DATE OF BIRTH: July 24,1976 GRADUATE AND UNDERGRADUATE SCHOOLS ATTENDED: University of Oregon. Eugene, Oregon. University of Nevada, Reno. Reno, Nevada. Calvary Chapel Bible College. Murrieta, California. DEGREES AWARDED: Doctor of Philosophy in Communication Disorders and Sciences, 2009, University of Oregon. Master of Science in Speech Pathology and Audiology, 2002, University of Nevada. Bachelor of Science in Speech Pathology and Audiology, 2000, University of Nevada. AREAS OF SPECIAL INTEREST: Cognitive Rehabilitation Executive Function Impairment Cognitive Strategy Training for Adolescents Psychosocial Impact of Traumatic Brain Injury Vi Vll PROFESSIONAL EXPERIENCE: 2002-2005 Speech-Language Pathologist, Washoe Medical Center, Reno, Nevada. 2005-2009 Graduate Teaching Fellow, Communication Disorders and Sciences, University of Oregon. 2006-2009 2007-2009 Guest Lecturer, Communication Disorders and Sciences, University of Oregon. Instructor (summer term) Motor Speech Disorders; University of Oregon. GRANTS, AWARDS AND HONORS: Graduate Teaching Fellowship. University of Oregon. 2005-Present Clare Wilkins Chamberlin Memorial Award. University of Oregon. 2007-2008 Leon Culbertson Scholarship. University of Oregon. 2007-2008 Kasey Elizabeth Lindeleaf Scholarship. University of Oregon. 2005-2006 Jerry & Betty Wilson Scholarship in Speech Pathology. University ofNevada, Reno. 2000-2002 PUBLICATIONS: Ness, B., & Sohlberg, M.M., Albin, R. (2008). An evaluation of a second tier classroom-based assignment completion strategy for middle school students in a resou,rce room context. Submitted to Journal ofRemedial and Special Education. V111 ACKNOWLEDGMENTS I wish to express my sincere appreciation to Dr. McKay Moore Sohlberg for her support, wisdom, and tireless commitment to this project and every endeavor I undertook during my doctoral program. I also thank the members of my dissertation committee for your collective and individual contributions to this project. I immensely appreciate the efforts of M~ry Feusahrens for her help with data collection. Finally, I wish to thank Ms. Jennifer Martin and the Bethel School District, whose commitment to expanding evidence-based interventions for students who qualify for special education made this study possible. IX Dedicated to my loving wife and her undying support, encouragement, and patience; to my beautiful daughter who is my inspiration; to the Lord who is my Joy and Strength. xTABLE OF CONTENTS Chapter Page I. INTRODUCTION 1 Statement of the Problem 2 A Potential Solution 4 Purpose Statement.......................................................................... 6 II. REVIEW OF THE LITERATURE 7 Assigmnent Management in Secondary Education Required for 7 Advancement . Social Cognitive Theory: A Model of Learning 10 Self-Regulated Learning................................................................................ 17 Struggling Learners and Self-Regulation ;.................................... 21 Intervention for Deficits in Self-Regulation :.......................... 27 Research Questions III. METHODOLOGY 38 40 Pilot Study 40 Pilot Methods............................................................................................ 41 Pilot Results and Conclusions 42 Dissertation ;.................................... 45 Participants 45 Research Design 49 ----~------- - --_ .._---- ... _----_.. _... Xl Chapter Intervention Package: Self-Regulated Assignment Attack Strategy (SAAS) . Dependent Variables . Pr.ocedures ; . Data Collection and Analysis : . IV. RESlTLTS . Implementation . Assignment Attack . Generalized Impact . V. DISCUSSION . Fidel~ty of Implementation in Natural Context .. Improved Assignment Attack · . Generalized Impact . Lil)1itations and Future Directions . Summary . Page 51 58 69 70 74 74 78 82 95 96 97 99 103 104 APPE]~DICES 105 A. TEACHER SCRIPT FOR INITIAL INSTRUCTION 105 B. TEACHER SCRIPT FOR DAILY PROMPTS 107 C. TEACHER SCRIPT FOR FADING 4P'S CHECKLISr.......................... 108 BIBLIOGRAPHY 109 xu LIST OF FIGURES Figure Page 1. Assignment Attack Rating Scale 43 2. The Four P's Checklist 52 3. Self-Regulated Strategy Development Stages 54 4. Modified Checklist 57 5. Homework Problem Checklist 60 6. Student Self-Regulation Probe 63 7. Treatment Fidelity Worksheet 65 8. Social Validity Measure for the Teacher 67 9. Social Validity Measure for Students 68 10. Assignment Attack Data 80 11. Percentage ofIntervals On Task 91 X111 LIST OF TABLES Table Page 1. Demographic Data 48 2. Summary of Study Results ;.................................... 75 3. Student Social Validity Data :.......................... 77 4. Teacher Social Validity Data........................................................................... 78 5. Classroom Grades and Assignment Completion Rate 84 6. Self-Regulated Learning Outcome Data and Analysis 86 7. BRIEF Outcome Data and Analysis 88 8. Homework Problem Checklist Data 89 9. Correlation Coefficients for Assignment Attack Domains and On-Task 92 CHAPTER I INTRODUCTION "Give a man a fish and you feed him for a day. Teach a m~n to fish and you feed him for a lifetime." (Anonymous) The ancient proverb succinctly illustrates the motivation of educational researchers endeavoring to equip students with the tools necessary for effective learning. Teaching students to become self-sufficient learners has become increasingly important as educational attainment is closely linked to future opportunity for young people. At a time when students of all learning abilities are taught and evaluated under similar standards, it is imperative that educational professionals understand the process of effective learning and have the tools to teach all students to become self-sufficient in the classroom. This dissertation examines potential learning and academic effects from the implementation of a cognitive strategy intervention designed to help struggling students self-regulate their learning. The intervention is grounded in social-cultural theory. The mechanisms of learning as a developmental process have been a point of philosophical and theoretical debate for decades. Most notable are Piaget's learning stages model, Vygotsky's social-cultural perspective and "zone of proximal development," and Bandura's social-cognitive theory. Each ofthese theories has left an indelible mark on 1 2education, but social-cognitive theory purports the most comprehensive explanation of the cognitive processes linked to efficient learning. Social-cultural theory proposes that learning results from interacting with the environment-and is shaped by inter-related motivational, behavioral, and cognitive processes (e.g., Bandura, 1986; Boekaerts & Cascallar, 2006; Scliunk & Zimmerman, 1997). One critical cognitive process essential for learning and academic achievement is self-regul~tion. Self-regulation refers to the capacity for evaluating and controlling thought processes and behavior based on situational demands (Boekaerts, Maes, & Karoly, 2005). Applied to educational contexts, students self-regulate their learning by continually assessing their cognitive and motivational state during academic tasks, making adjustments through the use of strategies (Zimmerman, 1990). The degree to which students self-regulate their learning is closely related to higher academic achievement (Pintrich & DeGroot, 1990). Statement ofthe Problem Students who struggle academically frequently lack or are unable to apply the cognitive-motivational processes imperative for self-regulated learning (e.g., Happe, Booth, Charlton, & Hughes, 2006). The largest proportion of research examining self- regulation and struggling learners concentrates on students identified with learning disability, attention-deficitlhyperactivity disorder, and autism (e.g., Denckla, 1996; Hill, 2004). It is increasingly evident that impaired self-regulation processes are a shared characteristic across these students. For example, impaired executive functions, or the cognitive processes responsible for managing goal-directed activity, is a prevalent finding 3across students who qualify for special education (Pennington & Ozonoff, 1996; Willcutt, Pennington, Olson, Chabildas, & Hulslander, 2005). The number of students who exhibit difficulty with self-regulated learning is large. There are over 6 million students enrolled in special education, and the number of students in special education has increased by over 20% in the last 10 years (National Center of Educational Statistics, 2006). According to the U.S. Department of Education, nearly 3 million students at the secondary education level receive special education services under IDEA legislation, half of whom are identified with learning disabilities, ADHD ("Other health" category), and autism (Office of Special Education Programs, 2007). Poor self-regulated learning is magnified as students advance to secondary education. The transition out of elementary school necessitates the use of more complex organizational schemes to manage increased academic workload (Ylvisaker & DeBonis, 2000). Additionally, students must cope with increased expectatiQns of autonomy and varied teachers' expectations demanding implementation of flexible cognitive- motivational strategies (Lord, Eccles, & McCarthy, 1994; Rudolph, Lambert, Clark, & Kurlakowshy, 2001). An example of an academic skill that requires self-regulation is assignment management. The vast majority of students at the secondary education level are expected to complete assignments outside the classroom independent of teacher support (Gajria & Salend, 1995; Struyk & Epstein, 1995). Students who struggle academically often lack organizational skills and strategic approach to managing 4assignments resulting in excessive time spent on assignments, lost materials, and negative attitude toward out-of-class academic work (Bryan, Burstein, & Bryan, 2001). Independent assignment completion is particularly critical to academic success (Cooper, Robinson, & Patall, 2006). The majority ofteachers assign homework and use assignment completion for evaluation purposes (Munk et at, 2001; Struyk & Epstein, 1995). In addition, completing assignments independently supports mastery of academic concepts. Research demonstrates students who complete more assignments outside the classroo~ achieve higher academic success (Cooper et aI., 2006; Trautwein & Koller, 2003; Trautwein, Koller, Schmitz, & Baumert, 2002). The proposed mechanism explaining the effect of assignment completion is that students benefit from more time spent engaged in academic concepts (Cooper, Lindsay, Nye, & Greathouse, 1998; Trautwein, Ludtke, Schnyder, & Niggli, 2006). A Potential Solution Researchers grounded in social-cognitive theory have developed and evaluated interventions targeting self-regulation to improve performance on complex academic tasks such as assignment completion. The most documented approaches for teaching students to self-regulate their learning fall under the broad label of self-management strategies (Belfiore & Hornyak, 1998). Self-management approaches focus on helping students actively reflect on their own learning using techniques such as "self-instruction," self-evaluation, and following task-specific checklists (Belfiore & Hornyak, 1998; Reid, 1996). These approaches have been utilized with students who qualify for special education with positive effects seen on academic tasks such as reading comprehension, 5homework completion, and solving math problems (e.g., Shimabukuro, Prater, Jenkins, & Edelen-Smith, 1999; Uberti, Mastropieri, & Scruggs, 2004). Two recent systematic reviews of self-management literature documented the effectiveness of these interventions for students who qualify for special education. Reid and colleagues (2005) evaluated the effects of four strategy instruction approaches (self- monitoring, self-monitoring plus reinforcement, self-reinforcement, and self- management) on academic behavior for students identified with ADHD. Mooney and colleagues (2005) published a similar synthesis of self-management interventions for students identified with emotional behavioral disorders. Both of these meta-analyses demonstrated consistent positive effects of self-management strategy instruction on a variety of academic skills including reading comprehension, on-task behavior, and academic productivity (Mooney, Ryan, Uhing, Reid, & Epstein, 2005; Reid, Trout, & Schartz, 2005). While evidence supporting the use of self-management approaches to support academic tasks is increasing, there are gaps in the literature. For example, while it is evident that teaching students self-regulation strategies positively affects the target behavior (e.g., Graham, Harris, & Mason, 2005; Harris, Friedlander, Saddler, Frizzelle, & Graham, 2005), little is known about the broader impact of these approaches on other cognitive-motivational domains such as executive functions. Also, since self-regulation interventions are applied in school settings, implementation is a vital aspect of any approach but little is known about key aspects that make strategies easy to apply in classroom se~tings. 6Purpose Statement The purpose of this study was to examine the effects of an assignment completion strategy on academic, cognitive, and motivational variables for students who qualify for special education. This study extended the literature base in two ways. First, this study evaluated both the direct and indirect effects of self-management strategy instruction. Second, this study validated practices related to strategy implementation including teacher preference for presentation mode and teaching strategies at the classroom versus individual level. T~e rationale, procedures, and results for this study are outlined in the following chapters. Chapter Two ofthis dissertation provides an overview of effective learning processes based on social-cognitive theory. The importance of managing academic tasks relative to academic achievement is addressed followed by a description of the target population for this study. Next, a review of current interventions targeting self-regulation is provided. Chapter Three outlines the research methods utilized in this study including descriptions of intervention materials, measures, procedures, and data analysis plan. Chapter Four provides a summary of the results, and Chapter Five concludes this dissertation with a discussion of the findings. 7CHAPTER II REVIEW OF THE LITERATURE Assignment Management in Secondary Education Requiredfor Advancement When students advance beyond elementary school, key cognitive-academic needs emerge (Rudolph et aI., 2001). These requirements include more advanced organizational schemes, planning skills, and abstract thinking (Ylvisaker & DeBonis, 2000). Students must function more autonomously in secondary education and take responsibility for their own learning (Valiente, Lemery-Chalfant, Swanson, & Reiser, 2008). Successful advancement requires stude!1ts to develop effective strategies for managing academic tasks. A critical required academic task in secondary education is completing assignments. Students are expected to complete assignments at home and in class independently and assume more personal responsibility for their learning (Harter, 1992). Most students are assigned homework several nights a week at the secondary education level (Munk et aI., 2001; Struyk & Epstein, 1995). Completing out-of-class assignments leads to increased academic competence, and homework is often included in classroom evaluation (Cooper et aI., 2006). The primary benefit of out-of-class assignments is providing students opportunities to spend time individually engaged in academic concepts (Cooper et aI., 82006). Trautwein and colleagues (2002) conducted a longitudinal study of 1,976 students in Germany examining the relationship between student characteristics, classroom characteristics, homework practices and academic achievement. They reported that students who spent time engaged in out-of class assignments, especially when the tasks were challenging, obtained higher academic achievement. Homework Management Homework is defined as tasks assigned by teachers to learners with the intent the assignment is completed outside the classroom without teacher assistance (Cooper, 1989). There are two general reasons for assigning homework. First, teachers design homework to provide students opportunity to review and practice material that has been learned in school. Second, assignments can provide opportunity to amplifY or enrich previously learned material (Hong & Milgram, 1999). The tasks are usually more time consuming than in-class work and focus on helping students process the material (Hong & Milgram, 1999). Assignments may include tasks that are completed daily (e.g., math worksheets) ~r longer projects targeting information synthesis (e.g., book reports, term projects, or science experiments). Numerous studies have established the importance of time spent engaged in independent studying on academic outcome; students must independently manage these assignments to benefit from the additional practice (Cooper, Jackson, Nye, & Lindsay, 2001; Cooper et aI., 2006). Homework completion is a process that requires students to manage an array of personal and contextual factors. Cooper's (1989) homework process model outlines five factors that influence the assignment completion process: exogenous factors, assignment 9characteristics, initial classroom factors, home-community factors, and classroom follow- up. Exogenous factors are student characteristics (i.e., ability, motivation, and habits), subject matter, and grade level. Assignment characteristics include length, purpose, and skill area. Initial classroom factors are provision of materials and facilitators (i.e., suggested approaches, links to curriculum, and "other rationales"). Home-community factors are competitors for student time (e.g., sports, television, chores), home environm~nt (e.g., space, noise, materials), and others involvement (i.e., parents, siblings, others). Finally, classroom follow-up factors are feedback, testing of related content, and use in class discussion (Cooper, 1989; Cooper & Nye, 1994). Managing homework involves self-directed cognitive and motivational processes (Cooper, 1989). Students must have the capacity to recall assignments, gather materials, initiate assignments, and remain engaged without direct teacher supervision. Completing assignments also requires motivation since there is often competition for students' time and attention (Wolters, 1999). In-class Assignment Management In addition to homework, students are more responsible for regulating their classroom learning as they advance academically (Paris & Paris, 2001). There is more autonomy in secondary education classrooms that necessitates more student control over the learning process (Valiente et aI., 2008). Students who approach their classroom learning systematically are more likely to succeed academically (Pintrich & DeGroot, 1990; Zimmerman, 1990). 10 In-class work is most commonly prescribed to help students practice concepts already learned in the curriculum (Struyk & Epstein, 1995). Classroom activities commonly include writing exercises, drill work, and reading comprehension tasks (Kameemii & Simmons, 1990). To complete these tasks independently, students must be skilled at setting task-oriented goals, selecting and applying cognitive-academic strategies, and self-monitoring their learning process in the classroom (Boekaerts & Cascallar, 2006; Boekaerts, Koning, & Vedder, 2006; Reid et aI., 2005). The transition to secondary education is accompanied with increased expectations for independent academic functioning. Students who succeed aca~emically, manage both in-class and out-of-class academic tasks efficiently (Cooper et aI., 2006). There are many interacting variables that affect independent assignment completion, hence powerful learning processes are necessary to accomplish the task. The next section of this paper provides a brief overview of the cognitive-motivational processes involved in typical learning. Social Cognitive Theory: A Model ofLearning The leading theory explaining how students acquire complex tasks such as assignment management is social cognitive theory. Social cognitive theory provides a comprehensive framework explaining learning (Bandura, 1986). According to this theory, the extent to which learning takes place is dependent on the interaction of behavioral, environmental, and cognitive factors. 11 Role ofBehavioral Principles in Learning Behavior in learning contexts refers to both level of effort and use of strategies (Boekaerts & Cascallar, 2006). Behavioral influences are best understood from traditional behaviorist theory, which attributes learning to positive reinforcement for academic performance. Students are more likely to put more effort into tasks they previously found rewarding. Behavior that supports learning is therefore shaped by reinforcement (Wolters, 1999). Students who learn and use learning strategies successfully learn to apply similar strategies in other contexts (Hattie, Biggs, & Purdie, 1996). One effective academic strategy is note taking (Ries et aI., 2000). Students learn the benefit of note taking in one class by experience, and this shapes their use of note taking in other classes. Role ofEnvironment in Learning Environment plays a significant role in learning, according to social cognitive theory. Cogriitive development is stimulated by the presence of sensory input during the early stages of development (Novak & Pelaez, 2004). Later in life, other environmental factors including access to books or other educational materials, stable horne environment, and positive peer relationships all have significant impacts on cognitive, personality, and academic development (e.g., Hallahan et aI., 2005; Thompson, 1999; and Spear, 2000). Access to various experiences is important to development, but the mechanism through which these experiences contribute to knowledge acquisition is a matter of debate. 12 Social-cognitive theorists suggest one way environmental factors impact learning and development is through observational learning. Observational learning is the idea that children expand their knowledge of the world by observing others perform either a task they would either like to learn or have never done before (Novak & Pelaez, 2004). Observational learning has been credited as one mechanism through which children learn behaviors that are rarely taught explicitly, such as aggression (e.g., Bandura, 1977). In academic arenas, students' knowledge and skill acquisition develop by through observation and shared experiences with experts (Schunk & Zimmerman, 2007; Zimmerman, 1990). Peer or teacher models are effective observational learning tools because they. highlight key salient features and provide instruction in a relevant context (e.g., Kameenui & Simmons, 1990). Using academic strategies, managing homework assignments, and mastering math concepts are some skills research suggests develop by observing expert models (Boekaerts & Como, 2006). Role ojCognition in Learning Social cognitive theory extends prior behaviorist theories by expounding on the role of cognition in learning (Zimmerman, 1990). An elemental understanding of social cognitive theory is the notion that humans have certain cognitive "capacities" (Bandura, 1986). One such capacity is "self-direction". Though individuals learn through vicarious experience, they have control over their actions and are able to ditect their own learning. Three additional capacities highlighted by Bandura are the capacity to self-motivate, self- reflect, an.d self-regulate. 13 Capacity to sel.fmotivate. Motivation is the term applied to an individual's willingness to learn or engage in new activities (Wolters, 1999). Boekaerts and Como's (2005) dual processing model is an illustration of how motivation processes affect student behavior in academic contexts. This model dichotomizes motivation according to two personal pathways. The "growth track" is a process of top-down motivation where an individual chooses to pursue learning goals because of personal interest or expected satisfaction. These learners apply effort because they value the goal ofleaming. The "well-being" t~ack is a bottom-up process whereby motivation is derived from environmental stimuli. Students focus on avoiding unfavorable learning contexts or situations. In this case, student effort is concentrated on not completing assignments or engaging in academics because their perceived well-being is dependent on avoiding the task (Boekaerts & Como, 2005). The relationship between academic motivation and achievement has been demonstrated for students in general education settings (Pintrich & DeGroot, 1990; Wolters, 199.9). In a retrospective analysis of the National Education Longitudinal Study (NELS; 1988), Singh and colleagues (2002) used structural equation modeling to demonstrate the relationship between motivation and achievement in math and science for over 2~,000 eighth grade students. Their results show statistically significant correlations between motivation variables and achievement in math and science (Singh, Granville, & Dika, 2002). Motivation is important in assignment management because external motivators (i.e., teachers or parents) are typically absent. Under this independent condition, 14 Trautwein and colleagues (2003) link Expectancy-Value theory with assignment completion research to explain individual student motivation. This theory relates a student's expectation of success and value ofthe task to homework behavior. Students who expect to do well on a task are more likely to spend time on assignments. Likewise, students who enjoy doing certain activities or who value doing well on specific tasks are also more likely to persist with challenging work (Trautwein & Koller, 2003). Capacity to selfreflect. According to social cognitive theory, a potent influence on learning is the capacity for self-reflection. The distinctly human capacity for self-awareness enables individuals to analyze their experiences and thought processes (Prigatano, 2005). Self- reflection describes the ability to analyze how effectively thoughts or actions help an individual complete daily activities. A cognitive process associated with self-reflection is meta;cognition. Metacognition is known colloquially as the ability to think about thinking. A more formal definition of metacognition is a " ... self-evaluative and regulatory function that encompasses self-appraisal or monitoring of one's own cognitive processes ... " (Levin & Hanten, 2005). Metacognition allows students to evaluate the effectiveness of their approach to learning and monitor their progress on academic tasks (Zimmerman, 1998). A common metacognitive process important for academics is self-monitoring (Reid, 1996; .Reid et aI., 2005). Self-monitoring involves periodic checks of both thought processes and academic productivity. Students who self-monitor their behavior and learning recognize when their initial expectation (i.e., task requirements or knowledge of 15 the subject) is inaccurate and adjust their thought processes accordingly (Boekaerts, 1997). Efficient learners evaluate their own cognitive approach during academic tasks (e.g., using learning strategies) based on how effectively the strategy facilitated their thinking (Zimmerman, 1998). Capacity to sel.fregulate. Self-regulation refers to the capacity for controlling behavior using internal standards. Self-regulation occurs when a person evaluates their actions or thoughts and adjusts based on discrepancies between the actual and ideal behavior (Bandura, 1986). Self-regulation is the process that allows for cognitive flexibility in the face of environmental or motivational barriers (Boekaerts et aI., 2005). When describing self- regulation, researchers often provide behavioral examples (see review in Zimmern1an, 1990). For instance, students self-regulate their academic behavior by delaying gratification. Every student faces the dilemma of choosing between immediate gratification (i.e., hanging out with friends) or delaying gratification by pursuing activities that have long-term beneficial consequences (i.e., doing homework). The cognitive processes closely linked to self-regulation are known as executive functions. Executive functions are processes essential for managing complex cognitive tasks. These processes are often personified as "managers" because they direct subordinate domains such as memory, language, and motor skills.for accomplishing goals (Levin & Hanten, 2005; Miller & Cohen, 2001). Common behavioral correlates to normal executive functioning include starting or stopping behavior, task persistence, planning, problem solving, and organization (Sohlberg & Mateer, 2001). These behaviors 16 indirectly support learning since they provide the foundation for attending to all academic tasks. Students who self-regulate their academics manage these processes using cognitive strategies. Self-instruction is an example of self-regulation cognitive strategy whereby students generate a set of instructions for completing a task (Harris, 1990). They repeat the instructions internally to help guide them through multi-step tasks. Self- instruction has been used to help students problem solve difficult academic tasks such as multi-step equations in math (Wood, Rosenberg, & Carran, 1993). Self-instruction strategies are also effective for managing maladaptive thought processes and emotions which interfere with learning, such as anxiety (Kamann & Wong, 1993). ImpactofseljCejTTcacy. If students believe they are good at a task, they will exert more effort on the activity (Bandura, Barbaranelli, Caprara, & Pastorelli, 1996). The. role of personal belief and efficacy in academics is a central theme in social cognitive theory and has been elaborated on by Bandura (1996) and others. Self-efficacy can be defined as an individual's belief that they have the ability to control their level of functioning in response to environmental demands (Bandura et aI., 1996). Academic self-efficacy develops with experience. Positive learning experience (e.g., mastering a concept) is the strongest predictor of high academic self-efficacy (Usher & Pajares, 2006). Perceptions of efficacy influence the learning process. Applied to academic contexts, Pajares (2003) summarizes the role of self-efficacy, " ... ·(efficacy) affect(s) what students do by influencing the choices they make, the effort they expend, the 17 persistence and perseverance they exert when obstacles arise, and the thought patterns and emoti.onal reactions they experience." Perceptions of efficacy account for students' initial willingness to engage in new or difficult tasks (Vollmeyer & Rheinberg, 2006). Students who feel they have a higher initial "probability of success" on a task are more likely to complete the task effectively than students who do not (Vollmeyer & Rheinberg, 2006). Students may have high self-efficacy in certain academic domains such as math, reading, writing (Pajares, 2003). Students can also have perceptions of self-efficacy towards learning in general (Bandura, 1996; Usher & Pajares, 2003). Students with high sdf-efficacy, who strongly believe in their ability to control their own 'learning, consistently exhibit higher academic achievement than students with low efficacy (Bassi, Steca, Fave, & Caprara, 2007). Bassi and colleagues (2007) found that students with high academic self-efficacy devoted more time to academics outside of school and managed stressful academic situations (e.g., studying for an exam) with less anxiety than students with low self-efficacy. Also, students with higher academic self- efficacy are more likely to persevere during difficult tasks or tasks for which they have little prior experience (Schunk, 1989). Self-Regulated Learning Self-regulation is a psychological construct with broad application. It has been examined from cognitive (Baddeley, 1996), behavioral (Carver & Sheirer, 1998), and educational (Zimmerman, 1990) perspectives. The importance of self-regulation for functioning in every life domain renders this a critical concept. The current work is interested in .self-regulation as applied to leaming. 18 Definition ofSe?f-Regulated Learning Examining self-regulation in educational context refers to understanding how and why student~ govern their own learning (Zimmerman, 1990). As Zimmerman (1990) stated, "The task of exampling why and how students assume resflonsibility for regulating their own acquisition of knowledge and skill, often in the face of obstacles, provides one of the most demanding tests for theories of self-regulation" (p. 184). A recent review of self-regulation in learning provides a basic overview of the main premises (Boekaerts & Como, 2005): All theorists assume that students who self-regulate their learning are engaged actively and constructively in a process of meaning generation and that they adapt their thoughts, feelings, and actions as needed to affect their learning and motivation. Similarly, models assume that biological, developmental, contextual, and individual difference constraints may all interfere with or support efforts at regulation. Theorists are in agreement that students have the capability to make use of standards to direct their learning, to set their own goals and sub- goals. Finally, all theorists assume that there are no direct linkages between achievement and personal or contextual characteristics; achievement effects are mediated by the self-regulatory activities that students engage to reach learning and performance goals. (p. 201) It is critical to remember that social-cognitive theorists view self-regulation in academics as a skill (Schunk & Zimmerman, 1997). As with other academic skills, students learn to hone the processes through experience. With respect to educational contexts, self-regulation develops largely through peer and teacher models, instructor guidance, an? direct instruction (Boekaerts, 1997; Shunk &Zimmerman, 1997). Boekaerts (1997) identified six regulatory skills students learn to apply in academics. These skills are divided into cognitive self-regulation or motivational self- regulation components. These six components are interwoven, affecting both student 19 effort and task performance during learning (Boekaerts, 1997). There are three cognitive self-regulation components in learning. The first is the capacity to acquire knowledge and access prior knowledge. Cognitive self-regulation also includes control of cognitive strategies such as rehearsal, selective attention, and decoding. Finally, self-regulation in learning necessitates cognitive regulatory strategies such as progress monitoring, planning, and evaluating goal achievement. These three behaviors are evidence of self- regulation over cognitive processes associated with learning (Boekaerts, 1997). The other three components in Boekaerts' (1997) self-regulation model pertain to control over motivational factors. There are three motivational self-regulation components students manipulate while learning. The first is meta-cognitive knowledge and motivational beliefs. Examples of these include beliefs or attitudes pertaining to the task and belief in personal capacity to succeed at the task. Second, motivational self- regulation includes using motivation strategies such as using social resources or creating a learning intention. Finally, students apply motivational regulatory strategies such as coping with stressors and considering past successes with similar tasks (Boekaerts, 1997). The Process ofSelf-Regulated Learning Boekaerts' model provides a framework for understanding the self-regulatory processes involved in learning. The term self-regulated learning has evolved to describe student application of cognitive-motivational processes toward the goal of academic achievement (Pintrich & DeGroot, 1990). Self-regulated learners are able to make adjustments (e.g., increase motivation, reset goals, etc.) to improve performance due to regular performance checks (Zimmerman, 1998). 20 Zimmerman (1998) outlines a cyclical process of self-regulated learning in three phases: (1) forethought, (2) performance, and (3) self-reflection. "Forethought" encompasses influential processes that "set the stage" for learning. Within this phase are the sub-processes of goal setting, strategic planning, self-efficacy beliefs, goals orientation, and intrinsic interest. "Performance" or "volitional control" describes processes students control during learning. These include attention, self-instruction, and self-monitoring. Finally, the "self-reflection" phase describes the process of learner reaction to the experience. "Self-reflection" includes the sub-processes of self-evaluation, attributions, self-reactions, and adaptability. Students who self-regulate their learning repeatedly engage in this open-ended process. The Strategies ofSelf-Regulated Learning Students' capacity to self-regulate their learning depends largely on their ability to utilize cognitive strategies (Boekaerts, & Cascallar, 2006; Zimmerman, 1990). Cognitive strategies help with information processing, retention, and retrieval (Gazzaniga, Ivry, & Mangun, 2002). One example of a cognitive strategy used for reading is the key word search strategy. Students can orient themselves to the content of reading passages before engaging in the text by scanning for key words. This effectively reduces the cognitive load during the task enabling students to spend more effort on comprehension. Other reading strategies include self-questioning for comprehension and decoding (Johnson, Graham, & Harris, 1997; Taylor, Alber, & Walker, 2002). A number of strategies target memory processes. Common memory strategies are rehearsal and, mnemonics. Self- 21 regulated learners evaluate how effectively each strategy helped them accomplish their goal and students develop schemas for dealing with similar tasks (Zimmerman, 1998). The most common strategies defined in self-regulated learning literature are metacognitive strategies (Boekaerts & Cascallar, 2006; Zito, Adkins, Gavins, Harris, & Graham, 2007). The National Research Council (NRC) published a review of research examining effective learning principles. One of the three critical learning processes they identified was the capacity to self-monitor learning (NRC, 2005). Students use self- monitoring strategies to evaluate a number of aspects relative to learning such as motivational. influences, strategy effectiveness, attention, progress and comprehension (Harris et aI., 2006; Reid, 1996; Zimmerman, 1998). Other metacognitive strategies utilized by self-regulated learners include strategic planning, self-instruction, visualizing goal attainment, and self-evaluation (Zimmern1an, 1998). According to self-regulated learning theorists, metacognitive strategies are subject to volitional control (Como, 2001). Students choose when and which strategies to apply based on context. With experience, students apply strategies automatically, which is referred to as "routinization" (Zimmerman, 1998). Some suggest that automatic strategy use signifies 'less demand to self-monitor as the learning process becomes routine. Students who attribute their success to certain strategies will routinize those strategies quickly (Zimmerman, 1998). Struggling Learners and Self-Regulation Studies show that students who exhibit characteristics of self-regulated learning are successful learners (see review in Zimmerman, 1998). For example, Feldman and 22 colleagues (1995) found that students who utilized self-regulatory behaviors completed more classroom activities than their peers who did not. Similarly, students who reported optimizing their environment and setting personal goals for completing homework were more productive students than their peers (Feldman, Martinez-Pons, & Shaham, 1995; Gajria & Salend, 1995). Self-regulated learning behaviors predict both overall academic achievement (i.e., grade point average) and concept mastery (Schunk, 1998; Wolters, 1999). However, a large proportion of students are unable to manage academic tasks in relation to increased expectations at the secondary education level. The following overview describes these students and their difficulties self-regulating the learning process. Evidence ofImpaired Self-Regulation with Struggling Learners According to the U.S. Department of Education, nearly 3 million students between the ~ges of 12 and 17 receive special education services under IDEA legislation (Office of Special Education Programs, 2007). These students not only struggle with academic concepts, but also have difficulty managing the more complex learning environment in secondary education (Lord et aI., 1994; RUdolph et aI., 2001). A striking commonality among these students is poor self-regulation. Struggling learners and cognitive control. There is a growing body of research demonstrating pervasive deficits in executive functions across students who qualify for special education under IDEA legislation (Denckla, 1996; Hill, 2004; Pennington & Ozonoff, 1996; Rucklidge & Tannock, 2002; Willcutt, Doyle, Nigg, Faraone, & Pennington, 2005). Evidence suggests students in six 23 of the thirteen IDEA-defined disability categories exhibit some form of executive function impairment, a critical cognitive component of self regulated learning (Anderson & Catroppa, 2005; Geurts, Verte, Oosterlaan, Roeyers, & Sergeant, 2004; Hill, 2004; Olvera, Semrud-Clikeman, Pliszka, & O'Donnell, 2005; Shear, DelBello, Rosenberg, & Strakows~i, 2002; Willcutt, Pennington et aI., 2005). Intact executive functions are required in each phase of Zimmerman's (1998) cyclical self-regulated learning model: forethought, performance/volitional control, and self-reflection. Happe, Booth, Charlton, and Hughes (2006) compared the executive function profiles of children and adolescents with attention deficit-hyperactivity disorder (ADHD), autism, and matched typically developing peers. The researchers used a variety of cognitive tests to measure the domains of response selection, cognitive flexibility, and planning/working memory. Group-wise and post hoc statistical analysis revealed'that students with ADHD performed significantly worse on measures of response selection and planning/working memory compared to students with autism or the typically developing group. Seidman and colleagues (2005) reported the results of a study examining higher- order cognitive skills in adolescents identified with ADHD. The sample of204 adolescents with ADHD and 109 matched control peers were compared based on performance on a battery of neuropsychological tests designed to sample attention, planning, organization, response inhibition, set shifting, and memory. Adolescents with ADHD scored lower on all tests compared to the control group, with levels reaching 24 statistical significance on measures of response inhibition and set shifting (Seidman et aI., 2005). One implication of reduced cognitive control for students who struggle academically is poor use of learning strategies. Students with learning disabilities (LD) reported lower metacognitive strategy use and lower perceived ability to self-regulate their learning compared to their peers (Klassen, Krawchuk, Lynch, & Rajani, 2008). They are also less skilled at self-monitoring (Harris et aI., 2005), strategic planning (Sikora, Haley, Edwards, & Butler, 2002), and inhibiting responses to distracting stimuli (van Mourik, Oosterlaan, & Sergeant, 2005). These traits restrict volitional control over strategy selection and implementation in the self-regulated learning cycle (Zimmerman, 1998). Struggling learners and motivational capacities. There is a growing body of evidence that demonstrates motivation differences between higl) achieving and low achieving students (Fulk, Brigham, & Lohman, 1998; Pintrich, Anderman, & Klobucar, 1994). Students with LD are often motivated to avoid work because of prior experiences of failure (Boekaerts et aI., 2006; Fulk et aI., 1998; Pintrich & DeGroot, 1990). For example, one recent study of college students revealed that those identified with LD spent significantly more time in behaviors identified as "procrastinating" (Klassen et aI., 2008). There is also evidence that low achieving students are less skilled at managing their academic motivation, choosing "maladaptive" motivators more often than high achieving students (Anderman & Young, 1994). These 25 students are more likely to find motivation avoiding academic tasks and pursuing the "well-being" track described by Boekaerts and Como (2005). Another area where struggling students differ from high achieving students is academic self-efficacy. There is substantial evidence that struggling students lack confidence in their ability to succeed academically (Bandura et ai., 1996; Schunk, 1989). These students are more likely to under- or over-estimate their ability to complete tasks resulting in less persistence in the face of adversity (Bassi et ai., 2007; Klassen & Lynch, 2007; Pajares, 2003). These motivational variables are linked to all three phases of the self-regulated learning cycle (Zimmerman, 1998). Impact ofpoor self-regulation on assignment management. Impairments in the cognitive control and motivational processes necessary for self-regulation negatively impact assignment management. Bryan, Burstein, and Bryan (2001) published a review of assignment management and completion problems for students identified with LD. Their summary reveals a pattern of assignment completion difficulties. For example, students with learning disabilities spend more time on assignments due to poor organization and time management compared with peers. In addition, students who struggle academically lose materials and forget assignments more frequently resulting in less time engaged in academics (Bryan et ai., 2001). Gajria and Salend (1995) published the results of a study examining assignment completion problems for middle school students identified with LD. Using student completed questionnaire data, students with LD report more problems starting and completing homework than a group of matched control students. Examples of se1f- 26 reported problems that students with LD report and their peers do not included "forget to bring assignments back to class", "I don't finish my work", and "when I find an assignment too hard, I stop working on it" (Gajria & Salend, 1995). Using a parent and teacher questionnaire, Soderlund and Bursuck (1995) published a review of assignment completion problems at home and in school for students identified with behavior disorders. Parents and teachers completed the Homework ~roblemChecklist (HPC; Anesko and colleagues, 1987) for 82 students with behavior disorders and 114 students without disabilities. The results revealed statistically significant differences in perceived behavior differences between the two student groups. The three items with the largest between group differences identified by teachers were "easily distracted by noises", "responds poorly when told to do work", and "puts off doing work". The two items differing most on parent reports were "procrastinates" and "easily distracted" (Soderlund, Bursuck, Polloway, & Foley, 1995). Epstein and Polloway (1993) published a comparison of special education teacher, general education teacher, and parent perception of student assignment management. The students in this study were identified with learning disabilities, emotional behavior disorders, and age-matched peers. Analyzing questionnaire data, the results re,:,ealed statistically significant differences between students identified with disabilities compared to the control group. There were no statistically significant differences between students in the two disability groups. Similar to previous studies, the researchers reported problems related to poor assignment management including failure 27 to remember assignments, relying on others to help start and complete assignments, and losing focus (Epstein & Polloway, 1993). Intervention for Deficits in Self-Regulation The literature documents the pervasiveness of self-regulation deficits across a large proportion of students with special learning needs. The barriers imposed by poor self-regulated learning restrict student capacity to engage in academic tasks independently. Given the impact of these deficits on academic success, a number of interventions have been proposed targeting self-regulation processes in struggling students. Common interventions fall into three very broad domains: pharmacological, behavioral, and cognitive. Pharmacological interventions utilize medications that interact with neurochemical exchange resulting in improved capacity to engage in academic tasks. Students identified with ADHD or conduct disorder/emotional behavioral disorder are the most common recipients of pharmacotherapy (e.g., Biedennan et aI., 1999; Kratochvil et aI., 2002): Commonly prescribed medications include Clonidine (Steingard et aI. 1993) and Methylphenidate (Kratochvile et aI., 2002). Typically they are prescribed to target self-regulatory processes including attention and response inhibition (Spencer et aI., 1996). While medication effects can be quite positive for students with ADHD or conduct disorder, a large number of students who struggle with self-regulation are not candidates for pharmacotherapy. Behavioral interventions for managing academic tasks have typically targeted student behavior that supports self-regulated learning. Using observational techniques 28 such as functional behavior analysis, interventionists identify problem behaviors and their precursors to identify mitigating factors (Sugai, Homer et aI., 2000). By altering antecedents or reinforcing target behaviors and disciplining disruptive or non- constructive behavior, behavioral interventions can reduce detrimental or increase facilitating learning behavior (Sugai, Sprague, Homer, & Walker, 2000). While behavioral approaches have been shown to be effective in preventing and shaping behavior problems (e.g., Marquis et aI., 2000), they do not address the cognitive capacities linked with self-regulated learning or specific academic skills. The most promising line of research focusing on improving self-regulated learning examines the impact of teaching cognitive strategies (Zimmerman, 1998). There is increas~ng evidence that the processes supporting self-regulation in academic settings are malleable (Graham & Harris, 1993; Margolis & McCabe, 2006). Student capacities for self-regulation and self-reflection can be modified by direct instruction or teacher modeling (Boekaerts & Como, 2005; Schunk, 1998; Zito et aI., 2007). Specifically, research supports teaching contextually relevant metacognitive strategies to help students attend to the 'cyclical process of self-regulated learning (Boekaerts & Cascallar, 2006). There are several benefits ofthe cognitive strategy approach to helping students self-regulate assignment management. First, interventions can be designed to target specific difficulties in assignment management. For example, Hughes and colleagues (2002) taught a group of students to manage their homework assignments independently using a combination external organizer and metacognitive approach. Second, cognitive strategies can be designed for applicability across a range of students. For instance, 29 Peterson and colleagues (2006) taught a group of students with diverse backgrounds the same self-monitoring strategy. The results were improved behavior regulation in the classroom across students. Third, the strategy complexity can be modified according to student need and ability. The Hughes (2002) study described a complex homework management system, while other studies evaluate simple monitoring systems such as asking students to graph the number of problems completed during class (e.g., Shimabukuro et aI., 1999). The most widely reported intervention approaches have been referred to as self- management strategies (Belfiore & Hornyak, 1998). This review will address two general self-management approaches: self-monitoring strategies and self-instructional strategies. Also, a curricular approach to increasing academic self-regulation will be reviewed, as will two examples of out-of-class assignment management strategies. Self-Management Strategies In their review of instructional strategies and self-regulated learning, Belfiore and Hornyak (1998) outline an array of cognitive strategies that support "self-management" in academic contexts. Self-management processes included self-monitoring, self- instruction, informed decision making, problem solving, self-reinforcement, and self- evaluation. The following review provides examples of strategies targeting self- management that have been implemented with students who struggle with self-regulating academic tasks. 30 Self-monitoring strategies. Self-monitoring strategies have been taught to students with a wide range of abilities functioning in diverse academic contexts. Self-monitoring procedures involve student self-~valuations of whether the target behavior occurred or did not occur (Belfiore & Hornyak, 1998). The evaluation is made either during or after task completion and the student records the results for later review (e.g., Shimabukuro et aI., 1999). Data are often recorded on task-specific checklists and the frequency of data recording varies according to the frequency of the behavior (Belfiore & Hornyak, 1998). Harris and colleagues (2005) reported the results of a self-monitoring intervention for studying behavior in six students identified with ADHD. The target behavior was defined as students independently gathering materials to study spelling words and completing a short spelling exam while remaining on task throughout the procedure. The strategy taught to students consisted of a self-questioning routine ("Was I paying attention?") triggered by an audible tone played through headphones at varying intervals. The studept would then record, "yes or no", whether he/she was paying attention on a graph. The student also recorded the accuracy of their spelling test. The results revealed positive effects for both remaining on task and improving spelling (Harris et aI., 2005). Gureasko-Moore and colleagues (2007) taught six 11-12 year olds identified with ADHD, a self-monitoring strategy to help manage "classroom preparation behavior". They created a checklist defining four expected behaviors (e.g., student had pen or pencil on desk) and asked students to indicate yes/no whether they accomplished the tasks. The students checked in with thE researchers every day after class to review their self- 31 monitoring log. Following implementation of the self-monitoring procedure, the percentage of classroom preparation behaviors increased across all six subjects as measured by teacher report. The researchers then faded the checklist, instead using student interviews to facilitate self-monitoring. Eventually, the researchers faded the interviews, but student classroom preparation maintained a high level across students (Gureasko-Moore, DuPaul, & White, 2007). These are two examples of self-monitoring strategies applied to different contexts. There are other studies that utilize ~ssentially the same procedures to target homework completion (e.g., Shimabukuro et aI., 1999), improve math probl~m solving (e.g., Uberti et aI., 2004), and increase academic productivity (e.g., Carr & Punzo, 1993) for students with LD and ADHD, LD only, and behavioral disorders respectively. Reid (1996) published a systematic review of intervention studies using self- monitoring approaches for students with learning disabilities. The participants in the 23 studies more mostly between the ages of 9 and 11, although the range was 7 to 18 years of age. In the research, which dated from 1979-1996, all but six studies were conducted in a resource. or self-contained setting. Reid dichotomized the approaches as self- monitoring of attention or self-monitoring of performance strategies. While no objective metrics were provided (i.e., effect sizes), the author concluded that based on study results, the effects of self-monitoring are most robust for on-task behavior and academic accuracy. Also, the author stated, "(The effects) ... have been demonstrated across differing age levels and instructional settings." (p. 318, Reid, 1996) There was no report of generalization or maintenance of these effects in the review. 32 Se If-instruction strategies. Self-instruction is a self-management strategy that utilizes sub-vocal recitation of a series of steps to solve a problem (Belfiore & Hornyak, 1998). The use of self- instruction as a teaching approach increases the likelihood that students will learn independent self-management since the process lacks external mediation (Belfiore & Hornyak, 1998). Wood and colleagues (1993) published an example of a self-instructional strategy targeting math problems for nine students between the ages of 8 and 11 identified with learning disabilities. The strategy was taught in therapy rooms adjacent to the classroom and utilized a teacher-recorded script played through headphones the students listened to while working through the problem. For example, the first statement read, "First, I have to point to the problem." The headphones were faded and the students were expected to whisper or repeat the sequence sub-vocally. The results of the study indicated higher percentage of problems attempted and completed accurately for all nine students (Wood et aI., 1993). The results generalized to the faded headphone phase for six of nine students. This example illustrates the typical self-instructional process. Self-instruction may stand alone as a strategy, but is frequently incorporated in intervention packages (e.g., Johnson et aI., 1997). Self-instruction strategies are also useful for helping students manage their state of mind whil.e engaged in academics. For example, Kamann and Wong (1993) published a report examining the effects of teaching a self-talk strategy to students with LD on math performance. The self-instruction strategy targeted feelings of anxiety and confusion 33 while working on problems. The students were taught to periodically complete a self-talk routine including statements targeting assessment ("What do I have to do?"), recognizing negative thoughts ("OK, I feel worried and scared."), controlling thoughts ("Take it step by step."), and reinforcing ("Good for me.") while working on math problems in class. The results of teaching students to assess their mental state revealed a statistically significant increase in math problem productivity and accuracy for students identified with LD (Kamann & Wong, 1993). Syntheses ofselfmanagement research. Two recent reviews of self-management interventions for students who qualify for special education have been published. Reid and colleagues (2005) evaluated the effects of four strategy instruction npproaches (self-monitoring, self-monitoring plus reinforcement, self-reinforcement, and self-management) on academic behavior for students identified with ADHD. Self-monitoring strategies included both self-monitoring of attention and performance. Self-monitoring plus reinforcement includes adding reinforcement from an external agent for change in target behavior (Reid et aI., 2005). Self-reinforcement is the same approach as self-monitoring plus reinforcement but students are taught to choose their own motivation. And self-management strategies, as defined by Reid, are the same as self-monitoring except student self-evaluation of accuracy is compared to teacher evaluations. Sixteen studies were included in the analysis, half of which described self-monitoring plus reinforcement approaches, and four studies were conducted in general education classrooms. The target behaviors across studies included on-task behavior, inappropriate behavior, and academic accuracy and 34 productivity. Effect sizes were large across studies, especially for self-monitoring sample with a range of 0.59 to 2.96. Self-monitoring strategies yielded the largest effect sizes for reducing inappropriate behavior and increasing academic accuracy and productivity, while self-management produced the largest effect size for on-task behavior. No generalization or maintenance data were reported in this synthesis. Mooney and colleagues (2005) published a similar synthesis of self-management interventions for students identified with emotional behavioral disorders. They sampled studies examining the effects of self-management instruction (self-monitoring, self- evaluation, self-instruction, goal setting, and strategy instruction) on academic task completion and accuracy. A total of 22 articles were included, published between 1970 and 2002, and the student age range was 5-21 years. Self-management strategies were applied in math, reading, writing, social studies, and science. The majority of target behaviors were related to academic production (e.g., duration of engagement) and accuracy (e.g., % correct). Again, the effects sizes were very large ranging from -0.46 to 3.00. The mean combined effect size was 1.80. It should be noted that all the studies utilized small sample sizes (n =7 was the largest single sample), and the effect of small samples on effect size interpretation has been documented (Parker & Hagan-Burke, 2007). Generalization (two studies) and maintenance (13 studies) data were reported for 15 of the 22 studies, all reporting positive findings (Mooney et aI., 2005). Self-Regulated Strategy Development Self-Regulated Strategy Development (SRSD) is an intervention package designed to teach metacognitive strategies explicitly (Graham & Harris, 1993). The 35 rationale for SRSD is adopted from social cognitive theory which supports the notion that self-regulation skills are developmental and responsive to direct instruction (Zito et aI., 2007). While the instructional sequence may be modified to teach strategies for any academic task, the most common application of SRSD is for teaching writing strategies (Graham &, Harris, 1993). SRSD instruction follows a systematic approach that includes six general steps. The following is a summary from Santangelo, Harris, and Graham (2008). Step one involves instruction to develop student background knowledge of the essential skills required for the task (e.g., components of narrative writing)~ Step two ("Discuss it") focuses on building student motivation to learn a new strategy by.self-evaluation of their own skills. Step three ("Model it") utilizes modeling as an instructional paradigm as the instructor "thinks aloud" to demonstrate the strategy process (e.g., mnemonic to self- evaluate complete story grammar). Step four involves helping students memorize the strategy. Step five ("Support it") focuses on fading instructor supports, and step six ("Independent performance") emphasizes measuring strategy use, generalization, and maintenance (Santangelo, Harris, & Graham, 2007). The SRSD paradigm has been evaluated in a number of writing experiments (see review in Zito et aI., 2007), and experimental results indicate positive effects for both writing knowledge and improved in-class process management (e.g., Graham et aI., 2005). Out-o.fClass Assignment Management Strategies A number of reports have been published examining the effect of strategy instruction on out-of-class assignment management. One example of an out-of-class 36 assignment strategy was published by Olympia and colleagues (1994). The researchers trained the classroom teacher to implement a homework completion strategy for 16 students who struggled with math. Following two training sessions, teachers taught students to graph the number of assignments they completed and their percentage accuracy. The results indicated increased pre-post math achievement scores and increased curriculum-based assessment scores for the group. Homework completion and accuracy results were mixed, as measured by visual analysis of single subject research design data, due to high variability of target behaviors (Olympia, Jenson, Sheridan, & Andrews, 1994). A widely cited example on a homework completion strategy is provided by Hughes and colleagues (2002). The researchers taught students with learning disabilities an assignment management strategy that incorporated several metacognitive and memory components. Students were taught to record new assignments in weekly and monthly schedules. They also taught students a self-regulation strategy to actively manage the assignment completion process. The acronym PROJECT helped students remember the process. Briefly, PROJECT stands for Prepare, Record, Organize, Jump to it, Engage, Check, Turn in. The Organize step was further broken down into four steps, and the mnemonic BEST help students self-monitor this phase: Break into parts, Estimate time, Schedule, Take materials home. The strategy was modeled and reinforced by the researchers during 30 minute instructional sessions held four days a week for four weeks. Of the nine s~udents in the study, eight demonstrated mastery of the strategy for both simulated assignments and work assigned in general education classrooms. As a group, 37 the students completed more assignments, increased GPA over the course of the year, and were rated higher by teachers on measures of assignment management and completion (Hughes, Ruhl, Schumaker, & Deshler, 2002). Gaps in Intervention Literature W.hile the number of research studies describing effective self-regulated learning strategies for students who struggle academically is increasing, there are several gaps in the literature base. One component of self-regulated learning intervention is implementation. In a study attempting to evaluate the PROJECT homework intervention (Hughes et aI., 2002), the current researcher encountered significant and unexpected implementation barriers in when working in a typical special education context. Barriers included student resistance due to feelings of alienation when they were pulled out of class, erratic student attendance, and inconsistent strategy reinforcement across teachers (J~ess & Sohlberg, 2007). The results of these barriers were poor treatment delivery and adherence. A follow-up pilot study evaluated a modified assignment completion process strategy designed to mitigate the previously identified implementation barriers. Modifications included simplifying the materials, training the classroom teacher as the primary strategy mediator, and incorporating the strategy within the classroom context. These modifications proved successful, and the strategy resulted in improved in-class assignment management skills for students in a resource room context ~ess, Sohlberg, & Albin, in preparation). ---------------_.- 38 In addition to implementation barriers, another gap in strategy instruction literature is examining the broader impact of improved academic task management. The majority of self-management strategy research focuses on academic productivity and accuracy of specific tasks (e.g., Reid, 1996). Evaluation of other possible outcomes of self-management strategy use such as improved independence in assignment completion or changes in self-efficacy have been neglected. For instance, little is known of the effects of improved assignment management on motivational vari'ables such as academic self-efficacy for students who qualify for special education. Further, it has not been established whether students who learn a self-regulation strategy for one task will generalize strategy use for other academic tasks. The current study was designed to further the cognitive self-regulation strategy research and address some of the current research gaps. Research Questions The research questions guiding this investigation were: Student Implementation 1. Can students with diverse academic needs employ a self-regulated assignment attack strategy (SAAS) when taught as a whole group in a resource setting as measured by their ability to verbalize and demonstrate key elements of the strategy? Teacher Implementation 2. Can teachers implement and maintain the critical elements of the SAAS with high fidelity as measured by direct observation and teacher report following two thirty-minute training sessions? 39 Social Validity 3. Is SAAS perceived as a practical and effective strategy as measured by a modified version of the Treatment Evaluation Inventory-Short Form (teach~r) and modified Children's Intervention Rating Profile (student)? Assignment Attack 4. What are the effects on assignment attack for students who learn and apply SAAS during study supports class? Generalized Impact 5. What are the general academic effects for students who apply SAAS as measured by quarterly gra.de reports and classroom grades and assignment completion rates in math and social studies? 6. Do the students who implement SAAS report higher general academic strategy use, motivation, and academic self-efficacy as measured by self-report? 7. If students improve independent assignment management, do the benefits generalize to activities requiring self-regulation in non-academic settings as measured by the Behavior Rating Inventory of Executive Functions (BRIEF: parent version)? 8. What are the behavioral effects for students who learn and apply SAAS as measured by the Home'work Behavior Checklist (HBC) and on-task behavior in class? 40 CHAPTER III METHODOLOGY This section describes the research activities completed during this study evaluating the effects of a self-regulated assignment management intervention package on in-class assignment attack behavior using a single subject multiple-baseline design. The target population for this study was students who qualify for special education and struggle to manage the assignment productivity requirements inherit in secondary education settings. In addition to in-class assignment management behavior, this study examined the generalized impact of self-management strategy instruction on different domains of self-regulated learning including improvements in academic strategy use and cognitive-be~aviorself-regulation processes. This chapter describes the methodology applied for the study. First, a summary of the pilot study preceding this project is provided, followed by a description of how the current work was designed to extend the pilot findings. Next, a description of participant characteristics and recruitment is provided. The research design elements and data analysis procedures are subsequently detailed. Pilot Study The pilot study (Ness, Sohlberg & Albin, in preparation) evaluated an intervention designed to improve student management of assignments. Pre-pilot research, 41 based on the Hughes and colleagues' (2002) PROJECT model, had revealed implementation barriers for strategy instruction inherent in "real-world" special education settings (Ness & Sohlberg, 2007). The pilot study attempted to address these implementation barriers by using whole-class instruction and targeting teacher implementation of strategy instruction. The pilot study evaluated the effectiveness of a classroom-based external assignment tracking system on individual student assignment attack behavior. Pilot Methods Six students in a resource context, all of whom qualified for resource support based on academic difficulty and failure to manage required assignments, participated in the intervention. All six students in the classroom participated in this study because the intervention was designed to be part of the daily instructional routine. Outcome data were collected on the three students with the lowest assignment attack scores. An interview with the special education teacher was conducted to identify a strategy that would work with his classroom routines and the school scheduling. An external assignment management tool was developed consisting of four components: (l) a bulletin board, (2) assignment recording tickets, (3) student access to bulletin board to remember assignments, and (4) reinforcement for completing assignments. A bulletin board was installed in the back of the classroom and was divided ~nto seven columns (one for new assignments, six for students) with heavy black lines. After a demonstration by the researcher, the teacher taught students to record target class, assignment detail, and due date on the "assignment tickets". He selected student volunteers to complete tickets 42 for new assignments daily and post them on the bulletin board. The teacher was instructed to refer students to the board to remind them what outstanding assignments they needed to complete. The teacher instituted a reinforcement consisting of a weekly raffle using completed assignment tickets where students could win rewards (e.g., "Front of the Line passes"). The dependent variable consisted of an assignment attack protocol developed and validated for. this study. Assignment attack comprised four student behaviors requisite for independent assignment management: (1) recalling assignment details, (2) gathering necessary materials, (3) initiating seatwork, and (4) ongoing task engagement. The assignment attack behavior was operationalized and high inter-rater reliability was established following multiple classroom observations and teacher interviews prior to the initiation of the pilot study. Assignment attack data were collected through classroom observation using the measure depicted in Figure 1. The pilot study featured an ABAB withdrawal design (Kennedy, 2005) to evaluate whether assignment attack behavior changed as a function of having the classroom assignment management strategy with the bulletin board and teacher supported process as detailed. Pilot Results and Conclusions Results indicated the assignment management strategy and assignment attack behavior were functionally related for two of the three students. Large and immediate changes in assignment attack level were demonstrated. The third student's assignment attack behavior improved, but high variability limited the interpretation of the findings. A Figure 1 Assignment Attack Rating Scale. Student behavior Score (1-5) 1. Student recalls homework assignment during homework support period. Recall requires knowing academic subject and assignment details. I = Teacher must tell student assignment or show student pink sheet/folder. Student has no recollection of assignment and acts like they I have no work. 2 = Teacher tells student assignment or shows student pink sheet/folder. Student acts like they know what to do but doesn't know subject 2 and details. 3 = The student knows about assignment if asked or checks pink sheet/planner with teacher once, must get repeated teacher clarification 3 regarding the details of the assignment (i.e., teacher pointing to pink sheet/folder or verbal instructions on multiple occasions). 4 = The student knows both the subject (e.g. science) and the details of the assignment and demonstrates this by gathering materials, but may 4 ask one clarifying question of teacher. Teacher does not need to review pink sheet or folder with student. 5 = Student requires no support for knowing subject and assignment details. 5 2. Student gathers appropriate material for assignment. Gathering materials requires looking for and retrieving book/worksheet, pencil/pen, and related materials. I = Teacher gives materials without independent student attempt to open bag, get notebook, etc. I 2 = Teacher tells student materials and helps student retrieve them. 2 3 = Teacher tells student what materials and student retrieves them. Usually prompts to get material trom locker. 3 4 = Teacher must clarify materials need after student retrieves some materials. 4 5 = Student independently looks for and retrieves materials or independently asking to leave and retrieve materials from another location. 5 3. Student initiates assignment completion. Initiating assignment refers to making a first attempt at working. I = Teacher must visit student at desk more than once to tell student to begin working on new assignment. I 2 = Teacher either visits student once or has to provide multiple verbal prompts before student starts on new assignment. 2 3 = Teacher provides only one verbal command to start working. 3 4 = Teacher only makes indirect comment like "it's time to work on your assignments" to prompt student to begin work. 4 5 = Student begins work with no individually directed teacher prompts. 5 .,J:::. w Figure 1 (Cant.) 4. Whole-class task persistence. Engagement means actively working on assignment towards completion. I = Student looks around class or turns head from work once every minute or two, and/or wanders from desk more than once. Repeated I teacher prompts to keep working. 2 = Student looks around class/turns head every five minutes or so. May wanders from desk once. Repeated teacher prompts to keep 2 working. 3 = Student looks around class and turns head once every five minutes or so. Doesn't wander from desk. Teacher provides one or two 3 prompts to remain on task. 4 = Student may look up and look around once every five minutes. 4 5 = Student remains on task for the entire period, requires no teacher prompts to remain engaged. 5 On-task Behavior: 1 minute 2 minutes 3 minutes 4 minutes 5 minutes 6 minutes 7 minutes 8 minutes 9 minutes 10 minutes ~ ~ 45 maintenance check conducted one month after study completion revealed that the teacher had continued use of the strategy and students' assignment attack behavior maintained previous high levels. The contribution of this study was threefold. First, the results supported findings in the self-regulation literature that strategy instruction improves academic performance for struggling students. Second, the successful implementation suggested that teaching strategies at the level of the classroom and utilizing the classroom teacher as primary implementer is effective in a special education setting. Third, this study provided a validated measure of in-class assignment management behavior. Dissertation The current study extended the previous work by incorporating the metacognitive or self-regulation aspect of strategy training that appears to be key in long-term maintenance and generalization. The pilot study relied on external supports that were not faded. Improvements in student behavior were the result of more organized, purposeful behavior (i.e., facilitated executive functioning), but the metacognitive capacity for self- regulation was not addressed. Students were not taught to construct an internal schema for self-regulating their assignment attack. The current study addressed this limitation by scaffolding a metacognitive approach to assignment attack using external supports, and systematically fading the supports to facilitate self-regulation. Participants· The participants (N = 3) for this study were selected based on the following inclusion criteria: (1) currently enrolled in sixth grade; (2) qualify for special education 46 services; (3) enrolled in general education classes, but spend a portion of their day in a resource room for help managing out-of-class assignments; (4) low academic achievement as measured by classroom grades; and (5) dependent upon teacher for organizing materials and initiating their work in the resource context as reported by teacher and confirmed by researcher observation using the assignment attack protocol. Recruitment and consent. University of Oregon IRB approval was ongoing from the pilot study and the Bethel School district agreed to facilitate the project. A modification to the original university IRB application was approved allowing alterations to intervention and research design as well as collecting additional outcome data for this study. Through word of mouth from the teacher participating in the pilot study, a sixth grade special education middle school teacher at a different school expressed interest in the study and provided written consent to apply the strategy during her homework support class and to complete measures. All students in the class were invited to participate in the study. The teacher sent home recruitment and consent letters to parents/caregivers. The intervention was applied as a part of the classroom routine so all students learned the strategy. However, data were collected only on students whose parents returned signed consent. Three students were selected for data analysis based on teacher nomination and low baseline assignment attack behavior. 47 Setting In the host middle school, there were three different resource support classes for sixth graders. The special education teacher taught the second period class, and she supervised educational assistants teach two separate homework support classes during third period. The participating teacher had six years of teaching experience, all in special education and in the same middle school. Over the duration of this study, there was one educational assistant for one third period class and three different assistants for the other. Each study support class was 45 minutes in duration and was designed to provide struggling students support for managing assignments and academic responsibilities in an inclusive setting. There were approximately twelve students in the second period group (including Adam), and there were six students and three students in each third period group (including Brett and Christina respectively). One participating student was nominated and selected from each group. The demographic information about these students is presented in Table 1. Adam Adam was a twelve-year-old boy who qualified for special education based on identifications of specific learning disability and speech-language impairment. Adam's struggle with assignment attack included rarely bringing materials to class and difficulty remembering outstanding a~signments, according to the special education teacher. His success during assignment completion time seemed largely dependent on internal factors (i.e., mood when class started) independent of work expectations. His academic supports and accommodations included a behavioral support plan, reduction of large homework 48 assignments, preferential seating, extended time for reading and writing assignments, allowing healthy snacks during class, morning check-in, teacher check for comprehension, and increasing active participation. Table 1 Demographic Data Student Adam Age Gender Ethnicity 12 M lW/C Free/Reduced Lunch Yes IDEA Identification Full Scale IQ Brett Christina 12 12 M F W/C W/C No No SLD 86 (WISC-IV) 108 (WAS!) I White/Caucasian 2 Specific Learning Disability 3 • Speech or Language Impairment 4 Emotional Disturbance 5 Reynolds Intellectual Assessment Scale Brett Brett was a twelve-year-old boy who qualified for special education based on an identification of emotional-behavior disturbance. He was selected to participate in this study due to struggles initiating assignment attack, most often the result of refusing to start non-preferred assignments. According to the special education teacher, Brett was able to stay on-task once he started, but even with preferred assignments, teacher support was necessary to assemble materials. Brett's other academic supports and 49 accommodations included providing immediate teacher feedback, repeating directions, preferential seating, re-teaching expected behaviors, and teaching self-monitoring. The primary manifestation of his disability was defiance toward adult instructions or anger following requests to change activities. Christina Christina was a 12-year-old girl who received study supports due to an identification of specific learning disability. Her learning disability was secondary to a diagnosis of phenylketonuria (PKU), which is an inherited metabolic disorder associated with impaired conversion of the amino acid phenylalanine. Excessive phenylalanine can result in damage to the central nervous system and subsequent developmental disabilities (National Institute of Child Health and Human Development, 2009). Christina's assignment attack difficulties were related to poor initiation and sustained attention as well as problems remembering assignment details according to the special education teacher. Other academic supports and accommodations included teacher check for comprehension, breaking down length assignments/projects, reducing workload, providing oral directions, sitting in close proximity to the teacher, teacher support with homework plamler, and extended time for assignments. Research Design The approach implemented to address the primary research questions was a multiple baseline across participant design. This design was selected to evaluate the functional relation between the Self-regulated Assignment Attack Strategy (SAAS) described in the next section and changes in student behavior. Unlike traditional, 50 between-subject group designs involving random selection, single subject methodology facilitates the evaluation of functional changes in individuals over time. Single subject methodology is particularly useful in intervention studies because each participant serves as his/her own control. This reduces threats to internal validity, such as maturation, common in group designs (Kennedy, 2005). It is particularly useful when the population is heterogeneous and the research questions seek to find clinical or practical significance of changes (Homer et aI., 2005). In contrast to statistical analyses in group designs, experimental control is evaluated.using visual analysis of raw data in single subject designs (Franklin, Gorman, Beasley, & Allison, 1997). Changes in behavior between baseline and intervention phases (i.e., change in level, trend, and/or variability) indicate a functional relation between the intervention and target behavior (Homer et aI., 2005; Kennedy, 2005). Three clear instances of behavior change at three different points of time across participants in multiple baseline designs has been recommended as the guideline for demonstrating experimental control (Homer et aI., 2005; Parker & Hagan-Burke, 2007) In this study, the baseline condition represented normal classroom conditions. The intervention condition represented implementation of SAAS. To facilitate self-regulation, the external prompts were faded in the intervention phase. This design permitted evaluation of whether SAAS and assignment attack were functionally related for students in the resource room. Also, this design enabled evaluation of whether the effects were externally mediated or if students learned to self-regulate assignment attack. 51 Because other constructs under investigation, such as self-efficacy and volitional academic .strategy use, were not expected to change on a daily basis, these domains were evaluated using pre- and post-intervention measurement. Intervention Package: Self-Regulated Assignment Attack Strategy (SAAS) The independent variable consisted of an assignment completion strategy package, the Self-regulated Assignment Attack Strategy (SAAS), containing a teacher- supported checklist with key assignment management behaviors that the students could follow in a classroom setting. Teacher support was faded over time with instruction to have students internalize the process by "saying the steps in their heads". All instruction was scripted promoting consistent implementation across groups (see Appendices 2 and 3). There were three critical components of SAAS. First, a step-by-step student checklist was presented, scaffolding self-regulation behaviors for students, as shown to be important in the self-management literature (e.g., Reid et aI., 2005). Another important facet of the package was systematic instruction and teacher demo~stration.The scripts and processes were fashioned after the SRSD model (Graham & Harris, 1993). Finally, the fading of external support to internalized self-regulation is necessary for long-term maintenance as discussed in the self-regulation literature (e.g., Boekaerts & Corno, 2005; Zimmerman, 1998) Checklist: The Four P's. The external support for self-regulating assignment attack was a checklist given to students during the resource period (see Figure 2). The four steps were: (1) Planner/Pink 52 Figure 2 The Four P 's Checklist. The Four P's How Did I Do? IMon8wed~ - - Proceed Planner Check I D CD ~ick an Assignment I D IT ~repare Materials I D IT I D IT Key: ~ = Did not have to be told - = Had to be told 53 sheet check, (2) Pick an assignment, (3) Prepare materials, and (4) Proceed. External supports ~re effective for modeling self-regulated learning processes, such as self- monitoring (e.g., Harris et ai., 2005). We demonstrated the effectiveness of using external aids to support assignment attack behavior for students in a resource setting in the pilot study. Systematic instruction. The Self-Regulated Strategy Development (SRSD; Harris & Graham, 1993) instructional model has been replicated in numerous studies teaching students who qualify for special education to self-regulate academic tasks (e.g., Graham et aI., 2005; Zito et aI.; 2007). The SRSD model includes six steps the teacher performs sequentially. The six steps are illustrated in Figure 3. The initial instructional sequence for introducing SAAS was mapped onto the SRSD Phases 1-3. The instructional sequence for fading the external supports mapped onto SRSD Phases 4-6. The instruction was scripted to promote consistent explanation across groups and maximize effective implementation. Three instructional goals were explicitly addressed in the script. First, the teacher was prompted to teach the strategy using direct instruction principles known to facilitate cognitive strategy acquisition for students who struggle academically (Kameenui & Simmons, 1990). Specifically, the teacher was prompted to explain the four steps of the checklist, model the procedure, and assess students' understanding through questions. Second, student motivation to apply the strategy in class was targeted in the script. The teacher was prompted to lead students in a discussion 54 about their experiences completing assignments and their need to manage this academic skill more independently in the future. Establishing personal relevance is a powerful predictor in student acceptance and use of self-regulation strategies in classrooms (Schunk, 1998). Third, the teacher was prompted to model self-regulation processes using Figure 3 Self-Regulated Strategy Development Stages*. Stage 1: Develop Background Knowledge Help students develop understanding of essential skills and knowledge of academic content, strategy, or task. Stage 2: Discuss It Teacher and students discuss current academic performance and need for effective strategies to support learning. The teacher introduces the proposed strategy and explains the purpose ~nd benefits. Stage 3: Model It The teacher models how to use the strategy using examples and "talk-aloud" procedures, explaining each phase of the strategy as it is completed. Stage 4: Memorize It This stage is not necessary for all students, but many students who qualify for special education benefit from memorization practices. To help students memorize elements of learning strategies, mnemonics can be applied to critical features. Stage 5: Support It Teacher and students use the strategy collaboratively during academic tasks. Stage 6: Independent Performance During the final stage, students use strategies and self-regulation processes independently. Students are encouraged to say steps or procedures covertly "in their heads" to fade the use of external supports. *Adapted from Graham and colleagues, 1998. 55 self-appraisal statements during strategy demonstration. The goal.of SAAS was to increase students' evaluation of their cognitive processes during assignment completion, and teacher modeling of self-appraisal statements is an effective method to convey self- regulation of learning (Paris & Winograd, 2001). Faded external supports. The components of SAAS were intended to facilitate internalization of self- regulatory processes. First, SAAS uses a mnemonic to help students remember key behaviors. Mnemonics are powerful memory strategies that work by producing mental representations of information that facilitate ease of recall. A classic example of a mnemonic in the form of a poem is, "When two vowels go walking, the first one does the talking." SAAS uses a first letter mnemonic strategy to help students remember key assignment management behaviors. All four behaviors begin with the letter "p", so the self-regulated assignment management process was referred to as the "4 P's". Second, the "4 P's checklist" was systematically modified (see Figure 4) and faded using the method of vanishing cues (Baddeley, 2004) until students were instructed to say the four steps "in your head" without the support of a checklist. Fading external supports using self-instruction, or "self-talk", is an effective approach for helping struggling students develop self-regulated learning practices (e.g., Johnson et aI., 1997). Current homework strategy. The intervention package was designed to be flexible so that it could work in variety of classroom/school contexts with differing teacher practices. The key components including the checklist of self-regulation behaviors, teacher modeling and 56 scripts, and fading of external prompts could be adapted to a variety of school and classroom routines. In this study, SAAS was superimposed on the existing homework management system described below. The students in the target sixth grade resource room used a daily planner intended to help them remember assignments from general education classrooms. They were expected to use the school's daily planner to remember their assignments. The planner consisted of daily sheets with pre-printed rows delineating the seven periods of the school day. The students were instructed to write new homework on the planner the moment assignments were assigned. The students were then prompted by the teacher to check their planner during resource period and work on outstanding assignments. Verbal report from the special education teacher indicated these students had difficulty entering their assignn1ents consistently and accurately necessitating the creation of a second system -- a weekly log of unfinished assignments. The student carried this "pink sheet" to each of their teachers once a week and the teachers wrote unfinished assignments on the sheet. The special education teacher then prompted the students to take the sheet home and have their parents/caregiver sign it. The pink sheets were used in the resource room to help students prioritize assignments. Discussions with the special education teacher indicated that she wanted to maintain use ofthe school planner and "pink sheet" but desired to help students improve their indepe~dent assignment completion skills as the current system was not meeting this goal. Figure 4 Modified Checklist. p - p - p - p - 57 The Four P's How Did I Do? IMonEJwed~ I D []] Key:. .y = Did not have to be told - = Had to be told 58 Dependent Variables The research questions guiding this study sought to answer questions about the potential impact of an academic strategy instruction package on both academic outcomes and broader cognitive-behavioral domains. Therefore, the dependent variables for the study were organized into those that measured academic outcome and those that measured self-regulated learning variables. Academic outcome. Five measures were administered to evaluate the impact of SAAS on academic outcome: (1) assignment attack, (2) academic achievement, (3) assignment completion behavior, (4) assignment completion rate, and (5) proportion of time spent on-task. Assignment attack was measured using the scale illustrated in Figure 1. The measure sampled four domains of independent assignment management: (1) recalling assignments,. (2) gathering materials, (3) initiation, and (4) whole-class engagement. The face and content validity of the scale were confirmed by implementation in a special education resource context (Ness, Sohlberg, & Albin, in preparation). The assignment attack sco.re is a composite offour domain scores (recall, materials, initiation, and engagement); each domain was measured on a 1-5 scale depending on the level of teacher prompting necessary for each area. The minimum score was four and the maximum score was twenty. Daily assignment attack data comprised the dependent variable in the pilot study. Reliability checks were conducted during the pilot study and researcher scores were highly correlated (r = .90) indicating this measure yielded consistent scores across raters. 59 Academic achievement was measured using changes in social studies and math grades over the course of the term. The grades were calculated on a percentage basis (100% maximum). The special education teacher endorsed these two classes for outcome measurement since students were assigned homework frequently and the data were accessible using the existing progress monitoring program at the school. Assignment completion behavior was measured using teacher and parent reported perceptions of student behavior during academic tasks. The measure was the Homework Problem Checklist (HPC, Anesko, Schoiock, Ramirez, & Levine, 1987). The HPC, illustrated in Figure 5, has been validated to measure out-of-class assignment completion behaviors by students who qualify for special education (e.g., Epstein et aI., 1993). The questionnaire consists of20 items, and uses a 0-3 Likert-type rating scale. While the scale has not been normed, previous studies have demonstrated the internal consistency (Cronbach alpha = .91) and discriminative validity of the HPC (Anesko et aI., 1987). Assignment completion rate was measured using the school progress report system. The host middle school produced regular progress reports for students. Included on these reports was a list of completed and unfinished assignments. Assignment completion rate was calculated by dividing the number of completed assignments by the number of total assignments in a one-month period. A one-month)nterval was selected because this was a reasonable amount of time to gather a representative sample of assignment completion according to the special education teacher. On-task behavior during assignment completion time was defined as the student actively engaging in work or seeking help after initiating academic tasks and the absence 60 of off-task behavior (Amato-Zech, Hoff, & Doepke, 2006). Off-task behavior was operationalized according to the framework described in Amato-Zech and colleagues (2006): off-task motor, off-task verbal, or off-task passive. Off-task motor included any Figure 5 Homework Problem Checklist. For each statement, rate the frequency of each problem's occun'ence in the past two weeks using the 1011O\ving scale: never (0); at times (1 ); often (2), and very often (3). "'--~--"-""_'_-"------~-~---~----~..--,----------, Fails to bring home assignment and necessary materials I (t~~!2Q9-~~_~Q~~~}1~t~!_~c.1. +.U_~ l_._.__.._~_ Doeso't know c~ac.~.what hCf~~e.wQrk.has been ass,i~cd':'- t-'0_1_2_3 -1 Denies 'having homework assignment 0 1 2 3 Refuses to do homework asshmment. °1 2 3 ~~Vhin~rcornPtl!!!!~abouth211leworL~ 0 1 2 3 Must be reminded to sit down and start homework. 0 1 2 3 Procrastinates, puts offdoing homework. 0 J 2 3 Doesn't do homework satisfactory unless someone is in the 01 2 3 mom. I D?c~'n~t ~oh()mework satisfactorily unless someone docs it i0 ] 2 3 wIth hmliher. ~~--------1 Daydreams or plays. with object\) during homework session. °1 2 3 Easily distracted by noises or activities ofothers. 0 1 2 3 o 1 2 3 o 1 2 3 Deliberately fails to bring assignment back to class, !S~lP.Q.f-l~~Q25!r_~~~~~I!J2!2_~},J~,~r~~.~~J2~21}"~~t,,~2!!!~~9!~· t o,,,,,,,,1__",w,2,,,,, ".3"',, -"'~"-''''-''-''''''-'1 Produces messv or sloppy homework. 0 1 2 3 Hurries through homework and makes ~3!!~Te§s !nist~k~;.~~,--·~"i'O'·""'1"·-'-2·'~-3,,~~,,_w'-~'--l Shows dissatisfaction with work, even when he/she docs a 0 1 2 :3 good job. '--~---"-'---------------+----~-----l forgets to bring assignment back to class. 61 motoric movement not associated with the task including randomly flipping pages, out of seat, etc. Off-task verbal included any audible verbalization not associated with the task or permitted during the task such as talking to peers, humming, calling out answers. Off- task passive included passive disengagement for a period of at least three consecutive seconds. Cognitive-behavioral variables. To address the questions of the potential impact of SAAS on broader cognitive- behavioral factors, this study included non-assignment completion related variables. Changes in executive functions, academic self-efficacy, motivation, academic strategy use, and actiJ!g-out behavior were measured as potential indicators of domain-general cognitive-behavioral changes. Generalized changes in self-regulated cognitive processes were assessed using the Behavior Rating Inventory of Executive Functions: Parent Form (BRIEF, Gioia, Isquith, Guy, & Kenworthy, 2000b). The BRIEF is an 86 item questionnaire designed to capture behavioral correlates of impaired executive functions in everyday situations (e.g., getting ready to go to school). The measure yields three standardized scores: the Behavior Regulation Index, the Metacognitive Index, and the General Executive Composite. The Behavior Regulation Index includes items that sample executive domains including inhibition, set shifting, and emotional control; the Metacognitive Index consists of items measuring initiation, working memory, planning/organizing, organization of materials, and self-IlJonitoring. The General Executive Composite is a sum of the two indices. The index and general raw scores are standardized and reported as T scores with percentile 62 ranking, and higher scores represent greater perceived dysexecutive symptoms (Gioia et aI., 2000b). The original normative sample for the BRIEF did not include students who qualified for special education, but a subsequent study demonstrated the validity of the scale for evaluating executive function deficits for students identified with ADHD, autism, traumatic brain injury, and learning disability (Gioia, Isquith, Kenworthy, & Barton, 2002). During initial evaluation of the scale, the test-retest reliability coefficient of the BRIEF-parent form was .81 for a duration of two weeks, internal consistency was .80-.98, and inter-rater reliability between teachers and parents was .32 (Gioia, Isquith, Guy, & Kenworthy, 2000a). The low inter-rater reliability was considered a function of evaluating students in different contexts where presumably the behavioral impacts of executive function impairment were context dependent. There were no inter-rater comparisons of BRIEF scores in this study since only the parent form was administered. To examine the potential impact of SAAS on broad constructs of self-regulated learning, a scale was created sampling three student characteristics: academic self- efficacy, motivation, and academic strategy use (Figure 6). The 13 items on the scale were selected from existing measures, but the content was significantly altered to match the purpose of this study. The original items were selected from the Academic Volitional Strategy Inventory (McCann & Turner, 2004), the Self Efficacy of Learning Form: Abridged (Zimmerman & Kitsantas, 2007), and the Motivation section of the Motivated Strategies for Learning Questionnaire (Pintrich et aI., 1991). The question wording was modified for developmental appropriateness and relevance to middle school academic 63 Figure 6 Student Se(fregulation Probe. Please circle the number that best fits you. Always Sometimes Never 1 2 3 4 5 1. I promise myself something like watching a movie if! study. (8) 1 2 3 4 5 2. I tell myself, "you can do this!" when if! get stuck on 1 2 3 4 5 assignments. (8E) 3. On difficult assignments, I remember how I did a similar work in 1 2 3 4 5 the past. (8) 4. I prefer difficult assignments because I like work that challenges 1 2 3 4 5 me. (M) 5. When I don't feel like working, I remember how important it is 1 2 3 4 5 to graduate. (8E) 6. I ask for help when I get stuck on assignments. (8) 1 2 3 4 5 7. I really like to get good grades on assignments. (M) 1 2 3 4 5 8. When I try hard, I learn from all of my assignments. (M) 1 2 3 4 5 9. I remind myself to stay focused if! get distracted. (8) 1 2 3 4 5 10. I like to find new ways to make assignments interesting. (8E) 1 2 3 4 5 11. I am good at getting caught up on assignments. (8E) 1 2 3 4 5 64 Figure 6 (cont.) 12. I feel I let myself down if I don't keep up with school work. (M) 13. If I try, I can get the best grade in class. (M) SE = Academic self-efficacy. S = Strategy use. M = Motivation. 1 1 2 2 3 3 4 4 5 5 contexts. This was an informal measure since it had not been validated or normed for this population. It did, however, have high face validity and theoretical grounding being a composite of questions from existing measures. Treatment fidelity. Treatment fidelity refers to the accuracy and consistency with which independent variables are delivered (Gresham, MacMillan, Beebe-Frankenberger, & Bocian, 2000). Documenting treatment fidelity is essential for demonstrating a strong relationship between the treatment and the outcome variable, as well as supporting the external validity of research findings (Gresham et aI., 2000). Because this intervention was conducted in a school setting and not a laboratory context, it was important to measure the implementation. Adequate treatment fidelity is critical to the validity of inferences that can be drawn about the impact of the treatment (Zvoch, Leternou, Parker, 2007). Following Gersten and colleagues' recommendations (2005), both fidelity of intervention delivery ("treatment delivery") and fidelity of student implementation ("treatment adherence") were measured (See Figure 7). Treatment delivery and adherence fidelity data were measured once a week during each phase. A total percentage 65 Figure 7 Treatment Fidelity Worksheet. Treatment Delivery Intervention Phase 1. The teacher makes a clear declaration that assignment completion time begins. 2. Teacher hands out 4 P's checklist. 3. Teacher asks students to fill out "How did I do?" boxes. 4. Teacher reads at least one self-reflection statement during self-evaluation. 5. Teacher collects students' checklist after each class. First two days only: 1. Teacher explains need and benefit of using strategy. 2. Teacher models use. 3. Teacher asks students reflection questions. Treatment Delivery Fading Initial sequence (two weeks) 1. The teacher makes a clear declaration that assignment completron time begins. 2. Teacher hands out the 4 P's checklist. 3. Teacher asks students to fill out "How did I do?" boxes. 4. Teachers initial each students' checklist after each class. +/- Fade procedure 1. The teacher makes a clear declaration that assignment completion time begins. § 2. The teacher hands out shortened checklist (first two weeks only) 3. The teacher asks students to say the 4 P's "in your head". Treatment Adherence: Student use of SAAS 1. Students recite the four P's when prompted (esp. first week Phase B). 2. Students look at the template at least once while working on assignments. 3. Students record checks or minuses on the checklist. 4. Students report reciting 4 P's "in your head". (first week Phase B') 66 was calculated for both treatment delivery and treatment adheren~e. The outcome variable for both was total percentage (Gresham, 1997). Social validity. Social validity has been defined as "the estimation of the importance, effectiveness, appropriateness, and/or satisfaction various people experience in relation to a particular intervention" (Kennedy, 2005, p. 219). Social validity is an important construct since research consumers are less likely to implement interventions they perceive as less acceptable (Foster & Mash, 1999). Perception of SAAS as a socially valid intervention was measured to examine relationships between fidelity, student outcomes, and replication likelihood. T~e most common measure of social validity is participant questionnaire (Kennedy, 2005). For this study, teacher and student perception of treatment acceptability was measured using questionnaires. The teacher questionnaire contained four items based on questions from the Treatment Evaluation Inventory-Short Form (Kelly et aI., 1989). The teacher rated each item using a five level scale. Also, a fifth item was included; an open ended question asking for teacher input for making the intervention more acceptable. Student perception of intervention acceptability was measured using a modified version of the Children Intervention Rating Profile (CIRP; Turco & Elliot, 1986). See Figures 8 and 9 for illustrations of these measures. 67 Figure 8 Social Validity Measure for the Teacher. Place an 'X' in the corresponding box below: Strongly Disagree Neutral Agree Strongly Disagree Agree I think the students liked the assignment completion strategy (i.e., the 4 P's and self-evaluation). I believe, the assignment strategy helped the students work independently. I believe the assignment completion strategy is likely to result in permanent improvement. Overall, I have a positive impression 9f the assignment completion strategy. Anything you would like to see done differently? Figure 9 Social Validity Measure for Students. 68 Circle one number for each statement. Please Read I Agree I Do Not Agree The assignment worksheet helped 1 2 3 4 5 6 me work on my homework. I did fine on assignments without 1 2 3 4 5 6 the worksheet. There are better ways to help me do my homework than the 1 2 3 4 5 6 assignment worksheet. I like the assignment worksheet. 1 2 3 4 5 6 I think the assignment worksheet 1 2 3 4 5 6 will help me do better in school. 69 Procedures The following section outlines the procedures that were used to investigate the research questions motivating this study. Described below are descriptions of activities for each phase of the multiple baseline design, data collection procedures, and related research design procedures. Baseline phase, The purpose of the baseline phase was to measure assignment attack behavior under normal classroom conditions. To adequately define a stable', predictable pattern of assignment attack for each student, baseline data were collected until future behavior level and trend could be predicted (Horner et ai., 2005). The length of baseline phases across participants approached the recommended guideline of ten data points (Horner et ai.,2005). Intervention phase, During this phase, the teacher introduced SAAS into the normal classroom routine using the script depicted in Appendix A. Assignment attack data were recorded until a clear change in behavior level, trend, and/or variability was evident. The decision of whether there was an experimental effect was based on evidence that assignment attack levels in the intervention phase were outside the range of predicted levels demonstrated in the baseline. At that point, SAAS was introduced to the next group with assignment attack data recorded for the participating student. 70 Fading ofexternal supports. The initial data points of intervention phase were recorded with the 4 P's checklist in place. When there was evidence assignment attack data had changed, the 4 P's checklist was replaced with the shorter "first letter cue" checklist for the next 2-3 data points. After a stable pattem of data was evident, the shortened checklist was removed according to the script illustrated in Figure 12 Data Collection and Analysis Assignment attack and on-task behavior were the experimental data collected on a daily basis. The researcher sat in the back of the classroom prior to the start of the resource period. Once the special education teacher prompted the students to begin working on assignments, assignment attack observation began. As soon as the student demonstrated the first three behaviors on the assignment attack rating scale (i.e., recalling assignment, gathering materials, and initiating work), scores were assigned for each domain, deducting points depending on need for teacher support according to the criteria depicted on the score sheet (see Figure 1). For example, with the "recalling assignments" domain, the evaluation criteria were as follows: 1 = Teacher must tell student assignment or show student pink sheet/folder. Student has no recollection of assignment and acts like they have no work. 2 = Teacher tells student assignment or shows student pink sheet/folder. Student acts like they know what to do but doesn't know subject and details. 3 = The student knows about assignment if asked or checks pink sheet/planner with teacher once, must get repeated teacher clarification regarding the details of the assignment (Le., teacher pointing to pink sheet/folder or verbal instructions on multiple occasions). 4 = The student knows both the subject (e.g. science) and the details of the assignment and demonstrates this by gathering materials, but may 71 ask one clarifying question of teacher. Teacher does not need to review pink sheet or folder with student. S = Student requires no support for knowing subject and assignment details. Finally, at the end of the period, the whole-class engagement score was recorded on the assignment attack rating scale. At the end of the class, the four assignment attack domains were combined to produce a total score, and the percentage of intervals on-task was calculated. On-task observation began immediately after the student initiated an assignment. The observation duration was 10 minutes, unless the student completed the task early or was asked by the teacher to do something else. On-task behavior was measured in 10 second intervals, and the student was deemed on task ifhe/she maintained engagement for at least eight seconds during the interval. A digital recording of an electronic timer set to beep at 10 second intervals was copied onto an MP3 player, and this served as the interval signal (e.g., Stahr, Cushing, Lane, & Fox, 2006). All other outcome measures (i.e., the HPC, grades, assignment completion rate, student completed probes, and the BRIEF) were collected once at the beginning and once at the end of the study. Classroom grades and assignment completion data were collected by the special education teacher and reviewed with the researcher. The student-completed probes measuring motivation, strategy use, and academic self-efficacy were administered in class by the special education teacher. The BRIEF was sent home to parents and collected at school by the special education teacher. 72 Inter-rater reliability procedure. Inter-rater reliability refers to the stability of scores across raters (Primavera, Allison, & Alfonso, 1997). When two or more researchers use the same measure to collect data, a significant threat to the internal validity of a study is poor inter-rater reliability (Gersten et aI., 2005). For this study, inter-rater reliability for both assignment attack and percentage of intervals on-task was measured using the exact percentage of agreement. Exact agreement is appropriate when data are non-dichotomous (i.e., assignment attack rating scale) and when there are only two observers (Primavera et aI., 1997). Agreement was calculated by dividing the number of agreements by the total number of observations. For assignment attack, agreement meant the same cumulative total, plus or minus two points (e.g., scores of 15 and 17 would be considered in agreement). Agreement was also calculated for each of the four domains constituting the cumulative assignment attack score. Exact agreement was defined as the same domain score, plus or minus one point. For on-task behavior, agreement meant either the same on-task 01' off-task rating in a ten-second interval. Obtaining a high degree of agreement between raters using one measure typically requires a certain amount of training (Kennedy, 2005). Reliability training for this study took place prior to baseline data collection. During the pilot study, it took three days and 12 student ratings to achieve an inter-observer reliability correlation ofr = .96. For this study, the researcher and communication disorders and sciences student observed students in the classroom, completed the assignment attack scale according to the process described above, and compared ratings for reliability training. Reliability training was 73 completed for ten measures, and exact agreement for assignment ~ttack scores was 90%. Subsequently, inter-observer agreement was measured for 28% of the total assignment attack an on-task behavior observations. Agreement for assignment attack was 89%. Agreement for each of the assignment attack domains was as follows: "Recall" was 93%; "Gather Materials" was 85%; "Initiation" was 85%; and "Whole-class Engagement" was 88%. Exact agreement for on-task behavior 91 %. 74 CHAPTER IV RESULTS This study was designed to evaluate the effects of metacognitive strategy training on self-regulation of assignment management in a resource setting for adolescents who qualify for special education. Research questions pertained to student and teacher strategy implementation (i.e., fidelity, usability, and social validity), assignment attack, and general impact on academic, behavior, and cognitive variables. This section presents the findings relative to these research questions under the general headings of Implementation, Assignment Attack, and Generalized Impact. Table 2 provides a summary of the study results, listing each dependent variable, corresponding measures, and direct,ion of related treatment effect. Implementation SAAS was implemented as prescribed across students following two days of instruction provided by the special education teacher. Initial instruction required approximately 10-15 minutes per day, and SAAS was applied to Adam's classroom first followed by Brett's and Christina's classes respectively. Subsequently, the special education teacher provided two additional instruction days for each group, once to introduce fading of the 4P's checklist and once to instruct students to repeat the 4P's internally'during assignment completion. Approximately 60 minutes of instruction were Table 2 Summary ofStudy Results 75 Results/Direction of Effect Measures. (Research Question) Implementation (R. Q. 1-3) Treatment Delivery Treatment Adherence Adam 97.1% 77.7% Brett 96.8% 90% Christina 100% 100% N/A N/A Social Validity (Teacher) Social Validity (Student) Assignment Attack (R. Q. 4) Assignment Attack Generalization (R. Q. 5-8) Classroom Grades Assignment Completion Rate Self-regulated Learning Probes (Total) BRIEF (Global Executive Composite) Homework Problem Checklist (Teacher) Homework Problem Checklist (Parent) % ofIntervals On-Task* N/A = missing data prevented evaluation of effect. * = Unable to interpret; failure to demonstrate stable baseline. + + +/- +/- + + + + +/- + N/A + + + +/- + + + + + 76 provided for each student across the intervention, fading, and maintenance phases of the study. Treatment Fidelity Observation of fidelity and teacher self-report of fidelity were measured using the form depicted in Figure 9. Treatment delivery for the special education teacher ranged from 80% to 100% (M = 97%), and the range of delivery fidelity for one educational assistant (i.e., Brett's class) was 80% to 100% (M = 96.8%). The treatment delivery fidelity mean for the other educational assistant (i.e., Christina's class) was 100%. Fidelity of treatment adherence ranges and average percentages were calculated for each student: Adam was 33% to 100% (M= 77.7%); Brett was 33% to 100% (M= 89.9%); and Christina was 100%. There were three instances of 33% treatment adherence between Adam and Brett. The respective teachers were provided feedback following each of these three sessions to point out the neglected elements. It should be noted these three days were in the first two weeks of the respective intervention phases and researcher feedback resulted in subsequent 100% adherence. These results provide affirmative support for Research Question 1 and 2; the data support a high degree of teacher implementation and student adherence using the SAAS procedure. Social Validity Social validity measures were distributed by the researcher to students and teacher after the study was completed. Students were asked to complete the measures in class, the special education teacher was asked to complete measures during a post-intervention interview. The results from the student-completed measure are presented in Table 3. 77 Table 3 Student Social Validity Data Student Question Adam Brett Christina Average 1 3* 6 6 5 2 3 1 2 2 3 3 1 4 2.7 4 4 6 3 4.3 5 4 6 6 5.3 * Each item is rated on a six point Likert-type rating scale ranging from 1 (Agree) to 6 (Do Not Agree). Student responses indicated they did not perceive the 4P's checklist and related metacognitive process helped with homework completion or academic success. In addition, there was agreement across students that they believed another approach would help them with homework completion and general academic achievement. Teacher social validity data are presented in Table 4. These findings demonstrated strong teacher endorsement of SAAS as an effective intervention. The responses indicate generally positive responses about the usability of the strategy, and the teacher reported she intends to repeat SAAS at the beginning of next school year. She shared that she plans on reinforcing the student self-reflection process by posting the 4P's checklist in a prominent spot in the classroom and asking students to write self-reflective statements at the end of class. Social validity measures (research question 3) were mixed with weak endorsement by students and strong endorsement by teachers 78 Table 4 Teacher Social Validity Data Response Item Strongly Disagree Disagree Neutral Agree Strongly Agree 1 X 2 X 3 X 4 X 5 " .. .I am excited to implement it (the 4P's) from the beginning of the year." Assignment attack To address research question 4, in-class assignment attack behavior was recorded over the course of thirteen academic weeks, not counting two weeks for winter break and one week for spring break. A total of 97 data points were collected across the three participating students: 31 for Adam, 32 for Brett, and 34 for Christina. There were seventeen instances when assignment attack data were not collected because the target behaviors did not occur (seven for Adam and ten for Brett). On these days, assignment completion performance could not be recorded because no work was attempted. If students refu.sed to participate in class and the teacher was unable to resolve the situation, no assignment attack variables could be assessed. Examples when it was not possible to take data on assignment attack included instances when a student argued he did not have 79 any work to complete, left class and returned without assignment and asked to leave class again to get ~ssignment, or refused to shift from a non-class activity such as reading a comic book. Assignment attack data were interpreted using visual analysis of raw data. For each phase in the multiple baseline design, assignment attack level, trend, and within- phase variability were analyzed for each student. Between-phase analysis also included evaluating change in level, trend, and variability. Additionally, the immediacy of effect, data overlap, and cross-subject consistency was assessed to determine the quality of the effect (Kennedy, 2005). The assignment attack data are presented in Figure 10. Adam and Brett revealed a clear change in level across phases with high average assignment 'attack during the intervention, fade, and maintenance phases relative to baseline. Christina's data reveal a high degr~e of variability and rising trend during the baseline phase, hence interpretation of intervention effects was not possible. Replication of treatment effects was thus observed across two of the three students at two different points in time. Guidelines for interpreting evidence of clear experimental control in single subject research involve demonstrating replication of effects at three distinct points in time (Homer et aI., 2005). Adam's baseline data revealed a low and stable baseline with assignment attack scores ranging between four and eight (M = 5.8). Assignment attack in the intervention Figure 10 Assignment Attack Data Adam Baseline (1) .... 200 0(f) 18 - ~ 16 - 0 co 14~ 12 - C 10 - (1) E 8 - c 6 - 0> 'iii 4 - en « Brett Intervention Fade t Maintenance r 80 (1) .... 20 -0 0 18 -(f) ~ 16 - 0 co 14 .- ~ 12 - C 10 - (1) 8 -E c 6 - 0> 'iii 4 - en « J Christina ~ 20 - 0 18 -0(f) 16 - ~ 0 14 -co ~ 12 - C 10 - (1) 8 -E c 6· 0> 'iii 4 en ~ ~ ~ ~ ~ ~« ~ ~~ § / 81 and fade phases was characterized by considerable variability, ranging between six and twenty, with a slightly delayed but distinct increase in level (M = 14.1). One data point late in the intervention phase was low (6), but this should be interpreted with caution since he was'involved in a fight before the class. Adam's assignment attack scores sustain the high level during the maintenance phase (M= 15.3). Brett's assignment attack data were slightly higher in level at baseline (M = 8.4) compared. to Adam's baseline. His data were also more variable, ranging between five and twelve, and there was a slight downward trend at baseline. There was some data overlap between Brett's baseline, intervention, and fade phases reflecting a high degree of variability, but there was a rapid and large increase in level for the intervention and fade phases (M = 14.8). Brett's assignment attack level remained high during the maintenance phase (M = 17). Christina's data reflected a high degree of variability in the baseline phase with assignment attack scores ranging from four to eighteen. Also, Christina's assignment attack began to trend up starting at the low on day nine. Despite this trend, her mean assignment attack score during baseline was 8.6, similar to Brett's baseline average. Christina's intervention and fade assignment attack levels were consistently high (M = 16.1) with very little variability compared to baseline, with scores ranging from twelve to nineteen. Due to the unstable pattern during the baseline phase, it was impossible to evaluate whether introducing SAAS contributed to this increased assignment attack level. 82 Generalized Impact Data for the generalized impact variables corresponding to research questions 5 - 8 were analyzed using initial and final individual raw scores, initial and final group means, as well as individual and group difference scores. The utility of inferential statistics to evaluate initial and final score differences was limited due to the small number of participants. Grades Classroom grades were calculated for each student based on performance at the end of the first and third grading periods during the academic calendar. At the host middle school, teachers prepared progress reports four times a year, and the data from students' social studies and math classes were used to evaluate possible academic impact of SAAS. The initial and final classroom grade data for the three students are presented in Table 5.. Classroom grade data indicated an inconsistent pattern between math and social studies performance. Across all students, the change in average math score was -14.8% while the change in average social studies grade was +13.1%. All students' math grades dropped by at least -13% while each students' social studies grade improved, ranging from +4.9% (Christina) to +26.3% (Brett). Given this inconsisten~ pattern, it is impossible to determine whether the intervention affected classroom performance since external factors (i.e., different classroom cultures) are more likely to account for the performance discrepancy. 83 Assignment Completion Rate Assignment completion rates for all schoolwork were calculated for each student based on the proportion of assignments completed divided by the total number of assignments given in a one-month period near the end of the first and third grading periods. The special education teacher assembled assignment completion data using electronic dajabases. The initial and final assignment completion data for the three students are presented in Table 5. The average assignment completion rate increased or was unchanged for all students except Adam in social studies. The average initial assignment completion rate in math for all three students was 88.3%, and the final average assignment completion rate for math was 89.3% across all students representing an average increase of 1%. Adam's math completion rate increased 3% while Brett's and Christina's completion rates were unchanged. The average initial assignment completion rate in social studies was 78.3%, and the final "average social studies completion rate was 87.3% across all students representing an average 9% increase. Brett's and Christina's social studies assignment completion rates increased 12% and 25% respectively, but Adam's final assignment completion rate in social studies dropped 10%. Table 5 Classroom Grades and Assignment Completion Rate. Academic outcome descriptive data Students (n= 3) Math Grade (%) Social Studies Grade (%) Math Completion Rate (%) Social Studies Complp.tion Rate (%) Initial Final Change Initial Final Change Initial Final Change Initial Final Change Adam 76 62 -14 69.5 77.8 +8.2 77 80 +3 89 79 -10 Brett 95 77.4 -17.6 49.5 75.8 +26.3 88 88 0 71 83 +12 Christina 85 72 -13 69 73.9 +4.9 100 100 0 75 100 +25 Average 85.3 70.5 -14.8 62.7 75.8 +13.1 88.3 89.3 +1 78.3 87.3 +9 00 +:>. 85 Self-regulated Learning An attempt to measure the constructs deemed crucial to self-regulated learning used student responses on the Self-Regulated Learning questionnaire. The constructs of interest were motivation, strategy use, and academic self-efficacy. Of note, higher scores on this measure indicated lower perceived strengths in these areas. The participating students completed the self-regulated learning probes once before the intervention and once during the maintenance phase. These data are presented in Table 6. The average cumulative score across all students decreased by 3 points (i.e., 40.3 - 37.3) indicating slightly improved perception of motivation, strategy use, and self-efficacy. Adam's and Christina's total score change was both -5, while Brett's score difference was +1. Adam and Christina reported improvements on motivation and self-effic.acy questionnaire items with little or no change on strategy use. Brett reported improvements on self-efficacy items, but indicated negative outcome on motivation and strategy use questions. Across all students, responses on motivation and self-efficacy items improved (M change = -1.3 and -3 respectively) while responses on strategy use did not improve (M change = +1.3). Se([-regulation in Non-academic Contexts The outcome measure used to assess potential changes in self-regulation outside academic confines was the parent version of the BRIEF. Data from the parent-completed BRIEF questionnaire are presented in Table 7. For each student, T scores (M = 50; SD = 10) and percentile rankings were calculated for Metacognitive Index (MI), Behavior Regulation Index (BRI), and Global Executive Composite (GEC) when possible. Change Table 6 Se(fRegulated Learning Outcome Data and Analysis. Self-Regulated Learning Probes* Students (n = 3) SRL Motivation (max = 25) SRL Strategy Use (max = 20) SRL Self-Efficacy (max = 20) SRL Total (max = 65) Initial Final Change Initial Final Change Initial Final Change Initial Final Change Adam 17 13 -4 12 12 0 16 15 -1 45 40 -5 Brett 12 15 +3 9 12 +3 15 10 -5 36 37 +1 Christina 13 10 -3 11 12 +1 16 13 -3 40 35 -5 Average 14 12.7 -1.3 10.7 12 +1.3 15.7 12.7 -3 40.3 37.3 -3 * Lower scores reflect increase in reported self-regulatory behavior. 00 0\ 87 scores were calculated for pre-post-intervention differences for each score, and lower scores indicate less evidence of dysexecutive symptoms. Adam's initial BRIEF results showed the MI and GEC both within 1.5 SD of the sample mean, which is within the clinical cutoffvalue (Gioia et aI., 2000b). Adam's BRI was exactly on this 1.5 SD cutoff at baseline. No change scores were calculated for Adam since his post-intervention BRIEF was not returned. Brett's BRIEF data showed evidence of significant executive function impairment. Due to missing data (i.e., incomplete questionnaire), no BRI or GEC difference scores were calculated for Brett. His pre-post-intervention MI difference was +15 points indicating an increase in perceived metacognitive impairment according to parental report. Christina's BRIEF data revealed clinically significant dysexecutive symptoms on the MI but not the BRI; change scores on these indices were +9 and -8 respectively. Her pre-intervention GEC indicated evidence of impairment, however Christina's final BRIEF results show a 15 point improvement, near the average score for the normative sample. Behavior The HPC was completed by teachers and parents as a pre-post- intervention comparison of perceived student behavior and efficiency while completing homework. For this measure, lower scores indicated fewer disruptive or acting-out behaviors during homework. Descriptive statistics for this measure are presented in Table 8. The results indicated a context-dependent improvement in homework-related behavior as teachers Table 7 BRIEF Outcome Data and Analysis. Students (n = 3) Adam Brett Christina BRIEF* Metacognition Index Behavior Regulation Index Global Executive Composite Initial T Final T Initial T Final T . Initial T Final T score score Change score score Change score score Change (%tile) (%tile) (%tile) (%tile) (%tile) (%tile) 58 N/A# N/A# 65 N/A# N/A# 61 N/A# N/A# (79) (91 ) (84) 68 83 +15 N/N' 95 N/N' N/N' 86 N/N' (93) (99) (99) (99) 83 92 +9 46 38 -8 66 51 -15 (99) (99) (42) (7) (92) (61 ) * Higher scores indicate greater degree executive function impairment. /\ Missing data prevented calculating scale scores. # Final questionnaire not returned. 00 00 89 reported improved behavior while parents did not. Adam's teacher-completed HPC indicated a slight decrease in disruptive homework behavior. His parent-completed post- intervention HPC was not returned. Brett's parent HPC score increased by eight points while his teacher HPC score decreased by 18 points. Christina's teacher-completed HPC decreased by nine points from 25 to 14, although it should be noted that different teachers completed the pre- and post-intervention HPC questionnaires which could have affected the reliability of the scores. Her parent-completed measure decreased by three points. These findings reveal a clinically significant change in homework behavior as reported by teachers. Table 8 Homework Problem Checklist Data Homework Problem Checklist* Students HPC-Parent HPC-Teacher (n = 3) (max = 60) (max = 60) Initial Final Change Initial Final Change Adam 38 N/A N/A 44 42 -2 Brett 52 60 +8 32 14 -18 Christina 31/57 28 -3 25 14# -9 Average 40.3 44 +2.5 33.7 28 -9.7 * Lower scores indicate fewer reported behavior problems. # Initial and final teacher HPC completed by different teachers. 90 On-task Behavior To evaluate the effect of improved assignment attack on subsequent task engagement, on-task behavior was measured immediately following assignment initiation. On-task behavior was measured using interval recording and the data were mapped onto the multiple baseline design as depicted in Figure 11. Using the guidelines for interpreting data in single subject research described above, it was evident there was no functional relation between SAAS and proportion of time students were on task. It was not possible to evaluate treatment effect because there was no stable baseline. There was a high degree of variability within and across phases for all three students resulting in significant data overlap. In addition, there was no change in variability following the intervention. While the intervention did not specifically address task engagement, it was hypothesized that once these students initiated a task they would remain engaged. Visual analysis of the data indicated this was not the case. Pearson Product-Moment Correlation coefficients were calculated to evaluate whether on-task and assignment attack measures were related. These data are presented in Table 9. Consistent with evidence from visual analysis, the findings indicated no relationship between cumulative assignment attack scores and the percentage of intervals on-task. Similarly, there was little evidence assignment attack domains or on-task were related with one exception. The correlation coefficients between whole class engagement and the proportion of intervals on-task ranged from r = .433 (Adam) to r = .711 (Brett) indicating a small to moderate relationship between the measures (i.e., Cohen, 1992). 91 Figure 11 Percentage ofIntervals On Task Adam Baseline -'"~ f- 100 -c: 0 '" 80 -«i C OJ 60 -E "- 0 40 -OJgp E 20 - OJ <.) .... OJ 0-c... Intervention Fade / Maintenance Brett -;:; 100 - '"f- c: 80 -0 '"«i C 60 - OJ E "- 400 OJ •gp E 20 - OJ <.) .... OJ c... o - Christina -;:; 100 - '"f- c: 80 -0 '" j«iC 60 -Ec:"- 40 -0 •OJ OJ) '" 20 -E OJ 2