SCHOOL WIDE POSITIVE BEHAVIOR INTERVENTIONS AND SUPPORTS (SWPBIS) TIERED FIDELITY INVENTORY (TFI): A PREDICTOR OF STUDENT OUTCOMES by LILLIAN ROSE GROFF A DISSERTATION Presented to the Department of Educational Methodology, Policy, and Leadership and the Graduate School of the University of Oregon in partial fulfillment of the requirements for the degree of Doctor of Education June 2020 DISSERTATION APPROVAL PAGE Student: Lillian Rose Groff Title: School Wide Positive Behavior Interventions and Supports (SWPBIS) Tiered Fidelity Inventory (TFI): A Predictor of Student Outcomes This dissertation has been accepted and approved in partial fulfillment of the requirements for the Doctor of Education degree in the Educational Methodology, Policy and Leadership by: Gina Biancarosa Chairperson Julie Alonzo Core Member Keith Zvoch Core Member Kent McIntosh Institutional Representative and Kate Mondloch Interim Vice Provost and Dean of the Graduate School Original approval signatures are on file with the University of Oregon Graduate School. Degree awarded June 2020 ii © 2020 Lillian Rose Groff This work is licensed under a Creative Commons Attribution (United States) License. iii DISSERTATION ABSTRACT Lillian Rose Groff Doctor of Education Educational Methodology, Policy, and Leadership June 2020 Title: School Wide Positive Behavior Interventions and Supports (SWPBIS) Tiered Fidelity Inventory (TFI): A Predictor of Student Outcomes Various instruments have been created to measure the implementation fidelity of SWPBIS. The most recent of the measures is the Tiered Fidelity Inventory (TFI), designed as a complete index of the fidelity of implementation across the three-tiered framework to guide planning, as a progress-monitoring tool to strengthen fidelity, and as a formative measure for tiers that are already being implemented. To date there has not been a study conducted linking the TFI subscales to lower Office Discipline Referrals (ODRs). This study explored which factors within Tier 1 of the TFI correlate with lower number of ODRs, and to what extent that relationship is dependent on whether the TFI was completed with or without an external coach. Participants were 1,008 (levels PreK-8) schools selected from an extant data base chosen based on their completion of TFI data and use of the SWIS data system. Results indicated little notable difference between correlations among the TFI variables and ODRs when accounting for the use of an external coach. The only notable result was that the TFI Team and Implementation scores were significantly and negatively correlated with ODRs, and when external coach presence was considered the relationship was only significant between Team scores and ODRs in the presence of an external coach. iv CURRICULUM VITAE NAME OF AUTHOR: Lillian Rose Groff GRADUATE AND UNDERGRADUATE SCHOOLS ATTENDED: Oregon State University, Corvallis University of Oregon, Eugene DEGREES AWARDED: Doctor of Education, Educational Leadership, 2020, University of Oregon Master of Science, Special Education, 2013, University of Oregon Bachelor of Science, English and Writing, 2009, Oregon State University PRESENTATIONS: Behavior Support Plans: If WE Design Them, WE Can Use Them, ORTII, 2018 Co Presenter, Implementation Science 101, ORTII, 2018 Co-Presenter, Behavior Support Plan Fidelity Round Table, NWPBIS, 2018 Co-presenter, Implementation Science 101, NWPBIS, 2018 PROFESSIONAL EXPERIENCE: Behavior consultant, 4J School District, Eugene, Oregon, 2018-Current Special Education Teacher, 4J School District, Eugene, Oregon, 2014-2018 Special Education Teacher, Sweet Home School District, Sweet Home, Oregon, 2013-2014 GRANTS, AWARDS, AND HONORS: Sammie Barker McCormack Scholarship, 2019 Travel Grant, EMPL Department, 2019 Alumni Scholarship, College of Education, 2018 Sammie Barker McCormack Scholarship, 2017 Equity Fellowship Scholarship, 2017 v Dedicated to my mom, my forever cheerleader, and my daughter, the light of my life. vi TABLE OF CONTENTS Chapter Page I. INTRODUCTION .................................................................................................... 1 Fidelity of Implementation .................................................................................... 1 SWPBIS ................................................................................................................. 3 Implementation Fidelity of SWPBIS ..................................................................... 5 Measuring SWPBIS Implementation Fidelity ....................................................... 8 Summary and Study Context ................................................................................. 10 II. METHODS ............................................................................................................. 14 Research Design .................................................................................................... 14 Sampling ................................................................................................................ 14 Instrumentation ...................................................................................................... 15 Office Discipline Referrals .................................................................................... 17 Data Collection and Analysis ................................................................................. 18 Research Question 1 ........................................................................................ 19 Research Question 2 ........................................................................................ 19 Research Question 3 ........................................................................................ 19 III. RESULTS .............................................................................................................. 21 RQ 1 ....................................................................................................................... 21 RQ 2 ....................................................................................................................... 25 RQ 3 ....................................................................................................................... 26 vii Chapter Page IV. DISCUSSION ....................................................................................................... 29 Limitations ............................................................................................................. 31 Validity ............................................................................................................ 31 Internal Validity ................................................................................... 31 External Validity ................................................................................. 32 Practical Implications ............................................................................................ 32 SWPBIS ........................................................................................................... 32 Practitioner ...................................................................................................... 33 Future Research ........................................................................................................... 35 APPENDICES ............................................................................................................. 37 A. LITERATURE SEARCH AND REVIEW ....................................................... 37 B. TIERED FIDELITY INVENTORY (TFI): TIER 1 ......................................... 40 REFERENCES CITED ............................................................................................... 46 viii LIST OF FIGURES Figure Page 1. Implementation Drivers created by Fixsen and Blase (2008) ............................... 3 2. PBIS Framework (Image Created by Anne Arundel County Public Schools) ...... 4 ix LIST OF TABLES Table Page 1. TFI Items by Tier and Subscale. ............................................................................ 11 2. Correlations Between TFI and Existing Measures of Fidelity of Implementation by Administration Condition. ................................................................................ 17 3. Descriptive Statistics and Correlations for Key Study Variables .......................... 23 4. Descriptive Statistics with and without a Coach ................................................... 24 5. Correlations Among Key Variables for Sites Where the TFI was Completed with and without an external coach ............................................................................... 25 6. Regression Analysis Summary for Key Variables Predicting ODRs per student ................................................................................. 27 7. Regression Analysis Summary for Key Variables Predicting ODRs per student ................................................................................. 28 x CHAPTER I INTRODUCTION Fidelity of Implementation Fidelity to implementation, or treatment integrity of interventions, can be defined as the use of Evidence Based Practices (EBPs) according to identified criteria outlining activities, materials, and behaviors that result in improved outcomes (Smith, Daunic, & Taylor, 2007). Flannery, Fenning, Kato, and McIntosh (2014) found that when the fidelity of School Wide Positive Behavior Intervention and Supports (SWPBIS) implementation increased, exclusionary discipline practices decreased. It is often assumed that educators are experts with all aspects of every intervention. However, all educators are not masters in every subject area. Therefore, interventions and school-wide initiatives should always be accompanied by a fidelity of implementation plan and/or measure to support teachers’ effective use of a practice (King-Sears, Walker & Berry, 2018). High fidelity for teachers’ use of interventions can be dependent on how well school personnel translate fidelity from research to practice (King-Sears et al., 2018). Teachers need clear parameters about what the intervention is, what it looks like, how to use it, and how to ensure implementation as intended. Therefore, capacity (i.e., knowledge, skills, abilities) and resources (e.g., funding for resources and training) should be well thought-out when considering the use of any EBP (Fixsen, Blase, Metz & Van Dyke, 2013). Several implementation Science frameworks exist including the Exploration, Preparation, Implementation, Sustainment (EPIS) framework created by Moullin et al., 1 (2019), a Conceptual framework for implementation fidelity created by Carroll et al. (2007), the Active Implementation Frameworks created by Fixsen and Blase, 2008, and the Consolidated Framework for Implementation Research (CFIR) created in 2009- just to name a few. A common theme among them appears to be that Implementation is a stage-based process. I chose to focus on the Active Implementation Frameworks as I am most familiar with it in practice. Current implementation science research suggests there are eight Implementation Drivers: selection, performance assessment (fidelity), decision support data system, facilitative administration, systems intervention, leadership, implementation components that are integrated and compensatory (Fixsen and Blass, 2008; see Figure 1). Of these drivers, I focused exclusively on the performance assessment (fidelity) driver, designed to assess the use of skills that are trained, expanded through coaching, and reinforced in iterative cycles (Fixsen et al., 2013; Metz et al., 2015). The Performance Assessment (fidelity) Driver includes three measures: context, compliance, and competence. Context measures identify and clearly define necessary training, availability of resources, caseload limit, etcetera for an intervention. Compliance measures clearly outline intervention components and their intended use. Competence measures determine the extent to which the interventions components were implemented with fidelity. Together this fidelity data can be used to inform professional development and assess training quality. Kaderavek and Justice (2010) noted that fidelity should be determined hand in hand with targeting EBPs because whether an intervention works as expected can be dependent on whether the intervention was implemented as intended. Figure 1. Implementation Drivers created by Fixsen and Blase (2008) SWPBIS SWPBIS (see Figure 2) is a multi-tiered system of support implementation framework designed to improve academic and behavioral outcomes for students while reducing disciplinary exclusion and disproportionality through the use of efficient and effective systems (Gage, Whitford, & Katsiyannis, 2018; Sugai & Simonsen, 2012). SWPBIS operates on three tiers: Universal (Tier 1), Targeted (Tier 2), and Intensive (Tier 3). Tier 1 practices are designed to prevent school-based problem behaviors by establishing, teaching, and reinforcing positive behavior expectations. Students needing 3 more support are referred for Tier 2. Tier 2 focuses on evidence-based interventions like check-in check-out, check and connect, social emotional instruction, and relationship building. Students who continue to show need for more support are referred for Tier 3, which generally involves conducting a functional behavior assessment (FBA), creating and implementing a behavior support plan (BSP). SWPBIS emphasizes the use of data for informed decision making and the organization of resources and systems in a way that improves sustained implementation fidelity (Sugai & Simonsen, 2012). Throughout each tier, schools engage in team-based, universal screening, and data-based decision making to build capacity for sustained implementation and improved student behavioral outcomes (Gage et al., 2018). When implemented to fidelity, SWPBIS is designed to support improved student outcomes for all students. Figure 2. PBIS Framework (Image created by Anne Arundel County Public Schools) 4 Implementation Fidelity of SWPBIS To examine implementation fidelity of SWPBIS McIntosh et al. (2017) conducted three studies that evaluated the psychometric properties of the TFI: a content validity study, a usability and reliability study, and a large-scale validation study. In the content validity study, participants with one of the following criteria were invited to participate; (a) either a researcher of SWPBIS with at least two published studies using or reporting SWPBIS fidelity of implementation data in the past 10 years or (b) an experienced SWPBIS implementer with at least 15 years of experience as a school or district implementer and team trainer. In the content validity study, survey data were collected to measure if the specific items of the TFI represented implementation of SWPBIS. Each school completed the TFI, with four other implementation measures (the Benchmarks of Quality (BoQ), Self-Assessment Survey (SAS), Team Implementation Checklist (TIC), and Benchmarks of Advanced Tiers (BAT)). McIntosh et al. (2017) found strong interrater agreement (α= .99) and test-retest reliability (α= .99), high internal consistency (α=.96), and high usability for action planning (easy and straightforward process: 100% agree, easy and straightforward scoring: 93% agree, validity for assessing fidelity: 100% agree). Results from the content validity study showed that the items, scoring criteria, and perceived factor structure were valid for assessing SWPBIS implementation. The concurrent validity analyses showed statistically significant correlations with existing SWPBIS fidelity measures. Scores from all five measures were found to be similar in their indication of implementation level (Mercer et al., 2017). The authors did, however, caution that all five measures were self-report, and teams could have inflated their scores. 5 The authors also suggested using improved student outcomes concurrently to measure improvements in fidelity of implementation, as higher fidelity scores of SWPBIS implementation are associated with increased student outcomes (i.e., lower percentages of office discipline referrals (ODRs) and less exclusionary discipline practices) (Flannery et al., 2014). External coaching and technical assistance were found to strengthen the reliability of TFI results (Mercer et al., 2017). Kittelman, Eliason, Dickey, and McIntosh (2018) conducted a study to determine how often schools completed the TFI during the school year as well which tiers were assessed, and the average scores at each tier. The researchers found 61% of schools completed the TFI once during the 2016-2017 school year, and that only 37% completed all three tiers at once. A quarter of schools completed the TFI twice per year, and even fewer schools completed the TFI three times per year. A large percentage of schools completed Tier 1 of the TFI (96%), a little over half of schools completed Tier 2, and a third of schools completed Tier 3. Third, the authors looked at the average scores of each tier. Tier 1 scores were found to average 74% (slightly over the 70% implementation criteria validated for Tier 1), Tier 2 scores averaged to 69%, and Tier 3 scores averaged to 62%. Kittelman et al. (2018) recommended schools complete the TFI at the beginning of the year with an external coach. Schools are then encouraged to create an action plan, progress monitor the tier of focus (with the TFI) every three to four months, and review action plan progress at each meeting. Massar, McIntosh, and Mercer (2019) conducted a study to assess the extent to which the TFI could be used as one measure of all three tiers and individual tiers, as well as a way to measure implementation on 10 subscales across all three tiers. The 6 confirmatory factor analysis (CFA) found that the revised TFI had consistent factor structure and a strong model fit. The authors found the TFI to be a valid measure of fidelity of SWPBIS implementation to measure by subscale, tiers, and as a comprehensive assessment for all three tiers (Massar et al., 2019). In accordance with previous studies conducted, the authors found scores to be stronger when the TFI was completed with the help of an external coach. Massar et al. (2019) state that these results are important in the ongoing support and effort to increase schools use of fidelity of implementation, and databased decision making to improve outcomes for students, teachers, and the school environment. McIntosh et al. conducted a study to compare the difference in TFI scores when implemented by an external coach versus without the support of an external coach. Researchers reported a moderate positive correlation (r = .64) between the TFI Tier 1 and the BoQ fidelity measure when SWPBIS was implemented with the support of an external coach, but only a weak correlation (r =.42) when SWPBIS was implemented without the support of an external coach. In their study, the external coach was an employee of the school district who helped facilitate the administration of the TFI, ensured accuracy of scoring, and guided the team through interpreting the results (McIntosh et al., 2017). Similarly, the researchers found a moderate positive correlation (r =.54) between the TFI Tier 1 and the TIC fidelity measure when SWPBIS was implemented with the support of an external coach and a slightly lower correlation (r = .42) when SWPBIS was implemented without the support of an external coach. This same pattern continued, with the relation between the TFI Tier 1 and SAS fidelity measure (r = .55) and the BAT stronger (r = .75) stronger when SWPBIS was 7 implemented with the support of an external coach, as compared to implementation without a coach (r = .36 and r = .474, respectively). Consistently, the authors found TFI scores to be more reliable and better aligned to other related measures when completed by an external coach (McIntosh et al., 2017). Multiple researchers suggest using an external coach to strengthen TFI scores (Kittleman et al., 2018; Massar et al., 2019; McIntosh et al., 2017; Mercer et al., 2017). The need for an external coach to complete the TFI may be due to the fact that the external coach can offer a more objective point of view in their assessment of SWPBIS efforts. Validity research on the TFI shows that school teams are more accurate when an external coach facilitates TFI completion (McIntosh et al., 2017). This finding has strong implications for fidelity of SWPBIS by district and school staff. Not all districts and schools have resources and/or access to an external coach, which may have important implications for the initial adoption of SWPBIS as well as the continued and sustained implementation efforts. Adequate resources are shown to develop the capacity of staff (Fixsen et al., 2013). Measuring SWPBIS Implementation Fidelity The TFI was created to offer schools and districts a valid, reliable, and efficient way to measure the extent to which school staff apply core features of SWPBIS (Algozzine et al., 2017) at all three tiers. The TFI was partially inspired by the Benchmark of Quality (BoQ), which measures Tier 1 implementation fidelity (Kincaid, Childs, & George, 2010). To date the TFI is the only SWPBIS implementation measure that addresses all three tiers. I specifically chose to use the TFI because it is the measure I 8 use as an Individual Positive Behavior Supports (IPBS) coach within my current school district. Tier 1 of the TFI assesses Universal SWPBIS Features, with 15 items distributed across three subscales: teams, implementation, and evaluation. Tier 2 assesses Targeted SWPBIS Features, using 13 items, organized into three subscales: teams, interventions, and evaluations. Finally, Tier 3 of the TFI assesses Intensive SWPBIS Features, with 17 items, distributed across four subscales: teams, resources, support plans, and evaluations (See Table 1). The TFI, available both online and in a paper/pencil format, is intended to be administered by a School Systems Planning Team with input from site-based Tier 1, 2, and 3 teams. The developers of the TFI strongly recommend that the TFI be administered by an external SWPBIS coach as the facilitator to ensure more accurate results. The TFI can be administered at any of the Tiers either individually or in any combination (e.g. Tier 1 and Tier 2, Tier 1, Tier 2, Tier 3, or Tier 1, 2 and 3) and takes 30 minutes per Tier to administer. Criteria for scoring each item on the TFI reflect the level of implementation (0=Not implemented, 1=Partially Implemented, and 2=Fully Implemented). Once the TFI has been completed, scores are entered online through PBISAPPS.org. The TFI then produces scale and subscale scores that indicate the extent to which Tier 1, 2 and 3 core features are in place. The developers suggest that the TFI be administered upon initial implementation and then every third or fourth meeting and report that a score of 70% or higher (for Tier 1) indicates a level of implementation that will result in improved student outcomes. Once a team has 9 met 70% fidelity on Tier 1 across three consecutive administrations, the TFI should be completed annually to ensure sustained implementation (Algozzine et al., 2017). The TFI was developed for three purposes (a) as an initial assessment to determine the extent which a school is using SWPBIS, (b) as a measure of SWPBIS fidelity of implementation at all three tiers of support, and (c) as a tool to guide action planning for further implementation efforts (Algozzine et al., 2017). The TFI is used to guide planning, as a progress-monitoring tool to strengthen fidelity, and as a formative measure for tiers that are already being implemented. Six subscales exist within the measure: teams, implementation, evaluation, interventions, resources, and support plans (Massar et al., 2019, p. 8). The TFI provides teams with scale and subscale scores indicating the extent to which Tier I, Tier II and Tier III core features are in place (Algozzine et al., 2017). The scores and item reports are intended to provide coaching guidance and action planning for the implementation of SWPBIS (Massar et al., 2019, p. 8). Massar et al. (2019) reported validity evidence supporting the TFI’s use for measuring fidelity of implementation, whether used as a comprehensive assessment for all tiers, by subscale, or by a sub-set of tiers. Summary and Study Context Many instruments have been created to measure the implementation fidelity of SWPBIS, most recently the Tiered Fidelity Inventory (TFI). The TFI was designed (a) as a complete index of fidelity of implementation across the three-tiered framework to guide planning, (b) as a progress-monitoring tool to strengthen fidelity, and (c) as a formative measure for tiers that are already being implemented (Algozzine et al., 2014). The scores 10 Table 1 TFI Items by Tier and Subscale Tier Subscales Items Scores I: Universal 0-30 SWPBIS Features 1. Teams Team Composition, Team Operating 0-4 Procedures 2. Implementation Behavioral Expectations, Teaching 0-18 expectations, Problem Behavior Definitions, Discipline Policies, Professional Development, Classroom Procedures, Feedback and Acknowledgement, Faculty Involvement, Student/Family/Community Involvement 3. Evaluation Discipline Data, Data-Based Decision 0-8 Making, Fidelity Data, Annual Evaluation II: Targeted 0-26 SWPBIS Features 1. Teams Team Composition, Team Operating 0-8 Procedures, Screening, Request for Assistance 2. Interventions Options for Tier II Interventions, Tier II 0-10 Critical Features, Practices Matched to Student Need, Access to Tier I Supports, Professional Development 3. Evaluation Level of Use, Student Performance Data, 0-8 Fidelity Data, Annual Evaluation III: 0-34 Intensive SWPBIS Features 1. Teams Team Composition, Team Operating 0-8 Procedures, Screening, Student Support Teams 2. Resources Staffing, Student/Family/Community 0-6 Involvement, Professional Development 3. Support Plans Quality of Life Indicators, Academic, 0-12 Social, and Physical Indicators, Hypothesis Statement, Comprehensive Support, Formal and Natural Supports, Access to Tier I and Tier II Supports 4. Evaluation Data Systems, Data-Based Decision 0-8 Making, Level of Use, Annual Evaluation 11 and item reports are intended to provide coaching guidance and action planning for the implementation of SWPBIS (Massar et al., 2018, p. 8). Data from several previously conducted studies have found there to be a positive impact between school wide implementation of SWPBIS and student outcomes (Bradshaw, Mitchell, & Lea, 2010; Childs, Kincaid, George, & Gage, 2015; Flannery et al., 2014; Nocera, Whitbread, Nocera, 2014). Effects of SWPBIS implementation fidelity showed a statistically significant inverse relation between SWPBIS fidelity of implementation (using the SET as the measure) and student problem behaviors (Flannery et al., 2014). Several studies using previous measures of fidelity of implementation (the BoQ, the SET, the TIC, and the SAS) were associated with better student outcomes with high fidelity of SWPBIS implementation. To date there has not been a study conducted examining the relations of parts of the TFI (subscales) with lower major ODRs. I am interested in which factors within Tier 1 of the TFI are associated with lower ODRs. In this case, I used the Tiered Fidelity Inventory (TFI), which measures the fidelity of implementation of SWPBIS at all three Tiers (evaluation teams in all three tiers, implementation in Tier 1, interventions in Tier 2, resources and support plans in Tier 3), as an indicator of implementation fidelity. The developers of the TFI suggest that a score of 70% or greater at Tier 1 is indicative of acceptable levels of implementation fidelity (Algozzine et al.,2014.) This study investigated the following questions: (1) What are the relationships among the TFI Tier 1 total score, scale scores, subscale scores and ODRs per student? (2) To what degree does the total TFI Tier 1 total score predict major ODRs per student, (when controlling for school level covariates typically associated with 12 ODRs) and to what extent does that relationship depend on whether the TFI was completed with or without an external coach? (3) To what degree does the Tier 1 teams subscale score predict ODRs per student, (when controlling for school level covariates typically associated with ODRs) and to what extent does that relationship depend on whether the TFI was completed with or without an external coach? 13 CHAPTER II METHODS Research Design This study utilized a correlational design to examine parts of the TFI to determine which parts were associated with higher fidelity of implementation (overall TFI score) and lower total major ODRs per student. A subscale of the TFI is labeled implementation, however in this study fidelity of implementation refers to the overall and tiered TFI scores. I conducted a secondary analysis of an extant data set from 2015-2016 provided to me through Educational and Community Supports (ECS) collected through the School Wide Positive Behavior Intervention and Supports (SWPBIS) School Wide Information System (SWIS) at the University of Oregon. The unit of analysis for this study was at the organizational level, the school. Sampling A nonprobability (convenience) sampling method was used (Creswell, 2014). The study sample includes public schools (pre-K and elementary) who completed Tier 1 of the TFI in the 2015-2016 school year and who also used the School Wide Information System (SWIS). The original data set included 4,412 schools; 1,008 of those schools utilized the TFI at the Tier 1 level as well as the SWIS data system to collect major ODRs. Schools were located across the United States (Arizona, California, Connecticut, Iowa, Illinois, Kentucky, Massachusetts, Michigan, Minnesota, Missouri, Montana, North Carolina, New York, Ohio, Oregon, Texas, Washington, Wisconsin); 21 schools did not identify their state. 14 Instrumentation The TFI was used as the main measure within this study. The TFI was created by Algozzine et al. (2014) to offer schools and districts a valid, reliable, and efficient measure of the extent to which school staff apply core features of SWPBIS. The core features of SWPBIS include (a) defining and teaching behavioral expectations, (b) monitoring and acknowledging expected behavior, (c) correcting behavioral errors using a continuum of consequences, and (d) using the information for decision-making (Sugai and Simonsen, 2012). The TFI was developed for three purposes: (1) as an initial assessment to determine the extent which a school is using SWPBIS, (2) as a measure of SWPBIS fidelity of implementation at all three tiers of support (Tier I: Universal SWPBIS Features; Tier II: Targeted SWPBIS Features; and, Tier III: Intensive SWPBIS Features), and (3) as a tool to guide action planning for further implementation efforts (Algozzine et al., 2014). Tier 1 of the TFI measures Universal SWPBIS features and has 15 items that measure three subscales: teams, implementation, and evaluation. Tier 2 of the TFI measures Targeted SWPBIS features and has 13 items that measures three subscales: teams, interventions, and evaluations. Tier 3 of the TFI measures Intensive SWPBIS features and has 17 items that measures four subscales: teams, resources, support plans, and evaluations. Table 1 shows each item construct by tier and subscale. Criteria for scoring each item on the TFI reflects level of implementation (0=Not implemented, 1=Partially Implemented, and 2=Fully Implemented). TFI scores are sums of several items. 15 One example of an item on the Tier 1 team subscale focuses on team composition: Tier I team includes a Tier I systems coordinator, a school administrator, a family member, and individuals able to provide (a) applied behavioral expertise, (b) coaching expertise, (c) knowledge of student academic and behavior patterns, (d) knowledge about the operations of the school across grade levels and programs, and for high schools, and (e) student representation. One example of an item on the implementation subscale focuses on behavioral expectations: School has five or fewer positively stated behavioral expectations and examples by setting/location for student and staff behaviors (i.e., school teaching matrix) defined and in place. An example of an item on the evaluation subscale focuses on discipline data: Tier I team has instantaneous access to graphed reports summarizing discipline data organized by the frequency of problem behavior events by behavior, location, time of day, and by individual student. The developers of the TFI strongly recommend that the TFI be administered by an external SWPBIS coach as the facilitator to ensure more accurate results (Algozzine et al., 2014). The TFI can be administered at any of the tiers either individually or all together (e.g., Tier 1 and Tier 2, Tier 1, Tier 2, Tier 3, or Tier 1, 2 and 3). Once the TFI has been completed, the scores are then entered online through PBISAPPS.org. The TFI then produces scale and subscale scores that indicate the extent to which Tier 1, 2 and 3 core features are in place (Algozzine et al., 2014). The TFI can then be used to guide planning, as a progress-monitoring tool to strengthen fidelity of PBIS implementation, and as a formative measure for PBIS tiers that are already being implemented. The TFI takes approximately 30 minutes per Tier to administer. It can either be completed in a paper pencil format, or online. Once a team has met a high level of fidelity across three 16 consecutive administrations it is suggested that the TFI be completed annually to ensure sustained implementation. The TFI was created for schools implementing SWPBIS K-12. The scores and item reports are intended to provide coaching guidance and action planning for the implementation of SWPBIS (Massar et al., 2018, p. 8). McIntosh et al. (2017) found strong Interrater Agreement (.99) and test-retest reliability (.99), high internal consistency (.96), and high usability for action planning (easy and straightforward process: 100% agree, easy and straightforward scoring: 93% agree, validity for assessing fidelity: 100% agree).The TFI was also found to be highly correlated with other measures of SWPBIS implementation as well. Table 2 illustrates the correlations between the TFI and existing measures of fidelity of implementation (BoQ, SAS, and TIC) by administration condition (McIntosh et al., 2017). Table 2 Correlations Between TFI and Existing Measures of Fidelity of Implementation by Administration Condition Measures External Coach No External Coach TFI Tier 1 and BoQ .416 .643 TFI Tier 1 and SAS .364 .551 TFI Tier 1 and TIC .258 .544 Office Discipline Referrals I reviewed total major Office Discipline Referral Data (ODR) for each public school (K-12) that completed the TFI and used the SWIS data base to enter major ODRs for the 2015-2016 school year (and signed an optional data sharing agreement). “ODRs are written records of schoolwide behavioral issues commonly collected in most schools…” (Flannery et al., 2014). There are two categories of ODRs: majors and 17 minors. Major ODRs are used to document high intensity behaviors like physical aggression and property destruction. Minor ODRs are often more subjective in nature some examples are: not following directions, being disruptive, physical contact, use of inappropriate language, and being unprepared for class. To generate the number of ODRs per student in a school, which was used in analyses, the number of major ODRs was divided by the total school enrollment. Data Collection and Analysis A secondary data analysis was conducted using the extant data set. I exported data from the extant data set on school demographics, level (pre-K and elementary), TFI scores, and major total ODRs for all public PreK and elementary schools that competed Tire 1 of the TFI and that used SWIS. Moreover, I used the IBM Statistical Package for the Social Sciences (SPSS Version 25). Data was displayed both descriptively and visually through the use of tables and graphs based on the findings of each research question. I conducted an analysis using nonparametric correlations and multiple regression in order to determine the relationship among implementation scores, tiers, major total ODRs per student, and the team subscale. For all of my research questions I used the same analysis approach by using the same statistical tests and alpha level of .05 for determining statistical significance. I first ran a descriptive exploration to identify to what degree any of these variables are related to each other. This included looking at the mean, standard deviation, histograms, skewness and kurtosis as well as the correlations. Below I describe the analysis for each research question. 18 Research Question 1. What are the relationships among the TFI Tier 1total scores subscale scores and ODRs per student? There are three subscales in Tier I (i.e., Teams, Implementation, Evaluation). The independent variables are the Tier 1 total TFI scores and the TFI scale subscale scores (each of which was the sums of several items). There are three subscales in Tier I (i.e., Teams, Implementation, Evaluation). The dependent variable was total major ODRs per student and was continuous. To address this question, correlation analyses were conducted in SPSS to examine the relationships between each of the TFI subscales and major ODRs per student. I visually examined bivariate scatter plots to see if the assumption that the relationships are linear were tenable; they were not, Spearman’s r was used. Research Question 2. To what degree does the total TFI Tier 1 score predict major ODRs per student, (when controlling for school level covariates typically associated with ODRs) and to what extent does that relationship depend on whether the TFI was completed with or without an external coach? The independent variable was total Tier 1 TFI score. The dependent variables were major ODRs per student and the use of an external coach to conduct the TFI measure. To address this question, a multiple regression was conducted in SPSS to examine the relationships between the total Tier 1 TFI scores, ODRs per student, and the use of an external coach to complete the measure. Research Question 3. To what degree does the Tier 1 teams subscale score predict ODRs per student, (when controlling for school level covariates typically associated with ODRs) and to what extent does that relationship depend on whether the TFI was completed with or without an external coach? The independent variable was the total Tier 1 team subscale score. The dependent variables were major ODRs per student 19 and the use of an external coach to conduct the TFI measure. To address this question, a multiple regression was conducted in SPSS to examine the relationships between the total Tier 1 team subscale score, ODRs per student, and the use of an external coach to complete the measure. 20 CHAPTER III RESULTS The purpose of this dissertation was to determine the relationship among TFI implementation scores and the team subscale for Tier 1 as well as major total ODRs per student. In calculating major ODRs per student I used school reported enrollment. NCES was not used because there were 21 schools with missing NCES data. To calculate the proportion of non-white students I subtracted the school’s white enrollment from the total enrollment divided by total school enrollment. Below I will describe the results for each research question. Due to the convenience nature of the sample caution will be used when interpreting results, specifically with regards to generalizability. RQ 1: What are the relationships among the TFI Tier 1 total score, subscale scores and ODRs per student and to what extent do those relationships differ for sites where the TFI was completed with or without an external coach? I first ran a descriptive exploration of the variables prior to running correlations to identify to what degree any of these variables were related to each other (see Table 3). Descriptive analysis included looking at the mean, standard deviation, minimum, maximum, histograms, skewness and kurtosis. The scores for Tier 1 Total Points ranged from 0-30 (M= 24). The scores for Team Total range from 0 to 4 (M=3.3). The scores for Implementation Total range from 0-18 (M=15). The scores for Evaluation range from 0-8 with (M=6.5). The number of total annual major ODR per student ranged form .00 per student to 5.3 ODRs assigned per student (M=.56). Non-white proportion (M= .5), Title-1 Funding (M=.85) and Coaching (M=.68) scores all ranged from .00 to 1. Next, correlations were conducted in SPSS to examine the relationships between 21 each of the Tier 1 TFI subscales and major ODRs per student (see Table 3). Spearman’s rho was used because assumptions of normal distribution were not met. Overall Tier 1 total points were positively correlated with each TFI variable: implementation total (.935) evaluation total (.807), and Team total (.598) (see Table 3). Major ODRs per student were negatively correlated with the Tier 1 total score, meaning as TFI Tier 1 total scores were higher major ODRs per student were lower. To address the second part of the research question, to what extent does that relationship depend on whether the TFI was completed with or without an external coach, descriptive exploration and correlations were again conducted. Overall teams who did not use an external coach scored higher on Tier 1 of the TFI when compared to teams who did use an external coach (see Table 4). Minimum and maximum scores for variables with and without a coach were identical to Table 3, with the exception of ODRs. The maximum number of ODRs per student were higher with the use of an external coach (5.27) when compared to schools who did not use an external coach (4.04). Given the distribution I again used nonparametric correlations. Correlations with an external coach are below the diagonal and correlations without a coach are above the diagonal in Table 5. There was little notable difference between correlations among the TFI variables and the use of an external coach. Major ODRs per student were negatively correlated with the Tier 1 total score, Tier 1 team total score, and Tier 1 implementation total, meaning higher TFI Tier 1 total scores were associated with lower major ODRs per student. Major ODRs per student were positively correlated with Evaluation tools for sites where the TFI was completed with (.018) or without (.019) an external coach. The non-white proportion of student variable were positively correlated with Title 1 funding 22 Table 3 Descriptive Statistics and Correlations for Key Study Variables Variables M SD Min Max Skewness Kurtosis 1 2 3 4 5 6 7 8 1. Tier 1 Total 24 5.5 0 30 -2.0 5.4 .60** .94** .81** -.05 -.06 .03 -.05 2. Team Total 3.3 .88 0 4 -1.4 2.3 .43** .42** -.08* .03 .03 .03 3. Implementation Total 15 3.5 0 18 -1.8 4.1 . .61** -.07* -.06* .03 .07* 4. Evaluation Total 6.5 1.7 0 8 -1.7 3.4 .02 -.04 .02 -.04 5. Major ODRs per Student .56 .60 .00 5.3 2.6 10 -.14** .17** .02 6. Non-White Proportion .50 .32 .00 1 .18 -1.4 .13** .05 7. Title 1 Funding .85 .31 .00 1 .17 -1.4 -.03 8. Coach .68 .47 .00 1 -.77 -1.4 Note. Spearman’s rho was used because assumptions of normal distribution were not met. n = 1008 * p < 0.05 ** p < 0.01 23 and moderately correlated with whether or not Tier 1 of the TFI was completed with a coach. Recipients of Title 1 funding saw a slightly negative correlation with the use of a coach to complete Tier 1 of the TFI. The non-white proportion of student variable was positively correlated with Title 1 funding and was moderately correlated with whether or not Tier 1 of the TFI was completed with a coach. Recipients of Title 1 funding saw a slightly negative correlation with the use of a coach to complete the Tier 1 of the TFI. I then visually examined bivariate scatter plots to see if the assumption that the relationships were linear was tenable. All variables were null in relation to major ODRs per student. Upon looking at the matrix scatter plot there appears to be close to no relationship among variables. All histograms for TFI Tier 1subscales and total scores showed a negative skewed distribution, or high score. The histograms for ODRs per student was positively skewed, meaning ODRs were on the lower end. Table 4 Descriptive Statistics with and without a Coach Variables w/Coach (n = 683) M SD Min Max Skewness Kurtosis 1. Tier 1 Total 24 5.5 0 30 -1.5 5.4 2. Team Total 3.3 .87 0 4 -1.5 2.6 3. Implementation Total 14 3.6 0 18 -1.8 3.9 4. Evaluation Total 6.5 1.7 0 8 -1.8 3.7 5. Major ODRs per Student .57 .62 .00 5.27 2.7 11 Variables w/out Coach (n = 325) M SD Min Max Skewness Kurtosis 1. Tier 1 Total 25 5.5 0 30 -2.1 5.5 2. Team Total 3.3 .92 0 4 -1.4 1.7 3. Implementation Total 15 3.4 0 18 -1.9 4.7 4. Evaluation Total 6.5 1.8 0 8 -1.7 2.9 5. Major ODRs per Student .54 .56 .00 4.04 2.3 7.1 24 Table 5 Correlations Among Key Study Variables for sites where the TFI was completed with and without an external coach Variables Tier 1 Team Implementation Major ODRs Total Total Total Evaluation Total per student Tier 1 Total .626** .923** .811** -.040 Team Total .587** .462** .432** -.066 Implementation Total .939** .428** .602** -.056 Evaluation Total .806** .423** .615** .019 Major ODRs per student -.061 -.083* -.072 .018 Note. Correlations with an external coach are below the diagonal and correlations without a coach are above the diagonal. Spearman’s rho was used because assumptions of normal distribution were not met. n = 1008 ** p< 0.01 RQ 2: To what degree does the total TFI Tier 1 total score predict ODRs per student, (when controlling for school level covariates typically associated with ODRs) and to what extent does that relationship depend on whether the TFI was completed with or without an external coach? To address this question, a multiple regression was conducted in SPSS to examine the relationships between the total Tier 1 TFI scores, major ODRs per student, and the use of an external coach to complete the measure. See Table 6 for the regression analysis summary of key variables predicting major ODRs per student. While all three models were statistically significant (p < .001), the change in R² for Models 2 and 3 was not significant (model 1: ΔR2 = .042, p < .001; model 2: ΔR2 <.001, p = .617; model 3: ΔR2 = .001, p = .597). Moreover, the semi-partial correlations for the added variables in Models 2 and 3 are very small in magnitude. Together these results suggest that after school Title I status and proportion of non-White students are accounted for, TFI Tier I total scores 25 and the presence of an external coach do not significantly predict major ODRs per student in SWPBIS schools. RQ 3: To what degree does the Tier 1 teams subscale score predict ODRs per student, (when controlling for school level covariates typically associated with ODRs) and to what extent does that relationship depend on whether the TFI was completed with or without an external coach? To address this question, a multiple regression was conducted in SPSS to examine the relationships between the total Tier 1 team subscale score, ODRs per student, and the use of an external coach to complete the measure. See Table 7 for the regression analysis summary of key variables predicting major ODRs per student. Once again, while all three models were statistically significant (p < .001), the change in R² for Models 2 and 3 was not significant (model 1: ΔR2 = .042, p < .001; model 2: ΔR2 = .002, p = .113; model 3: ΔR2 = .001, p = .511). Moreover, the semi-partial correlations for the added variables in Models 2 and 3 are very small in magnitude. Together these results suggest that after school Title I status and proportion of non-White students are accounted for, TFI Tier I total scores and the presence of an external coach do not significantly predict major ODRs per student in SWPBIS schools. 26 Table 6 Regression Analysis Summary for Key Variables Predicting ODRs per Student Model 1 Model 2 Model 3 B SE β t r VIF B SE B s B β t rs VIF B SE B β t rs VIF I .45 .05 8.3*** .49 .10 4.9*** .46 .16 2.9** Title 1 .29 .05 .2 5.6*** .17 1.0 .29 .05 .17 5.5*** .17 1.0 .29 .05 .17 5.5*** .17 1.0 Non- White -.27 .06 -.14 -4.5*** -.14 1.0 -.27 .06 -.14 -4.5*** -.14 1.0 -.27 .06 -.14 -4.6*** -.14 1.0 Tier 1 Total -.00 .00 -.02 -.50 -.02 1.0 -.00 .00 -.02 -.29 -.01 3.1 Coach .04 .18 .03 .19 .01 21 Coach X TFI Total <.00 .00 .00 .03 .00 23 Note. n = 1008. I = intercept * p < .05. ** p < .01. *** p < .001. 27 Table 7 Regression Analysis Summary for Key Variables predicting ODRs per Student Model 1 Model 2 Model 3 B SE B β t rs VIF B SE B β t rs VIF B SE B β t rs VIF I .45 .05 8.3* .56 .09 6.4* .49 .13 3.7* Title 1 .29 .05 .17 5.5* .17 1.0 .29 .05 .17 5.5* .17 1.0 .30 .05 .17 5.6* .97 1.03 Non- 1.03 White -.27 .06 -.14 -4.5* -.14 1.0 -.26 .06 -.14 -4.5* -.14 1.0 -.27 .06 -.14 -4.5* .97 Tier 1 Team 2.9 Total -.03 .02 -.05 -1.6 -.05 1.0 -.02 .04 -.03 -.64 .35 Coach .10 .15 .08 .67 .07 15 Coach X Tier 1 -.02 .04 .05 -.40 .06 17 Team Note. n = 1008. I = intercept * p < .001. 28 CHAPTER IV DISCUSSION This study was designed to investigate which factors at the Tier 1 level correlate with implementation fidelity overall and better outcomes for students. Based on the analyses conducted major ODRs per student were negatively correlated with the Tier 1 total score, meaning that higher TFI Tier 1 total scores were associated with fewer major ODRs per student. However, that relationship was not significant once school demographics were taken into account. The current findings are only partly consistent with previously conducted studies. Childs, Kincaid, George, & Gage (2015) and Flannery et al. (2014) both found there to be a positive impact between school wide implementation of SWPBIS and student outcomes. Flannery et al. (2014) found the effects of SWPBIS implementation fidelity showed a statistically significant association between SWPBIS fidelity of implementation (using the SET as the measure) and student problem behaviors over time when controlling for Free and Reduced Lunch as a covariate to control for the effects on both ODRs and implementation. Previous studies using other measures of fidelity of implementation have linked improved student outcomes with high fidelity of SWPBIS implementation as well (Flannery et al., 2014; McIntosh et al., 2017;). There was little notable difference between correlations among the TFI variables and ODRs when accounting for the use of an external coach. The only notable result was that the TFI Team and Implementation scores were significantly and negatively correlated with ODRs, and when external coach presence was considered the relationship was only significant between Team scores and ODRs in the presence of an external 29 coach. This trend could be due to the fact that teams were more accurate in their scoring when an external coach was present to facilitate the TFI (Algozzine et al., 2014, p. 3). The external coach may also be able to offer a more objective point of view in their assessment of SWPBIS efforts overall. SWPBIS is often more effective when Tier 1 supports are implemented with high fidelity (Kim et al., 2014). Schools that have low fidelity in Tier 1 systems and supports are less likely to have high fidelity of Tiers 2 and 3. Schools may want to focus on Tier 1 implementation before implementing other tiers. Future research could focus on which items within tier 1 are predictive of high TFI scores at Tier 2 and 3. Identification of subscales associated with lower schoolwide total major ODRs per student could inform a focus for coaching and training of SWPBIS teams. It would not be possible to randomly assign people to SWPBIS, but future research could focus on manipulating whether or not schools utilize the TFI and how being assigned to use of the TFI or not could predict ODRs. Research questions two and three focused on the relationship between the Tier 1 TFI Total scores, Team subscale scores, and ODRs. The team subscale focuses on items like team composition and operation procedures. Implementation teams are responsible for ensuring consistently improved student outcomes (Fixsen et al., 2003). While results for both research questions were nonsignificant, it was notable that the team subscale added slightly but non- significantly to explaining major ODRs, whereas the total score did not. Results from this study suggest that future research might work to identify which team items and characteristics are associated with higher levels of implementation, as well as lower levels of major ODRs per student. Many items on the TFI are non-unitary, asking if several items are in place. For example, item 1 in Tier 1 of the TFI asks about team composition and whether or not there are 30 eight specific members of the team who attend at least 80% of meetings. The TFI format was created with a focus on timeliness. In my personal practice when I complete the TFI with teams I have a checklist that breaks down each question into individual criterion. In my practice, I have found it easier for teams to answer items with one criterion, rather than several. This practice also helps me to create more action-oriented improvement plans with measurable goals and outcomes as well as informs future training and coaching for SWPBIS teams. Limitations Validity Threats to validity (internal and external) and reliability will now be discussed. Internal Validity Internal validity “refers to the validity of the cause and effect inference linking the independent variable and the dependent variable” (Wiersma & Jurs, 2005, p. 104). The first major threat to internal validity was instrumentations. The TFI is a measure that is used repeatedly and is linked to funding in some districts, which may encourage teams to inflate their scores. The second threat to internal validity was instrumentation in the use of ODRs as a data source because of individual variations in their use among school staff. Often, school staff do not have common agreement on specific indicators for documenting ODRs leaving the documentation up to individual interpretation. For example, staff members may assign the indicator of disrespect as a minor or major for the same student in the same situation leaving the data difficult to interpret. The third threat to internal validity was selection. Participants were chosen based on their completion of TFI data and use of the SWIS data system. One can hypothesize that since these teams consistently conducted these measures of fidelity and implementation, they might have 31 been more concerned with the ongoing monitoring and evaluation of their implementation fidelity. Creswell (2014) cautions researchers in drawing conclusions about specific populations. A third threat was ambiguity about the direction of causal influence. Many things were occurring at once within the school environment (i.e. training, coaching, and funding) making it impossible to discern what was causing what. In other words, it was as likely that fewer ODRs were behind better TFI scores as the opposite. A fourth threat to internal validity was mono-method bias, a construct or phenomenon being measured in only one way. I only analyzed extant data from a self- report measure (the TFI). External Validity External validity is defined as the extent to which the findings from a study are generalizable to other populations (Wiersma & Jurs, 2005). Wiersman and Jurs (2005) also defined it as including the generalization of the findings to other independent variables that may be related. In using a nonrandom, nonprobability convenience sampling method from an extant data set of schools who utilize the TFI to measures SWPBIS implementation and SWIS to measure major ODRs per student generalizability was limited to the population of the study (those who implemented SWPBIS, used the TFI to measure implementation, and used SWIS to measure ODRs). Practical Implications SWPBIS This section will focus on the practical implications for the implementation and use of SWPBIS. The analyses conducted showed lower numbers of ODRs per student were associated with higher Tier 1 total scores, although only when 32 school demographics were not controlled. Thus, although higher Tier 1 overall TFI scores were associated with fewer major ODRs per student, this relationship was not significant once school proportion of minority students and Title I status were controlled. Given the simple correlation results, schools may want to consider focusing their efforts on Tier 1 implementation before expanding Tier 2 and Tier 3 supports in the hopes of lowering overall major ODRs per student. As major ODRs per students decrease, theoretically fewer intensive supports at the Tier 2 and Tier 3 level should be needed in the future which could ultimately save schools and districts a significant amount of resources (i.e. time, money, staffing). Practitioner The findings from this study, as well as previously conducted studies, have implications for practitioners. As discussed previously, multiple researchers suggested using an external coach to strengthen the reliability of TFI scores (Kittleman et al., 2018; Massar et al., 2019; McIntosh et al., 2017; Mercer et al., 2017). Not all districts and schools have resources and/or access to an external coach, which may have important implications for the initial adoption of SWPBIS as well as the continued and sustained implementation efforts. Adequate resources are shown to develop the capacity of staff (Fixsen et al., 2013). However, schools within my district receive varying levels of resources dependent on coach time and availability. For example, three of the six schools I consult for (as a part-time employee) conduct their Individual Positive Behavior Interventions and Supports (iPBIS) meetings concurrently. I then have to choose which meetings to attend based on the most immediate need. In theory this should work, however it leaves schools 33 who have built implementation momentum or schools with the greatest need without the support of a coach. This can cause teams to lose momentum in their implementation efforts because they do not get the support they need when they need it. It should also be noted that as a behavior consultant and SWPBIS/IPBIS coach my job does not solely focus on coaching. In recent years our role has included more time devoted to professional development consisting of training without follow-up coaching and support based on the individual needs of teams (Fixsen et al., 2013). This implementation dilemma is common in education across the united states due to adequate resource allocation the challenge. The challenge schools face is not finding what works, but implementing what works (Fixsen, Naoom, Blase, Friedman, & Wallace, 2005). Although the current findings are equivocal, based on other existing research (Flannery et al., 2014; Mercer et al., 2017),school teams may want to focus PBIS efforts and resources on initial adoption or a reboot of Tier 1implementation and practices in an effort to reduce overall discipline referrals and the need for possible resources that accompany discipline (i.e., staffing). Generally, Tier 2 and Tier 3 supports require a larger investment in resources, which may become even more important in the coming 2020-2021 school year as school districts may be faced with lower budgets, decreased resources, and increased behaviors as students time out of brick and mortar schools increases. Additionally, teams who did not use an external coach did score higher on Tier 1 of the TFI when compared to teams who did not use an external coach. I hypothesize that this is due at least in part to the fact that coaches often offer points of clarification around multi-part questions on the TFI. Coaches may also help teams be more realistic in their assessments of their implementation efforts as well as student outcomes. 34 Future Research The research design was non-experimental. As such it was not possible to draw causal conclusions about the relations between fidelity of SWPBIS implementation and ODRs. The measures used were from an extant database, meaning that the precise composition of the teams completing the measures and the degree to which fidelity to its administration was adhered are both unknown. Future research could use a sample in which these variables are known and then be included in the analyses. Direct observation data was not used to compare reported level of implementation to actual level of implementation, but while an additional source of information regarding implementation would be valuable, it would also limit sample size to keep data collection manageable. Another avenue for future research efforts is the TFI instrument itself. Many, if not all, of the items on the TFI are not unitary, asking if more than one thing is in place. For example, item 1 in Tier 1 of the TFI asks about team composition and whether or not there are eight specific members of the team which can make it hard at times for teams to answer this question. However, the scoring criteria does give teams some direction. For example, on item 1 in Tier 1 a 0 indicates that a Tier 1 team does not exist and/or does not include three of the key members, a 1 indicates that a team does exist but does not include all of the identified roles or attendance of these members is below 80%, and a 2 indicates that a Tier 1 team exists with all identified roles and attendance is at or above 80%. It may be easier for teams to answer items if the items only measured one criterion, rather than several. The measure may also offer stronger validity if items only measured one aspect of the team. 35 In conclusion, the purpose of this study was to explore the relationship among TFI subscale scores for Tier 1, annual major ODRs per student, and the use of an external coach. The data suggests the team score and implementation score are significantly related to annual major ODRs per student. However, when the presence of an external coach is controlled the only significant relationship between the TFI and ODRs is the team score. The data suggests the team score will be more predictive when an external coach is present to provide the team coaching in context. Future research could focus on direct observation of team behaviors to compare observed level of implementation to self-reported level of implementation. Another future area of research could focus on disaggregating the data by school level (elementary, middle, and high) as well as specific times of year to determine if these factors result in varied levels of implementation and statistical significance. 36 APPENDIX A LITERATURE SEARCH AND REVIEW Topic I am interested in how implementation fidelity effects outcomes of rese/arch- based practices and how implementation fidelity of SWPBIS effects exclusionary discipline practices. Fidelity to implementation, or treatment integrity of interventions, can be defined as interventions used according to identified criteria outlining activities, materials, and behaviors that result in improved outcomes (Smith, Daunic, & Taylor, 2007). When an intervention is not implemented as planned, students’ performance can decline or remain unchanged. For example, Flannery, Fenning, Kato, and McIntosh (2014) found that when School Wide Positive Behavior Intervention and Supports (SWPBIS) implementation increased, exclusionary discipline practices decreased. I am interested in what factors correlate with implementation fidelity and sustainability of SWPBIS as well as improved outcomes for students, teachers, and the school environment. In this case, I used the Tiered Fidelity Inventory (TFI), which measures the fidelity of implementation of SWPBIS at all three Tiers. Search Procedures To ensure the most relevant references for my study I followed a multi-step process. To begin my literature search I took advantage of the University of Oregon Online Library digital database using several keywords and phrases. Next, I applied exclusionary criteria to narrow the article pool. Then, I used an ancestral search focused on references from the Technical Adequacy of the SWPBIS Tiered Fidelity Inventory (McIntosh, Massar, Algozzine, George, Horner, Lewis & Swain-Bradway, 2017). 37 Finally, I spoke with two experts in PBIS, implementation science, and equity who assisted me in the identification of additional articles. First, I used the following keywords and phrases: (a) PBIS, (b) positive behavior/al interventions and supports, (c) implementation, (d) Tiered Fidelity Inventory, and (e) TFI. I limited the search to English articles only. My search resulted in 21 possible references. Second, I limited my search to include peer-reviewed articles. My search resulted in 20 possible references. Third, since the TFI is a relatively new measure, I limited my search to include articles from the past 5 years narrowing my article pool to 16 articles Selection Criteria and Final Literature Pool The University of Oregon Library search produced 16 articles that I reviewed. I read the title and abstract of each article to eliminate articles that did not meet the following selection criteria: a focus on (a) SWPBIS implementation or (b) SWPBIS TFI and (d) K-12 settings. This resulted in an article pool of five articles, two articles focused on different forms of reliability and validity of the TFI when compared to previous SWPBIS fidelity/implementation measures. One focused on implementing educational practices; and the other two articles focused on sustained use and implementation of SWPBIS. All five articles were selected for the article pool. To ensure use of the most relevant and current references for my study I contacted a researcher in the field of PBIS, seven articles were suggested that focused on validity of fidelity measures, three from my original selection pool, the TFI Measure, and one unpublished article. Three additional articles from the field of implementation science were included after conferring with an Implementation specialist at the National 38 Implementation Research Network. These articles focused on a conceptual framework from implementation science in three disciplines, education, child welfare, and public service sectors. Increasing my article pool to twelve articles. To include literature on student outcomes in my article pool I conducted an ancestral search from the Technical Adequacy of the SWPBIS Tiered Fidelity Inventory (McIntosh, Massar, Algozzine, George, Horner, Lewis & Swain-Bradway, 2017). I read through the titles of the 43 articles applying the above selection criteria. In doing so I identified three articles that focused on effects of SW-PBIS on improved student behavioral outcomes, one of which I already had. This increased my article pool to fourteen articles. My advisor then suggested I also include two articles on the topic of discipline disproportionality. I contacted an expert in the field of equity; two more articles were suggested to me that focused on discipline disproportionality bringing me to a total of 18 articles. 39 APPENDIX B TIERED FIDELITY INVENTORY: TIER 1 Tier 1: Universal SWPBIS Features NOTE: This section may be completed individually or with other tiers as part of the full Tiered Fidelity Inventory Feature Possible Data Sources Scoring Criteria Subscale: Teams 1.1 Team Composition: • School organizational chart 0 = Tier 1 team does not exist or Tier 1 team includes a Tier does not include coordinator, • Tier 1 team meeting minutes 1 systems coordinator, a school administrator, or individuals school administrator, a family with applied behavioral expertise member, and individuals able to provide (a) applied behavioral 1 = Tier 1 team exists, but expertise, (b) coaching does not include all identified expertise, (c) knowledge of roles or attendance of these student academic and behavior members is below 80% patterns, (d) knowledge about the operations of the school 2 = Tier 1 team exists with across grade levels and coordinator, administrator, programs, and for high schools, and all identified roles (e) student representation. represented, AND attendance of all roles is at or above 80% 1.2 Team Operating Procedures: • Tier 1 team meeting 0 = Tier 1 team does not use Tier 1 team meets at least agendas and minutes regular meeting format/ monthly and has (a) regular agenda, minutes, defined roles, • Tier 1 meeting roles descriptions meeting format/agenda, or a current action plan (b) minutes, (c) defined • Tier 1 action plan meeting roles, and (d) a 1= Tier 1 team has at least current action plan. 2 but not all 4 features 2 = Tier 1 team meets at least monthly and uses regular meeting format/agenda, minutes, defined roles, AND has a current action plan Scoring Criteria: 0=Not implemented; 1=Partially implemented; 2=Fully implemented SWPBIS Tiered Fidelity Inventory 6 40 Feature Possible Data Sources Scoring Criteria Subscale: Implementation 1.3 Behavioral Expectations: • TFI Walkthrough Tool 0 = Behavioral expectations School has five or fewer have not been identified, • Staff handbook positively stated behavioral are not all positive, or are expectations and examples • Student handbook more than 5 in number by setting/location for student and staff behaviors 1 = Behavioral expectations (i.e., school teaching matrix) identified but may not include defined and in place. a matrix or be posted 2 = Five or fewer behavioral expectations exist that are positive, posted, and identified for specific settings (i.e., matrix) AND at least 90% of staff can list at least 67% of the expectations 1.4 Teaching Expectations: • TFI Walkthrough Tool 0 = Expected behaviors Expected academic and are not taught • Professional development social behaviors are taught calendar directly to all students in 1 = Expected behaviors are taught classrooms and across other • Lesson plans informally or inconsistently campus settings/locations. • Informal walkthroughs 2 = Formal system with written schedules is used to teach expected behaviors directly to students across classroom and campus settings AND at least 70% of students can list at least 67% of the expectations 1.5 Problem Behavior Definitions: • Staff handbook 0 = No clear definitions exist, and School has clear definitions procedures to manage problems • Student handbook for behaviors that interfere are not clearly documented with academic and social • School policy success and a clear policy/ 1 = Definitions and procedures procedure (e.g., flowchart) for • Discipline flowchart exist but are not clear and/or addressing office-managed not organized by staff- versus versus staff-managed problems. office-managed problems 2 = Definitions and procedures for managing problems are clearly defined, documented, trained, and shared with families Scoring Criteria: 0=Not implemented; 1=Partially implemented; 2=Fully implemented SWPBIS Tiered Fidelity Inventory 7 41 Feature Possible Data Sources Scoring Criteria 1.6 Discipline Policies: • Discipline policy 0 = Documents contain only School policies and procedures reactive and punitive consequences • Student handbook describe and emphasize proactive, instructive, and/ • Code of conduct 1 = Documentation includes and or restorative approaches emphasizes proactive approaches to student behavior that are • Informal administrator interview implemented consistently. 2 = Documentation includes and emphasizes proactive approaches AND administrator reports consistent use 1.7 Professional Development: • Professional development 0 = No process for teaching A written process is used for calendar staff is in place orienting all faculty/staff on 4 • Staff handbook core Tier 1 SWPBIS practices: 1 = Process is informal/unwritten, (a) teaching school-wide not part of professional development expectations, (b) acknowledging calendar, and/or does not include appropriate behavior, (c) all staff or all 4 core Tier 1 practices correcting errors, and (d) requesting assistance. 2 = Formal process for teaching all staff all aspects of Tier 1 system, including all 4 core Tier 1 practices 1.8 Classroom Procedures: • Staff handbook 0 = Classrooms are not Tier 1 features (school- implementing Tier 1 • Informal walkthroughs wide expectations, routines, acknowledgements, in-class • Progress monitoring 1 = Classrooms are informally continuum of consequences) implementing Tier 1 but no are implemented within • Individual classroom data formal system exists classrooms and consistent with school-wide systems. 2 = Classrooms are formally implementing all core Tier 1 features, consistent with school-wide expectations Scoring Criteria: 0=Not implemented; 1=Partially implemented; 2=Fully implemented SWPBIS Tiered Fidelity Inventory 8 42 Feature Possible Data Sources Scoring Criteria 1.9 Feedback and • TFI Walkthrough Tool 0 = No formal system for Acknowledgement: acknowledging students • Staff handbook A formal system (i.e., written set of procedures for specific 1 = Formal system is in place and behavior feedback that is is used by at least 90% of staff OR [a] linked to school-wide received by at least 50% of students expectations and [b] used across settings and within 2 = Formal system for classrooms) is in place and used acknowledging student behavior by at least 90% of a sample of is used by at least 90% of staff AND staff and received by at least received by at least 50% of students 50% of a sample of students. 1.10 Faculty Involvement: • PBIS Self-Assessment Survey 0 = Faculty are not shown data at Faculty are shown school- least yearly and do not provide input • Informal surveys wide data regularly and provide input on universal • Staff meeting minutes 1 = Faculty have been shown foundations (e.g., expectations, data more than yearly OR have acknowledgements, • Team meeting minutes provided feedback on Tier 1 definitions, consequences) foundations within the past at least every 12 months. 12 months but not both 2 = Faculty are shown data at least 4 times per year AND have provided feedback on Tier 1 practices within the past 12 months 1.11 Student/Family/Community • Surveys 0 = No documentation (or no Involvement: opportunities) for stakeholder • Voting results from parent/ Stakeholders (students, feedback on Tier 1 foundations family meeting families, and community members) provide input on • Team meeting minutes 1 = Documentation of input on Tier universal foundations (e.g., 1 foundations, but not within the expectations, consequences, past 12 months or input but not acknowledgements) at from all types of stakeholders least every 12 months. 2 = Documentation exists that students, families, and community members have provided feedback on Tier 1 practices within the past 12 months Scoring Criteria: 0=Not implemented; 1=Partially implemented; 2=Fully implemented SWPBIS Tiered Fidelity Inventory 9 43 Feature Possible Data Sources Scoring Criteria Subscale: Evaluation 1.12 Discipline Data: • School policy 0 = No centralized data system with Tier 1 team has instantaneous ongoing decision making exists • Team meeting minutes access to graphed reports summarizing discipline data • Student outcome data 1 = Data system exists but does organized by the frequency of not allow instantaneous access problem behavior events by to full set of graphed reports behavior, location, time of day, and by individual student. 2 = Discipline data system exists that allows instantaneous access to graphs of frequency of problem behavior events by behavior, location, time of day, and student 1.13 Data-based Decision Making: • Data decision rules 0 = No process/protocol exists, or Tier 1 team reviews and uses data are reviewed but not used • Staff professional discipline data at least monthly development calendar for decision-making. 1 = Data reviewed and used • Staff handbook for decision-making, but less than monthly • Team meeting minutes 2 = Team reviews discipline data and uses data for decision-making at least monthly. If data indicate a problem, an action plan is developed to enhance or modify Tier 1 supports 1.14 Fidelity Data: • School policy 0 = No Tier 1 SWPBIS Tier 1 team reviews and uses fidelity data collected • Staff handbook SWPBIS fidelity (e.g., SET, BoQ, TIC, SAS, Tiered Fidelity • School newsletters 1 = Tier 1 fidelity collected informally Inventory) data at least annually. and/or less often than annually • School website 2 = Tier 1 fidelity data collected and used for decision making annually Scoring Criteria: 0=Not implemented; 1=Partially implemented; 2=Fully implemented SWPBIS Tiered Fidelity Inventory 10 44 Feature Possible Data Sources Scoring Criteria 1.15 Annual Evaluation: • Staff, student, and family surveys 0 = No evaluation takes place, or Tier 1 team documents evaluation occurs without data • Tier 1 handbook fidelity and effectiveness of Tier 1 practices at least • Fidelity tools 1 = Evaluation conducted, but not annually (including year- annually, or outcomes are not used by-year comparisons) that • School policy to shape the Tier 1 process and/ are shared with stakeholders • Student outcomes or not shared with stakeholders (staff, families, community, district) in a usable format. • District reports 2 = Evaluation conducted at least annually, and outcomes shared with • School newsletters stakeholders, with clear alterations in process based on evaluation Scoring Criteria: 0=Not implemented; 1=Partially implemented; 2=Fully implemented SWPBIS Tiered Fidelity Inventory 11 45 REFERENCES CITED Aarons, G. A., Hurlburt, M., & Horwitz, S. M. (2011). Advancing a conceptual model of evidence-based practice implementation in public service sectors. Administration and Policy in Mental Health and Mental Health Services Research, 38, 4-23. Algozzine, B., Barrett, S., Eber, L., George, H., Horner, R., Lewis, T., Putnam, B., Swain-Bradway, J., McIntosh, K., & Sugai, G (2014). School-wide PBIS Tiered Fidelity Inventory. OSEP Technical Assistance Center on Positive Behavioral Interventions and Supports. Retrieved from www.pbis.org. Bradshaw, C. P., Mitchell, M. M., & Leaf, P. J. (2010). Examining the effects of schoolwide positive behavioral interventions and supports on student outcomes: Results from a randomized controlled effectiveness trial in elementary schools. Journal of Positive Behavior Interventions, 12, 133-148. Carroll, C., Patterson, M., Wood, S., Booth, A., Rick, J., & Balain, S. (2007). A conceptual framework for implementation fidelity. Implementation Science, 2, 40- 49. Childs, K. E., Kincaid, D., George, H. P., & Gage, N. A. (2015). The relationship between school-wide implementation of positive behavior intervention and supports and student discipline outcomes. Journal of Positive Behavior Interventions, 18, 89-99. doi:10.1177/1098300715590398 Fixsen, D. L., & Blase, K. A. (2008). Drivers framework. Chapel Hill, NC: University of North Carolina, The National Implementation Research Network, Frank Porter Graham Child Development Institute. Fixsen, D. L., Blase, K., Metz, A., & Van Dyke, M. (2013). Statewide implementation of evidence-based programs. Exceptional Children, 79, 213-230. Flannery, K. B., Fenning, P., Kato, M. M., & McIntosh, K. (2014). Effects of school- wide positive behavioral interventions and supports and fidelity of implementation on problem behavior in high schools. School Psychology Quarterly, 29, 111. Gage, N. A., Whitford, D. K., & Katsiyannis, A. (2018). A review of schoolwide positive behavior interventions and supports as a framework for reducing disciplinary exclusions. The Journal of Special Education, 52, 142-151. González, Thalia, Keeping Kids in Schools: Restorative Justice, Punitive Discipline, and the School to Prison Pipeline (April 2012). Journal of Law & Education, Vol 41, No. 2. 46 Horner, R., Sugai, G., & Fixsen, D. (2017). Implementing effective educational practices at scales of social importance. Clinical Child and Family Psychology Review, 20, 25-35. Kaderavek, J. N., & Justice L. M. (2010). Fidelity: An essential component of evidence- based practice in speech language pathology. American Journal of Speech- Language Pathology, 19, 369-370. Kincaid, D., Childs, K., & George, H. (2010). School-wide Benchmarks of Quality: Scoring Form (Revised). Unpublished Instrument, USF, Tampa, FL. King-Sears, M. E., Walker, J. D., & Barry, C. (2018). Measuring teachers’ intervention fidelity. Intervention in School and Clinic, 54, 89-96. Kittelman, A., Eliason, B. M., Dickey, C. R., & McIntosh, K. (2018). How are schools using the SWPBIS tiered fidelity inventory (TFI)? Retrieved from www.pbis.org. Massar, M. M., McIntosh, K., & Mercer, S. H. (2019). Factor validation of a fidelity of implementation measure for social behavior systems. Remedial and Special Education, 40, 16-24. McIntosh, K., Filter, K. J., Bennett, J. L., Ryan, C., & Sugai, G. (2010). Principles of sustainable prevention: Designing scale‐up of school‐wide positive behavior support to promote durable systems. Psychology in the Schools, 47, 5-21. McIntosh, K., Massar, M. M., Algozzine, R. F., George, P. G., Horner, R. H., Lewis, T. J., & Swain-Bradway, J. (2017). Technical adequacy of the SWPBIS tiered fidelity inventory. Journal of Positive Behavior Interventions, 19, 3-13. doi:10.1177/1098300716637193 McIntosh, K., Mercer, S. H., Nese, R. N., Strickland-Cohen, M. K., & Hoselton, R. (2016). Predictors of sustained implementation of school-wide positive behavioral interventions and supports. Journal of Positive Behavior Interventions, 18, 209- 218. Mcintosh, K., Mercer, S., Nese, R., Strickland-Cohen, M., Kittelman, A., Hoselton, R., & Horner, R. (2018). factors predicting sustained implementation of a universal behavior support framework. Educational Researcher, 47, 307-316. Mercer, S. H., McIntosh, K., & Hoselton, R. (2017). Comparability of fidelity measures for assessing tier I school- wide positive behavioral interventions and supports. Journal of Positive Behavior Interventions, 19, 195-204. doi:10.1177/1098300717693384 47 Metz, A., Bartley, L., Ball, H., Wilson, D., Naoom, S., & Redmond, P. (2015). Active implementation frameworks for successful service delivery: Catawba county child wellbeing project. Research on Social Work Practice, 25, 415-422. Moullin, J.C., Dickson, K.S., Stadnick, Rabin, B., and Aarons, G.A. Systematic review of the Exploration, Preparation, Implementation, Sustainment (EPIS) framework. Implementation Sci 14, 1 (2019). https://doi.org/10.1186/s13012-018- 0842-6 Nocera, E. J., Whitbread, K. M., & Nocera, G. P. (2014). Impact of school-wide positive behavior supports on student behavior in the middle grades. RMLE Online, 37, 1- 14. Smith, S. W., Daunic, A. P., & Taylor, G. G. (2007). Treatment fidelity in applied educational research: Expanding the adoption and application of measures to ensure evidence-based practice. Education and treatment of children, 30, 121- 134. Sugai, G., & Simonsen, B. (2012). Positive behavioral interventions and supports: History, defining features, and misconceptions. Center for PBIS & Center for Positive Behavioral Interventions and Supports, University of Connecticut. Retrieved from https://www.pbis.org/common/cms/files/pbisresources/PBIS_revisited_June19r_2 012.pdf Vincent, C. G., Sprague, J. R., & Tobin, T. J. (2012). Exclusionary discipline practices across students' racial/ethnic backgrounds and disability status: Findings from the Pacific Northwest. Education and treatment of children, 35, 585-601. Yeung, A., Craven, R., Mooney, M., Tracey, D., Barker, K., Power, A., Lewis, T. (2016). Positive behavior interventions: The issue of sustainability of positive effects. Educational Psychology Review, 28, 145-170. 48