EXPLORING SUSTAINABILITY, FIDELITY, AND INDICATORS OF DATA USE BY SCHOOLS IMPLEMENTING POSITIVE BEHAVIORAL INTERVENTIONS AND SUPPORTS by KATHLEEN MARGARET CONLEY A DISSERTATION Presented to the Department of Special Education and Clinical Services and the Division of Graduate Studies of the University of Oregon in partial fulfillment of the requirements for the degree of Doctor of Philosophy December 2021 DISSERTATION APPROVAL PAGE Student: Kathleen Margaret Conley Title: Exploring Sustainability, Fidelity, and Indicators of Data Use by Schools Implementing Positive Behavioral Interventions and Supports This dissertation has been accepted and approved in partial fulfillment of the requirements for the Doctor of Philosophy degree in the Department of Special Education and Clinical Sciences by: Robert Horner Advisor/Chairperson Kent McIntosh Co-Chairperson Beth Harn Core Member Kathleen Strickland-Cohen Core Member Ellen McWhirter Institutional Representative and Krista Chronister Vice Provost for Graduate Studies Original approval signatures are on file with the University of Oregon Division of Graduate Studies. Degree awarded December 2021 ii © 2021 Kathleen Margaret Conley iii DISSERTATION ABSTRACT Kathleen Margaret Conley Doctor of Philosophy Department of Special Education and Clinical Services December 2021 Title: Exploring Sustainability, Fidelity, and Indicators of Data Use by Schools Implementing Positive Behavioral Interventions and Supports Analysis of the variables that affect use of evidence-based and promising innovations in education has led to an emerging literature base addressing sustainability or the potential for sustained implementation over time. Examining the experiences of schools actively engaged in implementing evidence-based innovations holds promise for identifying factors that predict effective data use, high procedural fidelity, and sustained implementation needed to produce substantive school improvement. This study examined three constructs (sustainability, fidelity, and data use) related to implementation of Positive Behavioral Interventions and Supports (PBIS). PBIS is an evidence-based framework that specifically addresses school climate and student social, emotional, and behavioral needs. The purpose of this exploratory study was to examine sustainability of PBIS over time through relations with several simple measures of (a) data use by PBIS school teams, (b) PBIS fidelity, and (c) factors linked to and across tiers of support within the framework. Extant data were collected from a sample of 656 U.S. schools implementing the PBIS framework over three consecutive years. Results of this study indicate statistically significant relations between fidelity and factors of sustainability across PBIS tiers but indicate that more sophisticated measures of school team data use across tiers are iv needed. Implications of these findings, limitations, and suggestions for practitioners and future research are discussed. v CURRICULUM VITAE NAME OF AUTHOR: Kathleen Margaret Conley GRADUATE AND UNDERGRADUATE SCHOOLS ATTENDED: University of Oregon, Eugene, Oregon Northwest Christian College, Eugene, Oregon Lane Community College, Eugene, Oregon DEGREES AWARDED: Doctor of Philosophy, Special Education, 2021, University of Oregon Master of Science, Special Education, 2007, University of Oregon Bachelor of Science, Elementary Education, 2006, Northwest Christian College Associate of Arts, 2004, Lane Community College AREAS OF SPECIAL INTEREST: School-Wide Positive Behavioral Interventions and Supports Implementation Science Educator Professional Development PROFESSIONAL EXPERIENCE: Sr. Research Assistant, University of Oregon, Eugene, Oregon 2009-Present Adjunct Instructor, Bushnell University, Eugene, Oregon, 2018-Present Self-Contained Teacher for Students with Emotional and Behavioral Disorders, Spectrum Center, Pittsburg, California, 2007-2008 GRANTS, AWARDS, AND HONORS: Janette Gunther Drew Scholarship, University of Oregon, 2020-2021 Graham/Austin Memorial, University of Oregon, 2020-2021 Personnel Preparation Grant, Project ENLIST, University of Oregon, 2014-2020 President’s Award, Northwest Christian College, 2006 vi PUBLICATIONS: Conley, K. M., Everett, S. R., & Pinkelman, S. E. (2019). Strengthening progress monitoring procedures for individual student behavior support. Beyond Behavior, 28, 124-133. https://doi.org/10.1177/1074295619852333 Conley, K. M., Horner, R. H., & McIntosh, K. (2019). Use of I-SWIS by elementary schools to monitor Tier 3 behavior supports [Evaluation Brief ]. OSEP Technical Assistance Center on Positive Behavioral Interventions and Supports. www.pbis.org. Conley, K. M., Kittelman, A., Massar, M., & McIntosh, K. (2018). What are patterns and predictors of CICO participation in U.S. Schools? [Evaluation Brief]. OSEP Technical Assistance Center on Positive Behavioral Interventions and Supports. www.pbis.org. vii ACKNOWLEDGMENTS I want to thank the many individuals who directly and indirectly supported me through the dissertation process and the preparation of this manuscript. Rob, thank you for the wisdom and advice that you have shared as my advisor, and for sticking with me through health issues and world-wide pandemics. Kent, thank you for inviting me to implementation science and the ENLIST group, for sharing your research ideas and data, and for all the tips and tricks to understand the sometimes-confusing world of academia. Thank you, Beth and Kathleen, for the moral and emotional support you each provided, often at levels of advanced tiers. Ellen, I appreciate your kindness and flexibility as I muddled my way to a dissertation topic, thank you. I want to thank my wonderful family for their practical and emotional support. Everything from hugs and meals to listening to me ramble about my research, all of it and all of you mean the world to me. To my work family at ECS, especially my fellow PBISApps training team members, thank you for being flexible and kind and gracious as I balanced (and rebalanced) my work and school and life every few months. Finally, I want to thank the many researchers in the PBIS and Special Education communities who have shared knowledge with the world and provided the foundation on which this study and manuscript are built. viii This dissertation is dedicated to Jesus, my all in all. ix TABLE OF CONTENTS Chapter Page I. INTRODUCTION .................................................................................................... 1 Statement of Purpose ............................................................................................. 1 Literature Review................................................................................................... 4 Improvement Science....................................................................................... 5 Implementation Science ................................................................................... 7 Implementation Drivers ............................................................................. 8 Stages of Implementation .......................................................................... 10 Positive Behavioral Interventions and Supports (PBIS) .................................. 14 The PBIS Approach ................................................................................... 16 Tiers of Support in PBIS ............................................................................ 18 Tier 1 .................................................................................................. 20 Advanced Tiers ................................................................................... 24 Fidelity, Sustainability, and Data Use in PBIS ................................................ 31 PBIS Fidelity .............................................................................................. 31 PBIS Sustainability .................................................................................... 33 Tier 1 PBIS Sustainability ................................................................... 35 Advanced Tier PBIS Sustainability .................................................... 37 PBIS Data Use ........................................................................................... 40 Discipline Referral Data ..................................................................... 41 Student Intervention Data ................................................................... 47 Theory of Data Use .................................................................................... 51 x Chapter Page Tiered Data System ............................................................................. 53 Tiered Decision System ...................................................................... 53 Summary .......................................................................................................... 55 Study Purpose and Research Questions ................................................................. 56 II. METHODS.............................................................................................................. 58 Settings and Participants ........................................................................................ 58 Measures ................................................................................................................ 60 PBIS Fidelity .................................................................................................... 61 School-Wide Evaluation Tool (SET) ......................................................... 62 Tiered Fidelity Inventory (TFI) ................................................................. 64 TFI-Tier 1 ............................................................................................ 64 TFI-Tier 2 ............................................................................................ 65 TFI-Tier 3 ............................................................................................ 65 Benchmarks of Quality (BoQ) ......................................................................... 65 PBIS Fidelity Across Tiers ........................................................................ 66 PBIS Sustainability .......................................................................................... 67 Tier 1 PBIS Sustainability (SUBSIST) ...................................................... 67 Advanced Tier PBIS Sustainability (ALTITUDE) .................................... 69 PBIS Data Use ................................................................................................. 71 School-Wide Information System (SWIS) ................................................ 71 Check-In Check-Out School-Wide Information System (CICO-SWIS) ... 76 Procedures .............................................................................................................. 79 xi Chapter Page Original School Demographic, PBIS Sustainability, and PBIS Fidelity Variables .......................................................................................................... 79 Additional PBIS Fidelity and PBIS Data Use Variables ................................. 80 Data Cleaning and Preparation ........................................................................ 80 Data Analyses ........................................................................................................ 81 Research Question 1 ........................................................................................ 81 RQ1. Spearman’s Rho Correlations .................................................... 81 RQ1. Multiple Linear Regression ....................................................... 82 Research Question 2 ........................................................................................ 82 RQ2. Spearman’s Rho Correlations .................................................... 82 RQ2. Multiple Linear Regression ....................................................... 83 Research Question 3 ........................................................................................ 83 RQ3. Spearman’s Rho Correlations .................................................... 83 RQ3. Partial Correlations .................................................................... 84 Research Question 4 ........................................................................................ 85 RQ4. Kendal’s Tau-b Correlations ..................................................... 85 Research Question 5 ........................................................................................ 86 RQ5. Kendal’s Tau-b Correlations ..................................................... 86 III. RESULTS .............................................................................................................. 88 Research Question 1. To what extent is PBIS Tier 1 fidelity related to factors predicting sustainability of PBIS at Tier 1 (as measured by the SUBSIST)? ........ 88 Research Question 2. To what extent is PBIS Tier 1, Tier 2, and Tier 3 fidelity related to factors predicting sustainability of PBIS at advanced tiers (as measured by the ALTITUDE)? .............................................................................................. 90 xii Chapter Page Research Question 3. To what extent are factors of sustainability of PBIS at Tier 1 (as measured by the SUBSIST) related to factors of sustainability of PBIS at advanced tiers (as measured by the ALTITUDE)? ............................................ 92 Research Question 4. To what extent is access of progress monitoring data about Tier 1 student behavior patterns related to (a) fidelity of PBIS at Tier 1 and (b) sustainability of PBIS at Tier 1 (as measured by the SUBSIST)? ............ 94 Research Question 5. To what extent is access of progress monitoring data about student behavior related to (a) fidelity of PBIS at Tiers 2 and 3 and (b) sustainability of PBIS at advanced tiers (as measured by the ALTITUDE)? ........ 96 IV. DISCUSSION ........................................................................................................ 109 General Discussion ................................................................................................ 109 PBIS Fidelity and Sustainability ...................................................................... 110 PBIS Sustainability .......................................................................................... 112 PBIS Data Use, Fidelity, and Sustainability .................................................... 113 Limitations ............................................................................................................. 114 Measurement Limitations ................................................................................ 114 Sample Size Limitations .................................................................................. 117 Implications for Future Research ........................................................................... 118 Data Use and Implementation Science ............................................................ 118 School and District Data Use Across Stages of PBIS Implementation ........... 119 Implications for Practice ........................................................................................ 123 School Administrators and PBIS Leadership Teams ....................................... 123 Use of Implementation Focused Measures ................................................ 123 Comprehensive Decision Guidelines for Data Use and Sharing ............... 125 District Leadership Teams ............................................................................... 126 xiii Chapter Page Conclusion ............................................................................................................. 128 APPENDICES ............................................................................................................. 130 A. SWIS READINESS CHECKLIST ................................................................... 130 B. SWIS REFERRAL COMPATIBILITY CHECKLIST ..................................... 132 C. CICO-SWIS READINESS CHECKLIST ........................................................ 133 D. SET: SCHOOL-WIDE EVALUATION TOOL SCORING GUIDE ............... 135 E. TFI: SWPBIS TIERED FIDELITY INVENTORY (TFI) VERSION 2.1 ........ 138 F. BOQ: BENCHMARKS OF QUALITY FULL INSTRUMENT ...................... 158 G. SUBSIST: SCHOOL-WIDE UNIVERSAL BEHAVIOR SUSTAINABILITY INDEX: SCHOOL TEAMS................................................. 171 H. ALTITUDE: ADVANCED LEVEL TIER INTERVENTIONS TREATMENT UTILIZATION AND DURABILITY EVALUATION ............... 175 I. RQ3 PARTIAL CORRELATIONS SPSS SYNTAX AND RESULTS ............ 179 J. TIPS TIERED DECISION GUIDELINES ........................................................ 220 REFERENCES CITED ................................................................................................ 230 xiv LIST OF FIGURES Figure Page 1. Improvement Science: Plan-Do-Study-Act Cycles (PDSA-C) model................... 7 2. Implementation Science: NIRN Active Implementation Framework Drivers ...... 9 3. Implementation Science: NIRN Active Implementation Framework Stages ........ 11 4. Diagram of PBIS Approach to School Improvement ............................................ 17 5. Positive Behavioral Interventions and Supports (PBIS) Tiers of Support ............. 19 2. Positive Behavioral Interventions and Supports (PBIS) Teaching Matrix Example. ................................................................................................................ 21 7. (Tier 1) Discipline Referral Form .......................................................................... 23 8. (Tier 2) CICO Daily Progress Report .................................................................... 26 9. (Tier 3) Individualized Progress Monitoring Forms .............................................. 30 10. Theoretical Model for Tier 1 PBIS Sustainability ................................................. 38 11. Theoretical Model for Advanced Tier PBIS Sustainability ................................... 39 12. School-Wide Information System (SWIS) Core Reports ..................................... 45 13. School-Wide Information System (SWIS) Drill Down ......................................... 46 14. Check-In Check-Out School-Wide Information System (CICO-SWIS) School-Wide Report............................................................................................... 49 15. Check-In Check-Out School-Wide Information System (CICO-SWIS) Average Daily Points by Student Report. .............................................................. 50 16. Theoretical Model of PBIS Implementation Team Data Use ................................ 52 xv LIST OF TABLES Table Page 2.1 School Demographics ......................................................................................... 60 2.2 Descriptive Statistics for PBIS Fidelity Across Tiers for Three Years for All School Types................................................................................................. 62 2.3 Descriptive Statistics for Tier 1 PBIS Sustainability (SUBSIST) in Year 3 (2018-2019) for All School Types ...................................................................... 68 2.4 Descriptive Statistics for Advanced Tier PBIS Sustainability (ALTITUDE) in Year 3 (2018-2019) for All School Types ...................................................... 70 2.5 Descriptive Statistics for Generation of SWIS Core Reports for Three Years for All School Types ........................................................................................... 74 2.6 Descriptive Statistics for Generation of SWIS Drill Down Reports for Three Years for All School Types ...................................................................... 75 2.7 Descriptive Statistics for Generation of CICO-SWIS Reports for Three Years for All School Types ........................................................................................... 78 3.1 Correlations (Spearman’s Rho) for Tier 1 PBIS Sustainability and Tier 1 PBIS Fidelity Across Three Years for All School Types ............................................. 99 3.2 Regression Coefficients of Tier 1 PBIS Fidelity on Tier 1 PBIS Sustainability Overall in Year 3 (2018-2019) for All School Types ........................................ 100 3.3 Correlations (Spearman’s Rho) for Advanced Tier PBIS Sustainability and Fidelity Across Tiers for Three Years for All School Types .............................. 101 3.4 Regression Coefficients of Year 3 (2018-2019) PBIS Fidelity on Overall Advanced Tier PBIS Sustainability for All School Types ................................. 102 3.5 Regression Coefficients of Year 3 (2018-2019) PBIS Fidelity on Overall Advanced Tier PBIS Sustainability for Only Elementary Schools ................... 102 3.6 Correlations for PBIS Sustainability Across Tiers in Year 3 (2018-2019) for All School Types ........................................................................................... 103 xvi Table Page 3.7 Correlations (Kendall’s Tau-b) for Tier 1 PBIS Sustainability, Fidelity, and Exploration of Indirect Metrics of Data Use in Year 3 (2018-2019) for Elementary Schools ............................................................................................ 104 3.8 Correlations (Kendall’s Tau-b) for Tier 1 PBIS Sustainability in Year 3 (2018-2019), Fidelity Across Three Years, and Tier 1 PBIS Data Use (Average Count of SWIS Core Reports Generated per Month) Across Three Years for Only Elementary Schools ......................................................... 106 3.9 Correlations (Kendall’s Tau-b) for Tier 1 PBIS Sustainability, Fidelity, and Exploration of Indirect Metrics of Data Use in Year 3 (2018-2019) for Elementary Schools ............................................................................................ 107 3.10 Correlations (Kendall’s Tau-b) for Advanced Tier PBIS Sustainability in Year 3 (2018-2019), Advanced Tier PBIS Fidelity Across Three Years, and Advanced Tier PBIS Data Use (Average Count of CICO-SWIS Core Reports Generated per Week) Across Three Years for Only Elementary Schools .......... 108 4.1 Questions to Guide PBIS Decisions by Implementation Stage .......................... 120 xvii CHAPTER I INTRODUCTION Statement of Purpose Efforts to understand school improvement have expanded from the traditional (and critical) focus on outcomes produced by specific educational practices to include larger implementation concerns across multiple related practices (Bryk, 2020; Slocum et al., 2014). This expanded view of school improvement attempts to understand the inner workings of (a) adopting effective practices that meet specific improvement needs and implementing each with high procedural fidelity (Forman et al., 2013), (b) sustaining the core elements of adopted practices over long time periods (Bradshaw & Pas, 2011; Hall, 2015), and (c) collecting and using data to guide both implementation and adaptation of effective practices to maximize effectiveness (Breiter & Light, 2006). This study examines the intersection of these three areas of implementation to identify how educators may better plan, monitor and guide school improvement. Examining the experiences of schools actively engaged in implementing school improvement holds promise for identifying which factors best support and predict the high procedural fidelity, sustained implementation, and effective data use needed for substantive student gains. The unifying lens for this analysis is the use of data by schools to understand implementation efforts and outcomes. Schools have become increasingly intentional in efforts to collect and use data to drive and sustain school improvement efforts. Iterative and collaborative use of data for decision-making to improve academic, social, emotional, and behavioral outcomes in schools has become a standard component in school routines both for internal decision making about implementation efforts and external (e.g., district, state) 1 accountability (Boudett et al., 2013; Horner et al., 2001; Koczwara et al., 2018; Lewis, 2015). Data-driven school improvement efforts may include the selection of specific practices or interventions to support student outcomes (e.g., academic performance, social behavior, attendance), systems to support staff implementation of one or more practices (e.g., professional development, incentives, materials), or broader conceptual models and frameworks (e.g., multi-tiered systems of support, explicit instruction) that have external data or evidence of effectiveness to address specific gaps or challenges at scales that are meaningful to the community (Fixsen et al., 2005; Flay et al., 2005; Horner et al., 2017). There are an increasing number of improvement innovations (e.g., practices, interventions, frameworks) considered to be evidence-based or promising (i.e., with emerging evidence) to improve school outcomes (Fixsen et al., 2019; Flay et al., 2005). One example of an evidence-based school innovation is teacher use of behavior specific praise statements to positively reinforce desired social behavior (Ennis et al., 2020; Markelz et al., 2019). Another example is incorporating supplemental reading interventions in small group instruction for students struggling to meet grade-level reading benchmarks (Coyne et al., 2018). In recent years, researchers and practitioners have recognized the importance of acknowledging the array of variables that affect the selection, adoption, implementation, and sustained use of best and promising practices in schools (Adelman & Taylor, 2007; Auld & Morris, 2016; Hawkins & James, 2017). Selection and adoption decisions about school improvement efforts range from classroom-level practices for individual or small group instruction (Park et al., 2017; Stevenson & Mussalow, 2019) to broad systemic policies and procedures that impact every member of the school community such as staff, students, and family/guardians/caretakers 2 (Antoniou & Kyriakides, 2013; Bradshaw et al., 2012). The use of a formal or informal measure of readiness for a specific improvement area is often used to guide the analysis of existing student and teacher data or to collect additional data that will support informed decisions about whether an intervention meets the intended need and is feasible to implement. Decisions about implementation and sustained use of an adopted improvement innovation also include ongoing assessment and evaluation of core components to determine whether the improvement efforts (e.g., core elements of the innovation) have been maintained over time as agreed on and the extent that the improvement goal(s) have been reached. Data may indicate that improvement efforts are effective with no change or that changes may be necessary to improve implementation or effectiveness. Intentional implementation using internal or external measures of readiness for specific innovations (Fixsen et al., 2019). Monitoring and addressing gaps in student performance or outcomes also reach beyond academic competence to assessment of the social, emotional, and behavioral development of students. One evidence-based framework for addressing these social, emotional, and behavioral needs is positive behavioral interventions and supports (PBIS; Bradshaw & Pas, 2011; Horner et al., 2017; Horner et al., 2009). The PBIS framework guides school personnel to strategically organize research-driven or promising practices that provide universal preventative supports to all students, as well as identify and intervene when groups or individuals need additional tiers of social, emotional, and behavioral support. Deeply embedded in the PBIS literature is an emphasis on data-driven decision making. Creating progress monitoring procedures and iterative review cycles that allows 3 the school implementation team to address needs efficiently and effectively or improve implementation across tiers of support (Bruhn et al., 2019; Irvin et al., 2006; Sugai & Horner, 2006). Recent research on PBIS reinforces this emphasis on use of data, suggesting that iterative cycles of using data to monitor progress is key to sustained, high-fidelity implementation of PBIS over time (McIntosh et al., 2013; McIntosh, Mercer, Nese, et al., 2018). While use of data has been identified as a predictor of sustainability, little is known about the specific procedures of data use (e.g., collection, organization, analysis, sharing) by PBIS implementation teams. This study builds upon the sciences of improvement and implementation, the PBIS framework, and existing models for data-based decision making to examine measures of sustainability, fidelity, and use of data across tiers of support in PBIS schools. Additionally, the study examined access of standardized reports in pre- packaged progress monitoring systems as indirect measures of data-based decision making by PBIS implementation teams. This study informs future research of the role that data and decision systems impact sustainable and high-fidelity implementation of the PBIS framework in schools. Literature Review This study examined three constructs (sustainability, fidelity, data use) related to improvement of schools through implementation of Positive Behavioral Interventions and Supports (PBIS), an evidence-based framework that specifically addresses school climate and student social, emotional, and behavioral needs. Abandonment of evidence-based innovations has led to an emerging literature base to address the factors of sustainability or the potential for sustained implementation over time. Sustainability is often inclusive of the 4 construct of fidelity or acceptable quality of implementation. There is a broad literature base that generally supports the use of data to drive improvement and implementation decisions in schools as well as emerging research supporting use of data as a predictor of sustained, high-fidelity implementation of the PBIS framework. The following sections will highlight the role of data in improvement science, implementation science, and the PBIS framework and introduce the specific theoretical models that will guide this study in the examination of sustainability, fidelity, and school use of data. Improvement Science Schools are not alone in their focus on improving core outcomes. Across many fields of industry and research in the U.S. there is an ever-increasing attention to improvement. Improvement efforts span across healthcare, social climate, education, economy, and environment (Marshall et al., 2013; Neely, 1999). This widespread attention to improvement, driven by competition or dissatisfaction with current outcomes, has prompted the emergence of new fields of research dedicated to examining successes and barriers that are shared across fields. One of these new fields is Improvement Science, designed to address improvement for dynamic and complex organizational structures (e.g., schools, hospitals, businesses) through intentional and systematic decisions about change (Langley et al., 2009; Lewis, 2015; Mason, 2008). Improvement science emphasizes the need for both disciplinary and profound knowledge to enact improvement as well as cyclical routines for continued adaptation and improvement over time. Bryk (2020) identifies six principles of improvement science: 1. Make the work problem-specific and user-centered. 2. Variation in performance is the core problem to address. 5 3. See the system that produces the current outcomes. 4. We cannot improve at scale what we cannot measure. 5. Anchor practice improvement in disciplined inquiry. 6. Accelerate improvements through networked communities. Principles four, five, and six collectively identify the intentional (disciplined and structured) collection and collaborative use of data to drive improvement decisions. Schools may use these improvement principles to guide many areas of improvements but will require comprehensive data systems as well as training and coaching, to develop and maintain data-driven instructional systems that are effective (Halverson et al., 2015; Vanlommel & Schildkamp, 2019). At the heart of improvement science is the use of iterative improvement cycles, typically based on the Plan-Do-Study-Act Cycle model (PDSA-C; McNicholas et al., 2019; Taylor et al., 2014), a process for building rapid cycles of implementing a practice or improvement and studying (and using) the implementation and outcome data to improve the practice for the next iteration (Figure 1). Data measuring implementation efforts includes valid and reliable indicators about the extent that the “do” phase was carried out as anticipated and any adaptations (e.g., modification, supplementary actions) that may have impacted the results or outcomes. .. Data on outcomes includes valid and reliable indicators about the impact that the ‘doing’ of the plan had on stakeholders and community members (e.g., students, families, staff), especially on the target area of improvement. 6 Figure 1 Improvement Science: Plan-Do-Study-Act Cycles (PDSA-C) model Note. This figure from the National Implementation Research Network demonstrates the iterative nature of the Plan-Do-Study-Act Cycles within improvement science. From right to left, three four-step PDSA cycles are lined up with arrows showing movement between steps within a single cycle and then leading into the next cycle. Implementation Science Growing up alongside Improvement Science is Implementation Science. The two fields are closely intertwined and share many common features. Implementation science builds from the use of iterative improvement cycles and expands on features that impact the successful adoption of existing improvement practices into organizations that are likely to benefit (Fixsen et al., 2009; Fixsen et al., 2019; Fixsen et al., 2005). The National Implementation Research Network (NIRN) provides one framework referred to as the Active Implementation Framework for understanding and applying principles of implementation science. The Active Implementation Framework identifies four stages that an organization typically moves between as they implement and sustain a specific 7 innovation as well as the drivers that influence the quality and speed of implementation. Implementation Drivers Organizations using the Active Implementation Framework, or similar models of implementation, attend to several factors that drive implementation (Figure 2) toward strong fidelity, consistent use, and finally improved outcomes (Blase, Van Dyke, et al., 2013; Fullan, 2011; Lindland et al., 2015). These factors or Implementation Drivers are organized into three categories including leadership, competency, and organization. First are Competency Drivers which include the selection, training, and coaching (e.g., prompting, performance feedback) of any mechanisms identified to develop, improve, and sustain implementer capacity to deliver the improvement practice or intervention (Atkins et al., 2017; Bethune & Wood, 2013). Leadership Drivers represent decisions made by stakeholders who are uniquely positioned with authority to allocate resources or influence with other stakeholders to enact change (Atkins et al., 2008). Leadership may need to adapt to improve capacity for the innovation or to better meet the needs of other stakeholders in their implementation efforts. For more complex improvement efforts, an organization may create a new leadership committee or team to focus solely on the implementation of one innovation or a similar set of innovations (e.g., school climate innovations). Organization Drivers include system interventions, facilitative administration, and a data-driven decision system to support decisions related to the mechanisms needed to prepare an organization to effectively embed the practice or intervention into the environment (Hagger & Luszczynska, 2014). Data-driven decision systems are inclusive of the systems of data collection and organization needed to inform decisions as well as 8 the procedures for extracting information from those data and translating them into actionable decisions that will improve implementation fidelity and organizational outcomes, capacity, or sustainability. Figure 2 Implementation Science: NIRN Active Implementation Framework Drivers Note. From the National Implementation Research Network. At the bottom is a triangle with arrows showing the integrated and compensatory relation between three implementation drivers. From the top of the triangle, arrows show the relation between the implementation drivers and fidelity, between fidelity and consistent use of innovations, and finally between consistent use and improved outcomes. 9 Stages of Implementation There are four stages (Figure 3) within the Active Implementation Framework that an organization moves between: Exploration, Installation, Initial Implementation, and Full Implementation. The first stage of the Active Implementation Framework is Exploration. During Exploration, decision makers of an organization move to (a) identify local performance strengths and gaps (Gilbert, 1978), (b) identify effective and promising practices that will logically improve performance gaps (Blase, Kiser, et al., 2013; Merrell & Buchanan, 2006), and (c) systematically prepare the organization (e.g., personnel, budget) for the installation stage. During Exploration, organizational leaders and stakeholders collaboratively collect and use data to drive decisions, determining when the organization decision-makers and its members agree about the innovation and establish the foundation to move into the Installation stage. During Installation, the organization acquires resources necessary to move forward with implementation and establish each of the drivers to ensure strong and sustainable leadership, competency, and organization as it pertains to implementing the innovation. During the Installation stage the organization leaders begin to resolve challenges and adapt their capacity as well as organization policies and procedures to best support implementation. Personnel may be asked to change their role, engage in training and coaching opportunities, and provide input in aligning the innovation to the local context. The first attempt to implement an innovation with recipients (e.g., students) occurs during Initial Implementation. In this stage, a small number of personnel and recipients are chosen to pilot the innovation to establish competence, and organization supports as well as identify potential barriers to enable effective and efficient diffusion to other personnel. 10 Once at least 50% of the practitioners are implementing with fidelity (based on valid and reliable measures aligned to the innovation) then the organization is ready for the Full Implementation stage. Figure 3 Implementation Science: NIRN Active Implementation Framework Stages Note. From the National Implementation Research Network. At the top of the image are four circles that represent the four stages of implementation with bidirectional arrows between stages. Within each stage is a triangle to represent the implementation drivers. Below each stage is a list of potential activities associated with each stage. At the bottom of the graphic is a bidirectional arrow that spans across stages and indicates that it takes 2-4 years to reach full implementation for an innovation. 11 The Full Implementation stage may take 2-4 years to attain, possibly moving back and forth between stages to address barriers or identify contextual adaptations necessary to fit the innovation to the local needs and values. During this stage there is an effort to establish data systems that document fidelity to the innovation and improvement outcomes. These data form the foundation of the PDSA Cycles and improve sustainability of the innovation. Sustainability may be defined as the potential to sustain the innovation so that it can withstand the many challenges (e.g., new initiatives, personnel turn-over) that may threaten implementation efforts over time. Sustainability is often achieved by embedding the core elements deeply into the policies and routines of the organization to become integral in the identity and core values of the organization. Since movement between stages is not often linear, it should be based on specific indicators of readiness, capacity, fidelity, and outcomes. Data-based decision making protects the organization leaders and stakeholders from moving too quickly toward a stage that it is unready for or too slowly and losing support or momentum. For example, an organization in the Initial Implementation stage may pilot a math curriculum with two groups of students until a specific number of lessons have been tested and assessment data have been compared to data from the current curriculum before training all practitioners. Alternately, an organization may achieve Full Implementation criteria but later determine that staff commitment or fidelity has waned, indicating that the organization should move back to Initial Implementation or even Installation. To confidently make decisions about readiness, capacity, fidelity, or outcomes of educational innovations, school decision makers must be able to rely on their collective data sources to meet certain assumptions (Breiter & Light, 2006; Horner, Sugai, & Todd, 12 2001). The first assumption is that the data collected meets an ongoing information (decision-making) need in the school. Given the emphasis on iterative decision making in both improvement and implementation sciences, most data should be sensitive to changes, but not all changes are equally relevant to decision makers. Collection and presentation of excessive or irrelevant data to decision makers is likely to impact the efficiency of decision making just as the lack of useful data is likely to impact its effectiveness. A second assumption is that the data will be simple and efficient in both content and delivery. Implementation science relies on shared commitment from stakeholders to meet a common need using a promising or evidence-based innovation. To maintain this shared commitment, the content and presentation (e.g., charts, graphs) of the data should simply and directly connect to the commitments and goals identified by leaders and stakeholders. Given the natural adaptations that occur throughout the implementation stages, data sources, collection procedures, and presentation methods may also require adaptation over time to maintain simplicity and efficiency for decision makers. The third assumption that decision makers must rely on is accessibility of data to the team for both scheduled review cycles (e.g., team meetings) and unscheduled decision events (e.g., unexpected problem, external request). Accessibility does not inherently imply direct and unlimited access by all stakeholders. Direct access to data may at times be limited to small numbers of personnel to maintain confidentiality or reduce the need for wide-scale training on data systems. Instead, accessibility of data refers to maintaining an appropriate number of personnel with training, direct access, and time allotted in their schedules to provide data to decision makers before (or during) decision events, scheduled or unscheduled. 13 Positive Behavioral Interventions and Supports (PBIS) The innovation of interest for this study is the positive behavioral interventions and supports (PBIS) framework. PBIS is an evidence-based, multi-tiered approach to improvement of social, emotional, and behavioral outcomes across all students (Bradshaw & Pas, 2011; Horner et al, 2009). More than 29,000 schools in the U.S. currently implement the PBIS framework (Center on Positive Behavioral Interventions and Supports, 2021). According to the Handbook of PBIS (Dunlap, Sailor, et al., 2009) and its predecessors (Effective Behavior Support, Positive Behavioral Supports) the PBIS framework was built on four tenets: 1. “application of research-validated behavioral science”, 2. “Integration of multiple intervention elements to provide ecologically valid, practical support”, 3. “Commitment to substantive, durable lifestyle outcomes”, and 4. “Implementation of support within organizational systems that facilitate sustained effects”. During the 1960s, 1970s, and 1980s social justice movements received historically high levels of attention in the U.S. Social justice efforts included increased supports and rights for members of racial and ethnic minorities and individuals with a wide range of exceptionalities and documented disabilities. These movements included access to free, appropriate public education, which required schools to meet minimum standards of accessibility to academic and eventually social-behavioral supports for all students (Zettel & Ballard, 1979). During this same time, the technology of applied behavior analysis 14 (ABA) emerged, which quickly demonstrated effectiveness in improvement outcomes for individuals with severe emotional and behavioral disorders. During the 1990s and early 2000s, the principles of ABA were combined with a popular model of tiered public health supports to systematically assess and address social, emotional, and behavioral needs of students. This model evolved into the comprehensive framework of PBIS. Implementation of PBIS has been linked to several areas of school improvement of outcomes in schools (Horner et al., 2017), including increased perceptions of school safety (Horner et al., 2009), decreased incidents of student problem behavior such as office discipline referrals and suspensions (Barrett et al., 2008; Taylor-Greene et al., 1997), as well as improvements in student academic achievement (Bradshaw et al., 2010; Bradshaw et al., 2012). Practices and interventions implemented within the framework to support targeted or individualized behavioral needs have also demonstrated effectiveness. For example, Check-In Check-Out (CICO; Hawken et al., 2021) is an intervention commonly embedded within the PBIS framework for students who need just a small increase in dosage of social skills instruction, adult interaction, positive reinforcement, and performance feedback. This intervention has been primarily studied within the context of PBIS implementation and has been linked to decreased problem behavior and increased academic engagement (Hawken et al., 2014; Hawken et al., 2011; Maggin et al., 2015). The PBIS Approach There are four fundamental areas of improvement that schools must address to effectively implement PBIS, shared outcomes, core practices, organization systems, and use of data to drive decisions (Sugai & Horner, 2006). First, schools implementing PBIS identify a set of goals outcomes related to school climate as well as social, emotional, and 15 behavioral capabilities for students. These outcomes are developed collaboratively and must align with the core values across the stakeholders or members of the school community. The shared outcomes are used to guide in the selection and adoption of practices (e.g., expectations, social skills curriculum, interventions) embedded within PBIS tiers to address specific student performance gaps or needs. Schools are encouraged to choose the fewest possible student practices to reduce overlap and maximize resources while comprehensively meeting needs. Just as with the outcomes, the practices must be selected collaboratively so that all stakeholders are invested in implementing with fidelity. Implementing practices with fidelity requires attention to organizational systems (or the Organization Drivers). Systems are established to enable staff to implement the broader PBIS framework (e.g., consistent response procedures to expected and problem behavior) and any selected practices within the framework (e.g., social skills curriculum, targeted interventions) with fidelity over time. The systems are essentially the supports provided to adults as they fulfill their assigned role within the school. Finally, data systems are established to guide improvement decisions across practices, systems, and outcomes within PBIS. Valid and reliable measures of fidelity to core elements of PBIS and individual practices as well student outcomes make up the core of the PBIS data system. The PBIS data system is embedded within a collaborative data- based decision-making model to support iterative cycles of improvement. Together, these goals, practices, organizational systems, and information systems form the foundation of PBIS (Figure 4). 16 Figure 4 Diagram of PBIS Approach to School Improvement Note. From the Center on PBIS website. A Venn diagram shows three overlapping circles labeled practices, systems, and data. A larger outer circle is labeled outcome/goal to indicate that all practices, systems, and data should align or match to specific school improvement goals. 17 Tiers of Support in PBIS Identifying shared outcomes, core practices, organization systems, and use of data to drive decisions occurs within tiers of support intensity. The tiers within PBIS are intended to maximize resources and effort by layering the practices, systems, and data on a continuum that matches support intensity to the specific needs of students. Tier 1 is the most recognizable of these layers because it forms the universal prevention practices targeting for all members of the organization in all places at all times of day. Advanced tiers of support build upon the foundational practices, increasing the intensity and complexity of supports as needed for students. Tier 2 provide a modest and highly efficient boost in student supports through standardized interventions and practices, often in targeted routines or locations (e.g., cafeteria, small group reading) where challenging behavior is more likely to occur. Tier 3 is the most intensive and resource-intensive supports that are individualized based on behavioral assessments and the recommendations of a uniquely formed team that understands the student and family needs as well as the context and function of the student’s unique social, emotional, and behavioral challenges. Systems of advanced tiers of support in PBIS are usually adopted and implemented more slowly and preferably after Tier 1 is firmly in place and effectively supporting at least 80% of students (Kittelman et al., 2021). This allows the school to accurately identify student needs that cannot be met by universal supports. The tiers of support are most often represented in terms of a triangle that shows the proportion of student who receive one, two, or all layers of support (Figure 5). 18 Figure 5 Positive Behavioral Interventions and Supports (PBIS) Tiers of Supports Note. From the Center on PBIS website. Image shows a large green triangle labeled Tier 1 to represent all students. Within the green triangle is a smaller yellow triangle near the top labeled Tier 2 for some students who need a modest boost in behavioral supports. Finally, the smallest, red triangle is embedded within the yellow triangle and is labeled Tier 3 for a few students who require more individualized or intensive behavioral supports. This and similar graphics are often used to represent the tiered approach to behavioral supports in PBIS and other MTSS frameworks. 19 Tier 1. Tier 1 PBIS supports form the foundation of the PBIS framework and include the practices, systems, and data representative of all members of the community (often in terms of students). Tier 1 is often referred to as universal prevention or the green portion of the triangle, representing all students. It has sometimes been associated with 80%, not because it is delivered to 80% but because effective Tier 1 supports at least 80% of enrolled students without a need for additional behavioral tiers. A Tier 1 PBIS school leadership team is made up of representatives across stakeholder groups (e.g., administration, teachers, paraprofessionals, students, family members) who meet at least monthly to evaluate and make implementation decisions at the school or system level. Tier 1 practices and systems. Examples of PBIS practices adopted at Tier 1 include: 3-5 school-wide behavioral expectations, a school-wide social skills curriculum, standard procedures for reinforcing student behavior that aligns with school-wide expectations, and standard procedures for correcting and documenting student behavior that does not align with school-wide expectations. To support staff in implementing these practices competently, the Tier 1 leadership team establishes Tier 1 systems to ensure all Tier 1 practices are delivered with adequate fidelity. For example, staff training and coaching needed to implement each individual practice or and maintaining materials needed to teach the school-wide expectations (Figure 6; Sugai, 2008) or deliver social skills lessons within the adopted curriculum. Tier 1 systems may also include prompting procedures for staff like putting a certain number of reinforcement tokens or coupons into mailboxes at the beginning of each week as a reminder to acknowledge and reinforce desired behaviors. 20 Figure 6 Positive Behavioral Interventions and Supports (PBIS)Teaching Matrix Example Note. From the Center on PBIS website. This sample PBIS teaching matrix identifies school-wide behavioral expectations and how they are applied or interpreted in specific school settings. For example, Respect Ourselves in the Hallway setting means to walk and Respect Property on the Bus means wiping your feet and sitting appropriately. Tier 1 teaming and data use. To support Tier 1 leadership to make decisions, evaluation data are needed to measure fidelity of implementation and outcomes related to Tier 1 supports. Fidelity measures indicate overall Tier 1 PBIS framework across implementers and leaders, like the PBIS Tiered Fidelity Inventory (TFI-Tier 1; Algozzine et al., 2014), or specific Tier 1 practices like the Classroom Management Self-Assessment (Simonsen et al., 2006) and the self-reported counts of delivering positive reinforcement tokens. Tier 1 outcome data should represent patterns across all students, or at least 21 represent all student subgroups (e.g., gender, disability, grade level, ethnicity/race) across enrolled students. Examples may include a school climate survey (Bear et al., 2011; La Salle et al., 2018), attendance records, and universal screening measures like the Student Risk Screening Scale (SRSS; Drummond, 1994; Lane et al., 2012) to determine whether the Tier 1 supports are effective for most students (e.g., 80% or more) and identify any students that are not responding to their current level of support. One common source of Tier 1 student outcome data are discipline referrals, sometimes called office discipline referrals that collect details of an incident that involved one or more student engaging in problem behavior that violated the school-wide expectations or code of conduct. Discipline referral forms collect details such as student information, location, time of day, problem behavior, staff action taken, and the perceived motivation of the behavior (e.g., avoid task, get attention). Figure 7 provides an example of a discipline referral form. Tier 1 PBIS evaluation data (e.g., fidelity, outcomes) collectively inform the Tier 1 PBIS school team about the current implementation efforts across Tier 1 practices and systems and their effectiveness in improving the school behavioral patterns. The team primarily uses this information internally during iterative review cycles (e.g., monthly team meetings) to identify, prioritize, and solve problems but also shares the information with other stakeholder groups to celebrate improvements, recruit feedback on problems and priorities, and maintain general communications about PBIS implementation efforts. To maximize the use of data collected (e.g., discipline referrals), the team adopts a formal or informal data system to store and organize data so that it is accessible and formatted to clearly identify patterns, trends, and peaks that are useful for decision making (Horner et al., 2001). 22 Figure 7 (Tier 1) Discipline Referral Form Note. A sample discipline referral form from the PBISApps website showing the demographic and incident information (e.g., name, behavior, motivation) collected when a behavioral incident occurs. 23 Advanced Tiers. Advanced tiers of Support in PBIS describe the additional layers of practices, systems, and data that are available for up to 20% of students who are not responsive to Tier 1 supports alone or meet criteria for risk of school failure due to social behaviors. There are most often externalizing problem behaviors but may also be internalizing behaviors that interfere with the student’s social-emotional development (Hunter et al., 2013). Advanced tiers of support typically include Tier 2 and Tier 3 supports, although some models include four or more tiers to further identify the level of support individualization or intensity provided. Given the complexity of implementing advanced tiers, the Center on Positive Behavioral Interventions and Supports (PBIS) recommends that schools fully implement Tier 1 prior to introducing advanced tiers of support. Recently Kittelman et al. (2021) reported that staggering or lagging implementation as well as higher level of fidelity predicted higher initial fidelity when introducing advanced tiers. For example, researchers found that schools with a 2- or 3-year lag between initial Tier 1 launch predicted higher initial (year 1) fidelity of Tier 2. Tier 2. Targeted or Tier 2 support refers to one or more standardized interventions that large groups of students can simultaneously access when there is indication of mild-to- moderate risk of school failure. Tier 2 interventions are group-based and standardized to provide a highly efficient, low-cost, low-effort boost of supports that address the function and context of commonly identified social, emotional, or behavioral needs. Within the tiered logic of PBIS, a school organizes Tier 2 procedures and interventions around widely shared behavioral needs across students (e.g., attention-maintained classroom disruption, general social skill deficits, work-avoidance tardiness) and maintains the capacity (e.g., materials, implementers) to support 10-15% of students across the school year (Hawken et 24 al., 2009). A school’s PBIS leadership team (e.g., Tier 2 team, advanced tier PBIS team) will meet more frequently than Tier 1 teams (e.g., biweekly) to evaluate and make decisions about the Tier 2 practices and systems across the school. This team also monitors or appoints a subcommittee to monitor individual student outcomes to ensure that supports are matched appropriately and faded as soon as the student had demonstrated sufficient self-management skills. Data systems to support team decisions are adopted or developed based on specific intervention data and coordination needs across interventions. The most implemented Tier 2 practice is the Check-In Check-Out or CICO intervention, which consists of inviting students to meet with an adult mentor or facilitator at the beginning and end of each day as well as interacting with staff members periodically (e.g., 5-10 times) across the day to recruit and receive performance feedback on following school-wide expectations (Bruhn et al., 2013; Hawken et al., 2021). Performance feedback is collected using a daily progress report such as the one provided in Figure 8, and students receive reinforcement when their daily goal (e.g., earned 80% of points) is met. The CICO intervention has demonstrated effectiveness in improvement of academic engagement and decreases in problem behaviors (Maggin et al., 2015; Sullivan, 2015; Todd et al., 2008). Systems that support CICO implementation include a committee or team that manages intervention activities, training for all school staff on providing performance feedback, screening tools and criteria for identifying and matching students to CICO, and a private location for mentors to meet with students. Data-based decision making within CICO often includes the use of student point data to monitor outcomes (Scott et al., 2010) and measures of overall and implementer-specific fidelity to monitor whether the core elements of CICO (Algozzine et al., 2014; Sullivan, 2015) are maintained at acceptable levels. 25 Figure 8 (Tier 2) CICO Daily Progress Report Note. Sample CICO daily progress report or point card from the PBISApps website. This form includes placeholders for the school-wide expectations and generic time periods across a school day. Based on demonstrated behavior during eight brief (30-60 minute) periods, the student receives 0, 1, or 2 points for each school-wide expectation and can earn up to 48 points for a full school day. The total percent of points earned by the student is typically used to determine whether the student has met their daily goal, usually associated with receiving a pre-determined daily reinforcer (e.g., school behavior ticket, tangible item, activity). Over time, the percent of points earned is also used to determine whether modifications, fading, or graduation from CICO is appropriate. 26 Tier 3. Tier 3 refers to supports that are individualized, comprehensive, and designed to address a unique or intensive gap in social, emotional, or behavioral performance. Tier 3 supports often require the highest effort and cost and is the final layer of support provided when Tier 1 and Tier 2 supports have proved ineffective in addressing student needs (Bambara et al., 2012; Strickland-Cohen et al., 2018). When Tier 1 and Tier 2 supports are effective and proportional to the total school enrollment, Tier 3 supports should be required for 5% or less of the population. At this layer of PBIS there is increased attention to identifying the unique context and function of the behavior so that an intervention plan can be tailored to meet those unique needs. The progress monitoring system and implementation team are also individualized to match the uniqueness and intensity of the intervention plan. Given the individualized nature of Tier 3 supports, schools may include a wide and diverse range of practices, systems, and data within this tier. For example, a student whose behavior does not respond to a standardized Tier 2 (e.g., CICO) but does respond to a modified and individualized version of that intervention may be included in Tier 3 supports. However, another student who receives Tier 3 supports might receive wraparound supports that comprehensively and uniquely address home, school, and community needs (Eber et al., 2011). A PBIS school leadership team (e.g., Tier 3 team, advanced tier PBIS team) will meet regularly (e.g., twice monthly) to evaluate and make decisions about the Tier 3 practices and systems across the school. This team also appoints and supports student- centered subcommittees (e.g., student support team) to monitor individual student outcomes to ensure that supports are matched appropriately and faded as soon as the student has met a set of individualized goals and is likely to respond to Tier 2 supports (or 27 lower-intensity Tier 3 supports). Tier 3 practices that a student may receive often include the use of a model or procedural checklist for (a) completing a functional behavior assessment to identify individual student strengths and needs, (b) appointing a student-centered support team, (c) developing a behavior support plan to identify strategies to meet those needs, and (d) using an evaluation system to monitor the fidelity and outcomes of the behavior support plan (Alberto & Troutman, 2008; Umbreit & Ferro, 2015). Two examples of models for organizing Tier 3 supports and progress monitoring are the Basic FBA to BIP (Borgmeier et al., 2017; Strickland-Cohen & Horner, 2015) and the Prevent, Teach, Reinforce or PTR model (Barnes et al., 2020; Dunlap, Iovannone, et al., 2009). These models are structured or semi-structured procedures for conducting functional behavior assessments and developing the behavior support plans and progress monitoring procedures to meet an individual student’s needs. Tier 3 systems may include training and coaching for staff directly involved in implementing a behavior support plan and procedures for adjusting staff schedules and responsibilities to allow for conducting observations (to collect data) during routines when the student is most likely to exhibit the problem behavior. Tier 3 systems may also include training all staff in general principles of Tier 3 and procedures for nominating students who may benefit from advanced tiers of support. According to one measure of PBIS fidelity, the Tiered Fidelity Inventory (TFI; Algozzine et al., 2014), data-based decision making at Tier 3 includes both (a) student-level and (b) system-level decisions. Student-level decision teams are uniquely formed around a student and use individualized indicators of behavior support plan implementation and student 28 responsiveness or outcomes. Like the Tier 2 CICO intervention, daily progress reports or similar behavior rating scales are often incorporated into Tier 3 student progress monitoring procedures but are often individualized to address target behaviors rather than the school- wide expectations. Use of an individualized daily progress report increases structure and performance feedback on the student’s behavior in a way that is consistent with procedures for CICO (an intervention all school staff are trained to implement) as well as provide intermittent data to drive decisions that are tailored to the student. Additional data may also be collected to monitor detail about specific target (desired or problem) behaviors. Figure 9 provides examples of individualized data fidelity and outcome collection tools for a student who receives Tier 3 supports that includes both a daily progress report and frequency data for specific student behaviors. Given the individualized nature of interventions and systems to individual student needs, adopting procedures or standardized data systems to summarize Tier 3 data at the student- and school-wide levels present many challenges to student support teams as well as the school Tier 3 leadership teams. System-level decision making for Tier 3 is managed by a broader school-wide team that collects broader indicators of fidelity to the Tier 3 systems across staff delivering and general outcomes across students receiving Tier 3 supports within the school. For example, student support teams that manage individual behavior support plans may be required to report average fidelity scores, percent of days when fidelity was acceptable, and the status of meeting behavior support plan goals (e.g., progressing, not progressing, fading procedures implemented). These broad indicators can then be summarized across student support teams for an overall indicator that Tier 3 supports are being implemented with fidelity and are generally effective in meeting student needs. 29 Figure 9 (Tier 3) Individualized Progress Monitoring Forms Note. Sample Tier 3 progress monitoring forms from the PBISApps website. Top left shows an individualized daily progress report with a student-specific written and picture schedule, behavioral goals, and points earned. Bottom left provides teachers space to tally their own fidelity ratings along with occurrence of target problem behaviors and total instructional hours in the student’s day. Top right shows an implementation checklist with a list of behavior plan strategies (e.g., prompt use of picture schedule) for implementers to self-rate fidelity on a Likert-type scale of 1-4. Bottom right shows a student self-monitoring form to track individualized behavior goals (e.g., stay on task) with an interval prompting device. 30 Fidelity, Sustainability, and Data Use in PBIS This study examines three constructs shared across improvement science, implementation science, and the PBIS framework: fidelity of implementation, sustainability of implementation, and the use of data to drive improvement and implementation decisions. PBIS Fidelity Fidelity or integrity of implementation has been widely used within research to link innovations of interest with the results or outcomes observed (Ledford & Gast, 2014; Peterson et al., 1982). More recently there have been efforts to expand the measurement of fidelity to practitioners as a standard assessment procedure to maintain implementation standards over time, especially while making cultural adaptations to fit local values or needs (Castro-Olivo et al., 2018; Evanovich & Kern, 2018; Harn et al., 2013). PBIS has maintained a high emphasis on monitoring fidelity within schools. As outlined in the PBIS Evaluation Blueprint (2020), fidelity of implementation within PBIS most often refers to the standardized measures of school fidelity to core components of the framework such as appointing school leadership team(s) and subcommittees across tiers, establishing tiered practices and systems within each tier of support, adopting evaluation measures and procedures to guide decision-making, and protecting resources required to implement practices and systems with fidelity. Over the last 20 years, PBIS implementers have sought to develop valid and reliable measures of PBIS fidelity to inform both researchers and practitioners about the extent that core components of PBIS are embedded into school routines. Several different PBIS fidelity tools have been developed to-date, each attempting to serve a specific set of evaluation needs. For example, the School-Wide Evaluation Tool (SET; Horner et al., 31 2004) was designed to evaluate PBIS fidelity for research purposes and ensure that outcomes can confidently be associated with core components of the framework. The Benchmarks of Quality (BoQ; Kincaid et al., 2005) was developed to support school Tier 1 leadership teams and district coaches in self-monitoring a comprehensive set of Tier 1 core components. Most recently, the Tiered Fidelity Inventory (TFI; Algozzine et al., 2014), was developed to support school leadership team(s) and district coaches across tiers in more efficiently self-monitoring a set of high-priority core components of PBIS across tiers. Companion guides to the TFI have also been developed for schools that have successfully established strong PBIS implementation over time to evaluate more advanced initiatives such as cultural responsivity and mental health supports within the PBIS framework (Barrett et al., 2016; Leverson et al., 2021). Additional practice- or intervention-specific fidelity measures are administered as appropriate. For example, Simonsen et al. (2006) identified several classroom practices related to PBIS that teachers can self-assess (or a coach can observe and provide external feedback) including the ratio of positive-to- negative statements, opportunities to respond per minute, and arranging the room to minimize crowding and distraction. Measurement of fidelity is often viewed as simply a tool of external accountability or establishing a standard of quality that yields desired outcomes (Childs et al., 2016) but within the PBIS framework, the use of fidelity focuses on internal use of the data provided to PBIS implementers. During early stages of implementation, fidelity measures support school teams to identify features in place, partially in place, or not in place and to determine where to focus implementation efforts. During later stages of implementation, regularly monitoring fidelity to the PBIS framework over time supports decisions about 32 sustained implementation of core features especially as schools engage in programmatic or individual (teacher or student) adaptations made to core elements and may be shared with staff to encourage shared commitment to PBIS over time. PBIS Sustainability Maintaining core elements of PBIS is only useful if those features can be scaled to produce meaningful student outcomes that can be sustained over time. Recent efforts are underway to measure and examine the potential for sustained implementation over time or the sustainability of PBIS core elements separately from fidelity of implementation of those same features (McIntosh, Horner, et al., 2009). While fidelity measures provide a snapshot of whether core elements of the innovation are being implemented and are closely related to the continued use of those features (Yeung et al., 2016), the emphasis is on keeping basic features in place. Sustainability expands beyond simple maintenance and includes the examination of separate enablers and barriers that may impact the likelihood, speed, or quality of implementing the core elements in the future. Additionally, sustainability directly addresses the need for continuous regeneration or adaptations to the core elements to maximize resources and improve the relevance of those features to ever-changing local needs (McIntosh, Horner, et al., 2009). The need for research on sustainability is rooted in the same issue that sparked improvement and implementation science, and abandonment of evidence-based practices. McIntosh et al. (2016) analyzed fidelity of PBIS implementation data from 5,331 schools over a five-year period. The patterns of fidelity were grouped into four categories, including Sustainers, Slow Starters, Late Abandoners, and Rapid Abandoners. Sustainers were identified as schools that met recommended fidelity criterion across all five years. 33 About 29% of schools fit into the Sustainer category. Slow Starters were those that showed inconsistent patterns of reaching fidelity criterion across the first three years but increased and met criterion in the final two years. About 13% of participating schools fit in the Slow Starters category. Late Abandoners were identified as schools that reached fidelity criterion within the first three years but dropped off or were unlikely to reach criterion in the fourth and fifth year. About 24% of schools fell into the Late Abandoners category. Finally, Rapid Abandoners were schools that reported reaching fidelity criterion in the first year but abandoned or reported low fidelity in the remaining years. About 34% of schools fell into the Rapid Abandoners category. Research on enablers and barriers of PBIS adoption and sustainability (Hieneman & Dunlap, 2000, 2001; Kincaid et al., 2007) have evolved into a list of indicators or factors for PBIS schools and researchers to attend to as they attempt to create robust implementation procedures that will withstand a wide range of potential barriers that may be encountered (Bambara et al., 2009; Robertson et al., 2020; Turri et al., 2016). At Tier 1, both school- and district-level variables have been identified as influencers over PBIS sustainability (McIntosh, Horner, et al., 2009; McIntosh, Mercer, et al., 2018). Several of these indicators are measured in a survey of Tier 1 PBIS sustainability called the School- Wide Universal Behavior Sustainability Index: School Teams (SUBSIST; McIntosh, Doolittle, et al., 2009). Most recently, variables have been identified that collectively or separately influence sustainability of advanced tiers of PBIS (Tier 2, Tier 3). These variables were used to develop a new measure of advanced tier PBIS sustainability called the Advanced Level Tier Interventions Treatment Utilization and Durability Evaluation (ALTITUDE; McIntosh, Kittelman, et al., 2018). 34 A more recent study by Kittelman et al. (2020) followed up with personnel from 30 schools that abandoned and later readopted the PBIS framework. The study identified the perceived reasons for abandonment included lack of staff buy-in/commitment/enthusiasm, lack of school administrative support, lack of staff consistency, and low fidelity of implementation. The most frequently identified reason for readoption was new school administrator with additional responses relating to district support, observing success in neighboring schools that implemented, and additional funding. Initial efforts to understand the role and factors of adoption, abandonment, and sustainability all share the goal of developing a technology for PBIS sustainability that can be applied to school improvement and PBIS implementation efforts. Moving from theoretical models and measures of sustainability toward an applied technology will improve the ability of implementers to address issues of sustainability in the early stages of implementation and with greater precision. While this long-term goal has not yet been reached progress has been made toward predicting sustained implementation with adequate fidelity at Tier 1, the tier most likely to be implemented first within the framework. Tier 1 PBIS Sustainability. Because of the time required to reach full implementation of universal supports and the complexity of implementing advanced tiers, research on PBIS sustainability has primarily focused on universal or Tier 1 systems of support. School-level and district-level factors have been identified as predictors of sustained implementation over multiple years. Several school-level factors that have been found to significantly predict PBIS Tier 1 sustainability, including acknowledgement systems (e.g., token economy), matching instruction to student ability (e.g., age-appropriate social skills curriculum), self-reported Tier 1 implementation fidelity, and team actions 35 such as use and sharing of data (Mathews et al., 2014; McIntosh, Mercer, et al., 2018b). District-level capacity building has also been identified as a predictor of sustained PBIS implementation (Kittelman, McIntosh, et al., 2019). George et al. (2018) identified several additional themes that district staff perceived important to district-level capacity including coordination capacity, coaching, teaming, and a data infrastructure. In one study, McIntosh et al. (2013) identified Team Use of Data and District Capacity Building as two significant predictors of sustained Tier 1 PBIS implementation with fidelity at or above criterion. As a follow-up to these results, McIntosh et al. (2015) examined a larger sample of schools, including a set of three additional questions related to school team actions. The first school team action question asked respondents to identify frequency of PBIS team meetings on a 5-point Likert-type scale from every other month to weekly. Second, respondents were asked the frequency that teams shared data with the whole school staff on a 10-point Likert-type scale from less than once per year to weekly. Finally, respondents were asked to identify the hours of PBIS coaching received by the school team on a 7-point Likert-type scale from none to more than 5 hours per week. Results of latent regression analyses of variables predicting individual factors of Tier 1 sustainability indicated that frequency of team meetings was a significant predictor for School Team Use of Data, District Priority, and District Capacity building factors of Tier 1 PBIS sustainability. Frequency of sharing data with staff was a significant predictor of School Priority and school Team Use of Data. Access to coaching was only a significant predictor of District Capacity Building. The theoretical model for PBIS sustainability at Tier 1 (McIntosh, Doolittle, et al., 2009; McIntosh et al., 2011) is presented in Figure 10. The model identifies sustained 36 fidelity of core elements of Tier 1 PBIS as a mediator of the universal distal student outcomes targeted by the framework including (a) improved social competency, (b) improved academic achievement, and (c) reduced problem behavior (e.g., discipline referral rates, suspension). Sustained fidelity of implementation is a direct result of the four factors of PBIS Tier 1 sustainability (School Priority, Team Use of Data, District Priority, Capacity Building). Figure 10 Theoretical Model for Tier 1 PBIS Sustainability Note. Adapted from McIntosh, Doolittle, et al. (2009) and McIntosh et al. (2011). Theoretical model for Tier 1 PBIS sustainability. This model identifies four factors of sustainability that impact sustained fidelity of Tier 1 PBIS systems which in turn impact distal student outcomes of implementing PBIS at Tier 1 (improved social competence, improved academic achievement, reduced problem behavior). 37 Advanced Tier PBIS Sustainability. Very recently, interest in PBIS sustainability has turned to factors that would predict the implementation of PBIS at advanced tiers of support (Tier 2 and Tier 3), separately from the factors that would predict general PBIS implementation and Tier 1 specifically. The more complicated and nuanced nature of targeted (Tier 2) and individualized (Tier 3) supports will conceptually present challenges that differ from adopting universal (Tier 1) prevention systems and practices intended to be implemented by all staff members for all students. These predictors of advanced tier PBIS sustainability have been slower to emerge given that fewer schools report measures of implementation (e.g., fidelity, outcomes) of advanced tiers, possibly indicating slower adoption rates of Tier 2 and Tier 3 systems of support (Debnam et al., 2012; Kittelman et al., 2018). Potential factors related to sustainability of advanced PBIS tiers have been identified for a small number of specific Tier 2 interventions (Loman et al., 2010) as well as general models and procedures for Tier 3 supports (Bambara et al., 2009; Bambara et al., 2012). Figure 11 provides the theoretical model for PBIS sustainability at advanced tiers (Tier 2, Tier 3) of support (McIntosh, Kittelman, et al., 2018; McIntosh, Mercer, Horner, et al., 2018). Like the theoretical model for Tier 1 PBIS sustainability, this theoretical model identifies sustained fidelity of core elements of advanced tiers of PBIS as a mediator of three advanced tier distal student outcomes targeted by the framework, including (a) improved academic achievement, (b) reduced incidence of students identified with a behavior disorder, and (c) reduced discipline data for students identified with one or more disability. 38 Figure 11 Theoretical Model for Advanced Tier PBIS Sustainability Note. Adapted from McIntosh, Mercer, Horner, et al. (2018) and McIntosh, Kittelman, et al. (2018). This model identifies several factors of sustainability, context, and district support that impact sustained fidelity of advanced tier PBIS systems, which in turn impact distal student outcomes of implementing PBIS at advanced tiers (improved academic achievement, reduced incidence of behavior disorders, reduced discipline for students with disabilities). Given the complexity of advanced tiers, sustained fidelity of implementation for advanced tiers is impacted by a combination of school and district contextual factors, district supports available (e.g., training, coaching), and sustainability factors. The general factors of sustainability are identified as School Priority, Tiered Team Use of Data, and Implementation Capacity. Some features of advanced tier PBIS sustainability are 39 theoretically shared across Tier 2 and Tier 3 supports (e.g., administrative support, strong home partnerships), while others are distinct to Tier 2 (e.g., school personnel knowledgeable about Tier 2 logic and interventions, Tier 2 or group-based behavior data system) or Tier 3 (team with knowledge about Tier 3 supports, individualized behavior data system). By attending to the factors of sustainability shared across tiers and for specific tiers of support, school leadership teams may be better equipped to (a) identify when the school is ready to begin exploration and installation of the next tier of support without abandonment of previous tier(s) and (b) consider factors of sustainability while planning implementation activities at each stage. PBIS Data Use Use of data and data-based decision making in PBIS is a challenging construct given the wide range of data systems that schools may choose to collect and use as indicators of implementation fidelity and student outcomes across practices and systems. Newton et al. (2012) developed a model called Team Initiated Problem Solving (TIPS) for Tier 1 PBIS teams to use as they meet to analyze and translate data collected into useful information for decision making. During initial research on TIPS, the primary source of outcome data used were office discipline referrals collected within an online data system, the School-Wide Information System or SWIS (May et al., 1998) to review patterns of inappropriate behaviors and make decisions about school climate and system-level problems. Within the model, the team appoints a data analyst to generate reports prior to team meetings and present precise problem statements with supporting graphs to the team before or at the beginning of the meeting. The team then prioritizes one or two problems, and the remainder of the meeting is used to identify solutions and create an action plan to 40 address and monitor the problem (Newton et al., 2009; Todd et al., 2012). Another TIPS study examined contents of the Decision Observation, Recording, and Analysis (DORA) tool, a research tool developed and validated to capture observations of team meetings for PBIS teams that adopted the TIPS model. By analyzing 44 problem statements discussed across 18 meetings in 10 elementary schools documented by trained DORA evaluators, Algozzine et al. (2016) were able to examine specific teaming and decision procedures by PBIS teams that involved the use of data as well and compare them to student outcome data collected about discipline referral patterns in SWIS. Results indicated that team engagement in problem-solving and use of quantitative data recorded on the DORA were associated with positive changes in student outcomes (reduced discipline referrals in general or specific contexts). The most recent randomized waitlist-controlled study on implementing TIPS (Horner et al., 2018) collected team meeting observations and indicated that teams implementing the model were more likely to identify problems with adequate precision to develop actionable solutions, implement those solutions with fidelity, and document improved student outcomes (reduced rates of discipline referrals and out of school suspensions per 100 students). Discipline Referral Data. Within PBIS research, discipline referral patterns have been examined not only as a dependent variable or outcome of PBIS implementation (Lewis et al., 1998; Scott & Barrett, 2004), but also as a source of evaluation data within the framework. For example, Sugai et al. (2000) analyzed patterns of discipline referral rates as early indicators of problem behaviors that may lead to school violence. Understanding patterns of referral data, especially in the context of developmental level allows schools to target response and improvement efforts to specific needs. (Kaufman et 41 al., 2010) analyzed different ways to disaggregate discipline referral patterns that would be useful to schools as they make programmatic or system-level decisions or plan for specific practices and interventions within the PBIS framework. They determined that grade (developmental level), gender, and racial and ethnic contextual factors were all important considerations for the school to account for in driving decisions. Alternately, other studies have found that discipline referral is not appropriate or rather insufficient for making all decisions within PBIS. Skiba et al. (1997) cautioned that the seriousness of the problem behavior and the severity of the consequence were often disproportionate across the referral patterns. This finding pointed to the need for staff to build shared procedures for addressing and documenting problem behavior. One attempt to address this disproportionality in discipline practices was the development of a comprehensive decision system and web-based application for tracking discipline referrals. The School-Wide Information System or SWIS (May et al., 2013) was developed as part of an early PBIS grant in the 1990s (then called Effective Behavior Support or EBS) to provide pilot schools with effective tools for identifying patterns of discipline and engaging in problem-solving. Once the grant ended, school leaders and PBIS implementers expressed interest in maintaining SWIS for ongoing PBIS implementation efforts. Currently over 8,000 schools have adopted SWIS (PBISApps.org), working closely with a local SWIS facilitator who receives training on use of the SWIS application as well as coaching on procedures for installation and maintenance of school requirements. Schools and SWIS facilitators work to meet a set of fidelity or readiness requirements and agree to maintain those standards over time. The full SWIS Readiness Requirements are provided in Appendix A. Examples of SWIS readiness requirements 42 include the following: 1. administrative support for use of SWIS, 2. aligning the school referral form to SWIS requirements (Appendix B), 3. maintaining a leadership team that regularly meets to analyze and use SWIS data, 4. documenting school procedures for addressing and documenting problem behaviors, and 5. ongoing training, coaching, and readiness “check-ups” from the local SWIS facilitator. Within the PBIS literature, referral patterns in SWIS have been used in elementary and middle schools to examine individual student trajectories and identified as one source of data for identifying a student in need of additional tiers of support, often referred to as universal screening (McIntosh et al., 2010; Predy et al., 2014). Recently, Kittelman et al. (2019) analyzed the count of months when a SWIS user for the school generated one or more report as an indirect measure of intention to “use” discipline referral data for decision making and analyzed relation with the SUBSIST measure for Tier 1 PBIS sustainability. The results indicated that generation of discipline referral reports in SWIS was modestly and statistically significantly correlated with SUBSIST Team Use of Data factor scores across implementation groups. SWIS Reports. The use of discipline referral data is useful for decisions across tiers of PBIS but is commonly associated with Tier 1 or universal prevention systems. SWIS was designed around the principles of Tier 1 and the standardized or core reporting options within the application are tailored to identify system-level patterns across all enrolled 43 students. There are seven core SWIS reports (Figure 12) that are designed to use during monthly PBIS leadership meetings and shared out regularly with members of the school community (e.g., students, staff, families). Core SWIS reports include: 1. Average Referrals Per Day Per Month 2. Referrals by Location 3. Referrals by Problem Behavior 4. Referrals by Time of Day 5. Referrals by Student 6. Referrals by Day of Week 7. Referrals by Grade Additional customized reports of discipline referral patterns may be manually created in SWIS using a tool called “Drill Down” that allows for specific filters to be applied to limit the reports to patterns within a specific context, behavior, or sub-group of students (e.g., grade, gender, ethnic/racial group, intervention participants). For example, if core SWIS reports indicate high levels of referrals in the cafeteria, the Drill Down tool can be used to show the specific problem behaviors that have been referred in the cafeteria for a specified date range (e.g., the month of October, the last six weeks). The Drill Down tool often requires additional training but improves the precision and efficiency of decision making at Tier 1 or serves as supplemental data about advanced tiers of support. Figure 13 provides a snapshot of the SWIS Drill Down tool with filters for a specific date range (i.e., 9/1/20 to 10/30/20) and one type of behavior (i.e., Harassment) of interest. 44 Figure 12 School-Wide Information System (SWIS) Core Reports Note. From the PBISApps demonstration website for SWIS. This image includes seven examples of core reports in SWIS including average referrals per day per month, by location, by problem behavior, by time, by student, by grade, and by day of week. All reports are bar charts to allow for simple comparison across categories (or students). 45 Figure 13 School-Wide Information System (SWIS) Drill Down Note. From the PBISApps demonstration website for SWIS. This image demonstrates the functionality available in the SWIS Drill Down tool. At the left are data filters (e.g., problem behaviors, location) to choose from. At the top of the Drill Down tool are two boxes where filters can be added to either Include or Exclude in the dataset. At the bottom right is the report section showing results of the filtered dataset. A drop-down menu allows the dataset to be formatted in specific bar charts by type (location, problem behavior, gender). A summary table shows the number of referrals, number of students, and number of referring staff represented in the current dataset. At the bottom is a large table of individual referral information from the dataset. 46 Student Intervention Data. Currently there is no published research on the use of TIPS for teams that monitor advanced tiers of PBIS and limited guidance on procedures for using data at advanced tiers to make system-level PBIS decisions across interventions. However, the handbook for the Tier 2 Check-In Check-Out intervention (CICO; Hawken et al., 2021) identifies student point data from the daily progress report as a key source of data to monitor intervention outcomes. Many Tier 2 and Tier 3 interventions include the use of daily progress reports and similar direct behavior rating scales that may be used for progress monitoring of individual interventions (Daniels et al., 2017; Fabiano et al., 2017). Given the popularity of the CICO intervention, the developers of the SWIS application created a separate decision system and online web application to monitor student outcomes on this and similar interventions that are based on a standardized daily progress report. The Check-In Check-Out School-Wide Information System (CICO-SWIS; May et al., 2008) monitors the daily progress report (point card) data using a 3-point (0-2) Likert-type scale across 5-10 periods of a day. Report available in CICO-SWIS include graphs and tables that support decisions about the intervention systems as well as individual students. These CICO-SWIS graphs are used during twice monthly reviews by the team or coordinator and present patterns of student intervention data including: 1. Number of days that students participated (i.e., received points), 2. Number of days when the daily point goal (e.g., 75%) was met by one or all students, and 3. Average daily points earned across students or days or specific periods of a day. 47 CICO-SWIS Reports. Figure 14 provides an example of a CICO-SWIS school- wide report that organizes student point data across students into a daily whisker graph. The vertical line represents a range of the proportion of points that all participating students earned. The small horizontal dash or line represents the average score across students. Finally, the number at the top of the vertical line indicates the number of participating students for the given date. School decision-making teams use this report to determine the extent that students identified to participate are consistently using the daily progress report and the overall patterns of points received across students over time. Figure 15 provides another example that provides an average of points earned by all students who participated in the intervention over a specified date range (e.g., Nov. 30, 2020 – Dec. 18, 2020). This provides another representation of patterns of points earned across students and identifies whether individuals or groups of students were more or less likely to the school-wide goal (e.g., 80%) consistently for that date range. Three additional student-level CICO-SWIS reporting options are available to analyze: 1. day-by-day participation and percent of overall points earned across periods and behavioral expectations, 2. average percent of points earned by period of the day over a specified date range, and 3. day-by-day participation and percent of points earned across behavioral expectations for a specified period of the day. 48 Figure 14 Check-In Check-Out School-Wide Information System (CICO-SWIS) School-Wide Report Note. From the PBISApps demonstration website for CICO-SWIS. This image provides a sample School-Wide Report. This whisker graph shows day-by-day the number of contributing students for a given date along with the range of daily percent of points earned across students and the average score across all participating students for the day. The graph shows up to 30 days and indicates days where no data were entered (e.g., no school, no entry). This School-Wide report indicates that from October 19, 2020 to November 27, 2020 between 23 and 40 students participated in the intervention each day with high variability in the range of daily percent of points earned. Average daily scores across participating students ranging from 60% to 80% of points. Decision makers may want to further investigate the variability in both participation and points earned across students. 49 Figure 15 Check-In Check-Out School-Wide Information System (CICO-SWIS) Average Daily Points by Student Report Note. From the PBISApps demonstration website for CICO-SWIS. This image provides a sample Average Daily Points by Student Report. This bar graph shows the average scores for each participating student in a specified date range (Nov 20 to Dec 18). For each student, the average score and number of participating days (i.e., days that points were entered into CICO-SWIS) are provided to monitor both student participation and responsiveness to the intervention. Adoption of CICO-SWIS mimics that of the original SWIS component with a local, certified facilitator trained in the functionality and procedures for CICO-SWIS (and often the CICO intervention as well). The school and CICO-SWIS facilitator collaborate to meet a set of readiness requirements that highlight indicators of fidelity to Tier 2 systems of support and then the facilitator completes the licensing and subscription procedures to set 50 up the CICO-SWIS application for the school. Appendix C provides the list of readiness requirements that schools/facilitators agree to meet and maintain as part of implementing the data system. These requirements include similar items to the SWIS requirements related to administrative support, teaming, and documented procedures that personnel will follow in implementing both Tier 2 interventions (usually CICO) and the CICO-SWIS application. Theory of Data Use Based on the improvement, implementation and PBIS literature, a separate theoretical model was developed on team use of data across tiers within PBIS implementation teams (Figure 16). The model was developed to guide the present set of analyses and starts by identifying three factors that impact team use of data by PBIS implementation teams. Organization factors or drivers specific to PBIS implementation team use of data include the mission and values of the school including the multi-tiered systems approach to academic and behavioral supports, the communication procedures, and external (e.g., community, district) influencers to resource allocation and decision making. At the top center is teaming, the specific foundations and procedures adopted by the PBIS Implementation Team including membership (i.e., stakeholder representation), foundational practices (e.g., minute taking, time allocation), roles (e.g., facilitator, data analyst), and specifically the model used for iterative improvement cycles. PBIS Implementation Team(s) and sub-committees in the school often focus on a specific tier or even specific practices. These specific target practice(s) within the purview of the team will also impact the use of data for example the alignment of the practice to the needs off the school, the fidelity and contextual fit of the core elements for the practice, the school’s approach to dissemination (implementation phases), and the manager(s) that coordinate the 51 target practice(s) for example the PBIS Coach or the Check-In Check-Out (CICO) Coordinator. Figure 16 Theoretical Model of PBIS Implementation Team Data Use Note. Theoretical model for PBIS School Team Data use. This model identifies five factors that impact the effectiveness of school teams in using data for decision making. At the top are three factors that influence teaming procedures including organizational factors (e.g., school mission and values), PBIS implementation team (e.g., membership, roles), and PBIS Practice(s) of Interest (e.g., purpose, core components). At the center is the availability and use of a Tiered Data System or the measures and systems adopted to collect and organize PBIS data. At the bottom is the final factor, a Tiered Decision System that includes the systems and procedures for analyzing, sharing, problem-solving, making decisions, and action planning. 52 Tiered Data System Given the influence of organizational, team, and practice factors, the theoretical model divides use of data into two components: a tiered data system, and a tiered decision system. A tiered data system describes the organization of evaluation measures and related procedures that are adopted by the team to monitor (a) outcomes for example discipline referrals or student point data (e.g., CICO points), (b) fidelity such as the School-Wide Evaluation Tool or Tiered Fidelity Inventory. Additional measures of sustainability and capacity may also be used by the team, for example the SUBSIST and ALTITUDE to measure PBIS sustainability across tiers or indicators that the school has the system capacity to support the intended number of students. Just as with practices, each measure should serve a specific purpose and provide valid and reliable information to the team about overall PBIS implementation or specific tiers and practices within the PBIS framework. Adopting separate measures for each tier is important but insufficient to meet this goal, the team also needs a strong system for collecting and organizing data. Pre- packaged data systems such as the those offered by PBISApps (e.g., SWIS, CICO-SWIS) provide not only the technology to store and organize data but also recommended procedures for staff procedures that will maintain the integrity of the data from start to finish. Other data systems may use simpler technologies such as spreadsheets, but local procedures must be established to ensure data integrity. Tiered Decision System. With the data systems firmly in place, the final component is to build a system of decision routines and procedures to support team use of the data collected. These guidelines establish specific procedures for using the collection of outcome, fidelity, sustainability, 53 and capacity data to drive improvement and implementation decisions, especially those tied to resources such as staff time and effort or financial costs. Decision rules describe a set of pre-determined indicators used by the team for analyzing and sharing data with stakeholders in the organization. For example, schools often identify several potential indicators that despite high fidelity of current supports, an individual student may need additional behavioral intervention(s) including discipline referrals (e.g., 2+ referrals in a 6- week period), attendance (3+ unexcused absences in a single quarter), or intervention- specific data (e.g., below CICO point goal 8 out of 10 days). The TIPS model encourages each team to identify a data analyst who takes the lead in managing these tasks and becoming the local expert on the data system(s) and decision guidelines used by the team. This data analyst often follows a schedule for sharing a set number of reports that are routinely (e.g., monthly, quarterly, annually) used by the team but also uses their knowledge of the data to look for additional reports and information that may be timely to share with the team. Sharing of data outside the team is also an important procedure to build so that all stakeholders have access to the information relevant to their role within the organization and can build a shared understanding of PBIS or specific practices. When useful information has been shared with the team, the next focus is on systematically using it to identify and solve problems related to fidelity, outcomes, sustainability, or capacity and to maintain a set of decision rules that will allow the team to efficiently identify the direction of problem-solving (e.g., staff training, identify adaptations to the practice). The final element of the decision rules used by teams is building an action plan that translates solutions and decisions into actionable steps with procedures to follow up on the fidelity and outcomes of those decisions as they relate to the original problem, the practice of 54 interest, the PBIS framework, and the school as a whole organization. Summary Across the literature for improvement science, implementation science, and the Positive Behavioral Interventions and Supports framework, the systematic and iterative use of data is identified as an important feature that links initial and sustained fidelity to core elements of an innovation. Research on school team use of data within PBIS implementation is limited and lacking in detail that would allow for the quality or frequency of data use to be evaluated at large scales. One potential indicator of school data use at Tier 1 is the frequency that reports about discipline patterns are generated. While indirect, discipline referral patterns have been cited as one outcome of implementing the PBIS framework and examined as a source of data available to school teams for monitoring school climate and student behavioral patterns. Initial efforts to use access or generation of reports in SWIS resulted in a modest but statistically significant relation to PBIS Tier 1 sustainability, specifically the Team Use of Data factor on the SUBSIST measure. Indicators of data use at advanced tiers at the school level are nearly nonexistent within the existing PBIS literature. Given the use of student daily progress reports or point data earned within many Tier 2 interventions, use of student point data and generation of reports about student point data are a measure worth exploring further. The generation of reports about student outcomes (discipline patterns, student points) is not likely to capture the full construct of data-based decision making with sufficient sensitivity but adopting a data system and accessing reports may sufficiently capture intention to use data by school teams and leaders. This may serve as a small step toward identifying simple, indirect indicators of data use that are efficient for researchers and district or state leadership teams 55 to examine across many schools in the effort to monitor and study data use. Study Purpose and Research Questions This study proposed to expand the current body of evidence related to sustainability of PBIS in schools by analyzing sustainability scores, fidelity scores, and direct measures of data use patterns across tiers. The results of this study may contribute to improved understanding of overall sustainability of PBIS at each tier and the relation between sustainability, fidelity, and use of data for decision making. Studying two specific examples of data used by implementation teams may improve understanding of how data are used by schools to improve implementation and adaptation of the core elements of the PBIS framework as well as specific social, emotional, and behavioral practices. The hope is that this information will prompt further attention to improvement of data systems that inform implementation and outcomes of evidence-based practices in schools. Additionally, the study targets contribute to validation of two measures of PBIS sustainability (SUBSIST, ALTITUDE). Five research questions were considered: 1. To what extent is PBIS Tier 1 fidelity related to factors predicting sustainability of PBIS at Tier 1 (as measured by the SUBSIST)? RQ1 Hypothesis: There will be a positive relation between Tier 1 fidelity across three years and Tier 1 sustainability scores in year 3. 2. To what extent is PBIS Tier 1, Tier 2 and Tier 3 fidelity related to factors predicting sustainability of PBIS at advanced tiers (as measured by the ALTITUDE)? RQ2 Hypothesis: There will be a positive relation between Tier 2 and Tier 3 fidelity across three years and advanced tier sustainability scores in year 3. 3. To what extent are factors of sustainability of PBIS at Tier 1 (as measured by the 56 SUBSIST) related to factors of sustainability of PBIS at advanced tiers (as measured by the ALTITUDE)? RQ3 Hypothesis: There will be a positive relation between factors of Tier 1 sustainability and factors of advanced tier sustainability in year 3. 4. To what extent is access of progress monitoring data about Tier 1 student behavior patterns related to (a) fidelity of PBIS at Tier 1 and (b) sustainability of PBIS at Tier 1 (as measured by the SUBSIST)? RQ4 Hypothesis: There will be modest positive relations between the use of Tier 1 data related to student behavior patterns and both Tier 1 fidelity and Tier 1 sustainability, especially for the factor related to data use. 5. To what extent is access to student behavior data about advanced tier interventions related to (a) fidelity of PBIS at Tiers 2 and 3 and (b) sustainability of PBIS at advanced tiers (as measured by the ALTITUDE)? RQ5 Hypothesis: There will be modest positive relations between the use of advanced tier data related to student intervention scores and both fidelity and sustainability of advanced tiers, especially sustainability at Tier 2. 57 CHAPTER II METHODS Settings and Participants This study examined extant data from a sample of 656 U.S. schools implementing the PBIS framework over three consecutive years (Year 1, 2016-17; Year 2, 2017-18; Year 3, 2018-19), based on an existing sample of schools from a separate study. Data sets provided self-reported metrics of fidelity, factors predicting sustainability, and use of a data system designed to support PBIS implementation. During the 2018-2019 school year, one participant from each school (e.g., internal PBIS leader, external PBIS coach) was invited to participate in a separate longitudinal study relating to sustainability of PBIS across tiers of support by completing one or more surveys of PBIS sustainability. Completion of this survey became Year 1 of the longitudinal study and Year 3 for the present study. Most respondents were PBIS team leaders, facilitators, or internal coaches (n = 337, 51%), followed by school administrators (n = 168, 26%), school faculty or staff members (n = 81, 12%), and external, district, or regional coaches (n = 56, 9%). Additional respondent roles included PBIS team members and “other roles” (n = 14, 2%). Proportions of school representatives were similar across school types (elementary, secondary, other). Demographic data for each school were obtained from the National Center for Educational Statistics (NCES) databased. The data were collected by NCES in year 1 of the study (2016-2017) and were collected by the investigators managing the separate, longitudinal study during the 2018-2019 school year. The 645 schools were from 23 U.S. states across the Midwest (n = 262, 40%), West (n = 182, 28%), South (n = 149, 23%), and Northeast (n = 63, 10%) regions. Over half of the schools were elementary (n = 394, 58 60%), and most were located in suburbs (n = 234, 36%) or cities (n = 222, 34%). Additional NCES demographic information is provided in Table 2.1. Of the 656 schools in the sample, 634 (97%) reported at least one year of PBIS implementation fidelity data in the PBIS Assessment data system (Bragg et al., 2015). Using the first year that PBIS fidelity data were reported, we can estimate the school’s earliest history of PBIS implementation efforts. Across the sample, 487 schools (74%) reported implementing PBIS features prior to the first year of the study (2016-2017). Of these, 115 schools (18%) had first reported PBIS implementation in 2008-2009. There were 147 schools (22%) that first reported PBIS implementation during one of the three years included in the study with 32 schools (5% of schools) first reporting PBIS implementation in the final year of the study (2018-2019). The mean number of years since first reporting PBIS implementation fidelity was 5.89 (SD = 4.14). Measures Data from several instruments were collected and analyzed to explore the constructs of sustainability, fidelity, and use of data by schools implementing the PBIS framework during the 2016-2017, 2017-2018, and 2018-2019 school years. A total of five measures were used to examine these constructs. One measure of fidelity of PBIS implementation across tiers was collected per school per year (consolidation of scores from the School- Wide Evaluation Tool, Tiered Fidelity Inventory, or Benchmarks of Quality); two measures of sustainability factors (SUBSIST and ALTITUDE) were collected only in year 3; and two measures of data use (Schoolwide Information System: SWIS; and Check-in Check-out SWIS) were collected per school per year. 59 Table 2.1 School Demographics Characteristics Elementary Secondary Other Total Number (%) of schools 394 (60) 154 (24) 106 (16) 656 Number of U.S. states 21 18 13 23 Number of districts/regions/counties 191 103 69 262 Student enrollment, M (SD) 504 (197) 925 (516) 565 (364) 612 (369) % Non-white students, M (SD) 50 (29) 48 (28) 62 (33) 51 (30) % Students receiving FARMs, M (SD) 61 (26) 52 (25) 69 (27) 60 (26) School region % Northeast 9 8 16 10 % Midwest 41 36 43 40 % South 23 24 20 23 % West 28 32 21 28 School locale % Schools in cities 33 28 45 34 % Schools in suburbs 38 42 17 36 % Schools in towns 11 14 11 12 % Schools in rural areas 16 11 17 15 Note. Schools were included if they met inclusion for one or more research question. FARMs = free and reduced-price meals. Missing data for School Locale variable was between 2% to 9% across sub-samples. 60 PBIS Fidelity Self-reporting PBIS fidelity at the tiers being implemented is a common practice among PBIS schools, primarily for monitoring and planning of implementation activities. Given the evolution of PBIS fidelity measures over the past 20 years, schools may select from several Tier 1 measures and a small number of advanced tier measures of fidelity. To examine the extent that schools implemented core elements of the PBIS framework with fidelity, fidelity scores across tiers for the three years of the study were analyzed using the three measures most frequently reported, the School-Wide Evaluation Tool (SET; Sugai et al., 2005), the Tiered Fidelity Inventory (TFI-Tier 1, TFI-Tier 2, and TFI-Tier 3; Algozzine et al., 2014), and the Benchmarks of Quality (BoQ; Kincaid et al., 2005). Fidelity scores were obtained from the free, online data system called PBIS Assessment (Bragg et al., 2015) that is offered through an organization called PBISApps that is based at the University of Oregon and provides both free and subscription-based data systems related to PBIS implementation. Several valid and reliable measures of PBIS fidelity, especially for Tier 1 PBIS are offered through PBIS Assessment, and schools may choose to complete one or more measures that aligns with the specific needs of the school’s PBIS implementation team. Some schools are required by district leadership, policies, or grant- funding agencies to report specific fidelity measures. Given the complexity of decisions about which fidelity tier to administer, the three measures most reported across the sample were analyzed. These measures included Tier 1 PBIS fidelity items and one measure included separate components or subscales for each of the three PBIS tiers. Table 2.2 presents the descriptive statistics for schools in the sample that reported PBIS fidelity for one or more tiers. 61 Table 2.2 Descriptive Statistics for PBIS Fidelity Across Tiers for Three Years for All School Types Variables N M SD Year 3 (2018-2019) Tier 1 Fidelity 553 .84 .17 Tier 2 Fidelity 452 .77 .23 Tier 3 Fidelity 292 .68 .28 Year 2 (2017-2018) a Tier 1 Fidelity 394 .81 .21 Tier 2 Fidelity 252 .75 .25 Tier 3 Fidelity 125 .63 .31 Year 1 (2016-2017) b Tier 1 Fidelity 316 .80 .22 Tier 2 Fidelity 149 .77 .26 Tier 3 Fidelity 72 .67 .30 Note. Fidelity scores range from .00 to 1.00. Schools were included if they met inclusion for one or more research question. a Year 2 fidelity scores for schools that reported PBIS Tier 1 fidelity Year 2 and Year 3. b Year 1 fidelity scores for schools that reported PBIS Tier 1 fidelity for all three years of the study. School-Wide Evaluation Tool (SET) The School-wide Evaluation Tool (SET; Sugai et al., 2005) was used as the primary measure of fidelity to Tier 1 of the PBIS framework by 54 schools (8%) in year 3 of the 62 study (2018-2019). The SET is a research tool, designed to be conducted by a trained external reviewer to objectively evaluate fidelity to core features of the PBIS framework at Tier 1 based on observations during a school visit and interviews with administrators, staff, and students in the building. The evaluator or any individual in the district or school with permission to access the school’s PBIS Assessment account may be assigned to submit the final SET scores or generate reports to share with school stakeholders (e.g., PBIS team, all staff). The SET is one of the earliest measures of PBIS implementation fidelity at Tier 1 that was originally developed as a research tool but is also used in some regions as the primary Tier 1 annual evaluation. The SET has strong psychometric properties (Horner et al., 2004), including internal consistency (.96), interrater reliability (98.4% - 100%), test-retest reliability (89.9% – 100%). The 28 survey items of the SET are provided in Appendix D. The items are organized into seven subscales that correspond with core elements of PBIS implementation including having School-Wide expectations defined for staff and students (2 items, α = .64), expectations taught explicitly to staff and students (5 items, α = .92), a reward system in place to reinforce expectations (3 items, α = .78), a violations system in place to redirect and correct problem behavior (3 items, α = .63), a monitoring and evaluation system to drive decisions (4 items, α = .85), a management system to coordinate PBIS efforts across the school (8 items, α = .91), and support from the district in terms of policies, training, and data collection (2 items, α = .56). Each SET item includes an evaluation question is rated on a three-point Likert-type scale (0-2) with specific criterion and data sources (e.g., permanent product, interview question) used to rate each item. The proposed and widely adopted fidelity criterion for adequate implementation using the SET is 80% overall score 63 and 80% score on school-wide expectations taught. Tiered Fidelity Inventory (TFI) The second measure of Tier 1 fidelity, and the only measure of advanced tiers of PBIS fidelity, was the Tiered Fidelity Inventory (TFI; Algozzine et al., 2014). The TFI is a self-assessment measure for each of the PBIS tiers (Tier 1, Tier 2, Tier 3). School PBIS leadership teams are encouraged to meet with an external (e.g., district) PBIS coach to collaboratively discuss and agree upon final scores for individual items, although the team may choose to conduct the TFI internally without a coach. Any individual in the district or school with permission to access the school’s PBIS Assessment account may be assigned to submit the final TFI scores or generate reports to share with school stakeholders (e.g., PBIS team, all staff). The TFI (Appendix E) has strong psychometric properties for content validity including 91% - 93% reliability agreement, 96% for item validity, 95% for factor structure, and 89% for item scoring. The TFI also has strong internal consistency across tiers (α = .87 to .98) and overall (.96), test-retest reliability (r = .99), interrater reliability (r = .99 across raters, tiers, and items), and factor structure (Massar et al., 2017; McIntosh et al., 2017). TFI-Tier 1. The tier section of the TFI (TFI-Tier 1) scale consists of 15 items and three subscales (school teams, implementation, evaluation) with a recommended fidelity criterion of 70%. For purposes of this study, the school’s last TFI Tier 1 scale submission of the year that was completed with the guidance of an external PBIS coach were analyzed. If there was not a submission that was completed with a coach, the final submission of the year was analyzed. The TFI-Tier 1 was the reported Tier 1 fidelity measure for 232 schools (35%) in year 3 of the study (2018-2019) 64 TFI-Tier 2. The Tier 2 section of the TFI (TFI-Tier 2) were analyzed to measure fidelity of PBIS at Tier 2. With a structure and procedures identical to the TFI-Tier 1, the Tier 2 scale consists of 13 items and three subscales (teams, interventions, evaluation) with a recommended fidelity criterion of 70%. For purposes of this study, the school’s last TFI Tier 2 scale submission of the year that was completed with the guidance of an external PBIS coach were analyzed. If there was not a submission that was completed with a coach, the final submission of the year was analyzed. TFI-Tier 3. The Tier 3 section of the Tiered Fidelity Inventory (TFI-Tier 3) were analyzed to measure fidelity of PBIS at Tier 3. With a structure and procedures identical to the TFI-Tier 1 and TFI-Tier 2, the Tier 3 scale consists of 17 items and four subscales (teams, resources, support plans, evaluation) with a recommended fidelity criterion of 70%. For purposes of this study, the school’s last TFI Tier 3 scale submission of the year that was completed with the guidance of an external PBIS coach were analyzed. If there was not a submission that was completed with a coach, the final submission of the year was analyzed. Benchmarks of Quality (BoQ) For schools that did not submit either the SET or a TFI, Benchmarks of Quality (BoQ; Kincaid et al., 2005) scores were analyzed. The BoQ was the reported measure of Tier 1 PBIS fidelity for 74 schools (11%) in year 3 of the study (2018-2019). School PBIS leadership teams are encouraged to meet with an external (e.g., district) PBIS coach to collaboratively discuss and agree upon final scores for individual items, although the team may choose to conduct the BoQ internally without a coach. Any individual in the district or school with permission to access the school’s PBIS Assessment account may be assigned 65 to submit the final BoQ scores or generate reports to share with school stakeholders (e.g., PBIS team, all staff).The BoQ (Appendix F) has strong internal consistency (α = .96), test- retest reliability (r = .94), interrater reliability (r = .97), and concurrent validity (r = .51) with the SET (Childs et al., 2016; Cohen et al., 2007). The BoQ is a 53-item measure with 10 subscales including PBIS Team (4 items, α = .43), Faculty Commitment (4 items, α = .75), Discipline Procedures (3 items, α = .81), Data Analysis (5 items, α = .74), Expectations Developed (5 items, α = .76), Rewards Program (8 items, α = .87), Lesson Plans (6 items, α = .87), Implementation Plan (7 items, α = .79), Crisis Plan (3 items, α = .83), and Evaluation (5 items, α = .83). PBIS Fidelity Across Tiers The raw scores across the SET, TFI, and BoQ measures were converted to implementation percentage scores (0 to 1.00) using a total ratio of ratings divided by total possible score and were collapsed into a single score for each tier of PBIS. For Tier 1 PBIS fidelity, this required the use of a cascading logic model that prioritized measures in terms of participation from an external evaluator or coach. For Tier 2 and Tier 3, only the TFI scores were analyzed (McIntosh et al., 2013; Nese et al., 2016). To examine the history of PBIS implementation, PBIS fidelity scores at each tier for three years were collected and analyzed (e.g., Tier 1 PBIS fidelity in year 2, Tier 3 PBIS fidelity in year 3). A total of 316 schools reported Tier 1 PBIS fidelity across all three years, and the average score in year 3 was .84 (SD = .17), 149 schools in the sample reported Tier 2 fidelity for all three years and the average score in year 3 was .77 (SD = .23), and 72 schools in the sample reported Tier 3 fidelity for all years and reported an average score of .68 (SD = .28). Scores for all tiers across the three years of the study were near or above than average fidelity scores reported 66 by Kittelman et al (2018) in an evaluation brief that analyzed a national sample of school PBIS fidelity scores on the TFI during the 2016-2017 school year. PBIS Sustainability Tier 1 PBIS Sustainability (SUBSIST) To measure the factors of PBIS sustainability at Tier 1, the School-Wide Universal Behavior Sustainability Index: School Teams (SUBSIST; McIntosh, Doolittle, et al., 2013) overall score and four individual factor scores were analyzed. The SUBSIST (Appendix G) was developed as a research instrument and consists of 39 self-administered items that correspond to two school-level factors and two district-level factors. The School Priority factor includes 20 items, School Team Use of Data includes 11 items, District Priority includes five items, and finally the District Capacity Building factor includes three items. To complete the SUBSIST, a representative of the school (e.g., administrator, PBIS leader) responded via electronic survey using a sliding Likert-type scale with four response anchors including 1 = Not True, 2 = Partially True, 3 = Mostly True, and 4 = Very True. Respondents were also provided with a separate item of “Don’t Know/NA” which were treated as missing data in the analyses. Scores were then computed to provide an overall total ratio score using the sum of total ratings divided by the total possible so that scores range from 0 to 1.0. Factor-level scores were computed using the same procedures for total ratio score of items within specific factors (School Priority, School Team Use of Data, District Priority, District Capacity Building) divided by the total possible score for that factor. The psychometric properties of the SUBSIST have been validated and display strong content validity (.95), interrater reliability (.95), and test-retest reliability (.96). In 67 addition, the SUBSIST overall score strongly predicts PBIS fidelity at Tier 1 with r = .45 to .68 (McIntosh et al., 2011; McIntosh, Mercer, et al., 2013; Mercer et al., 2014). The School Priority factor includes 20 items (α = .94), School Team Use of Data includes 11 items (α = .94), District Priority includes five items (α = .71), and District Capacity Building includes three items (α = .74). Descriptive statistics for the overall and four individual factors of SUBSIST are presented in Table 2.3. A total of 615 school respondents completed at least one item on the SUBSIST measure, but two school respondents completed only items in the school- level factors. The average overall SUBSIST score across the sample was .78 (SD = .19). The average scores for the four factors ranged from .74 to .81 (School Priority (M = .79, SD = .19), School Team Use of Data (M = .81, SD = .20), District Priority (M = .74, SD = .22), and District Capacity Building (M = .77, SD = .24). Table 2.3 Descriptive Statistics for Tier 1 PBIS Sustainability (SUBSIST) in Year 3 (2018-2019) for All School Types Variables N M SD Tier 1 Sust. Overall 615 .78 .19 Tier 1 Sust. Sch. Priority Factor 615 .79 .19 Tier 1 Sust. Sch. Team Use of Data Factor 615 .81 .20 Tier 1 Sust. Dist. Priority Factor 613 .74 .22 Tier 1 Sust. Dist. Capacity Factor 613 .77 .24 68 Advanced Tier PBIS Sustainability (ALTITUDE) The Advanced Level Tier Interventions Treatment Utilization and Durability Evaluation (ALTITUDE; McIntosh, Kittelman, et al., 2018) overall and individual factor scores were analyzed as measures of potential for PBIS advanced tier sustainability. The ALTITUDE (Appendix H) was first introduced in 2018 and consists of 32 items organized into three factors, Tier 2 and Tier 3 General, Tier 2 Specific, and Tier 3 Specific. Like the SUBSIST measure, a representative of the school responds to each item via electronic survey using a sliding Likert-type scale with four response anchors, including 1 = Not True, 2 = Partially True, 3 = Mostly True, and 4 = Very True. Respondents were also provided with a separate item of “Don’t Know/NA” which were treated as missing data in the analyses. Scores are computed to provide an overall total ratio score using the sum of total ratings divided by the total possible so that raw scores were converted to implementation percentage scores from 0 to 1.0 or 0% to 100%. Factor scores were computed using the same procedures for specific survey items within the Tier 2 and Tier 3 General, Tier 2 Specific, and Tier 3 Specific factors. Validation efforts for the ALTITUDE are in early stages compared to the SUBSIST but are promising. Kittelman, Mercer, McIntosh and Nese (2021) analyzed reliability and relations to PBIS fidelity measures. Internal reliability was found to be strong for each of the three factors: Tier 2 and Tier 3 General (14 items, α = .95), Tier 2 Specific (9 items, α = .95), and Tier 3 Specific (9 items, α = .97). Test-retest reliability was also strong across factors: Tier 2 and Tier 3 General (r = .89, p < .001), the Tier 2 Specific (r = .90, p < .001), and the Tier 3 Specific (r = .95, p < .001). The ALTITUDE was also found to correlate with PBIS fidelity across all three PBIS tiers, correlations between the three 69 ALTITUDE factors and Tier 1 fidelity ranged from r = .27 to r = .46 (p < .001). Correlations between the ALTITUDE factors and Tier 2 fidelity included: Tier 2 and Tier 3 General (r = .43, p < .001), Tier 2 Specific (r = .48, p < .001), and Tier 3 Specific (r = .38, p < .001). Correlations between the ALTITUDE factors and Tier 3 fidelity included: Tier 2 and Tier 3 General (r = .34, p < .001), Tier 2 Specific (r = .24, p < .001), and Tier 3 Specific (r = .41, p < .001). Descriptive statistics for the overall and three individual factors of the ALTITUDE are presented in Table 2.4. A total of 631 schools completed at least one item on the Advanced Tier General factor while 618 schools completed one or more item on the Tier 2 Specific factor and only 542 schools completed at least one item on the Tier 3 Specific factor. The average overall ALTITUDE score across the sample was .61 (SD =.21), while average scores varied for the Advanced Tier General (M =.62, SD =.20), Tier 2 Specific (M =.65, SD =.22), and Tier 3 Specific (M =.54, SD =.27) factors. Table 2.4 Descriptive Statistics for Advanced Tier PBIS Sustainability (ALTITUDE) in Year 3 (2018-2019) for All School Types Variables N M SD AdvTier Sust. Overall 631 .61 .21 Adv Tier Sust. Tier 2/3 General Factor 631 .62 .20 Adv Tier Sust. Tier 2 Specific Factor 618 .65 .22 Adv Tier Sust. Tier 3 Specific Factor 542 .54 .27 70 PBIS Data Use To explore simple indicators of the extent that schools used student outcome data for decision making, measures were analyzed from two online data system that monitor student social behavior (e.g., discipline referrals, daily point card data), offered through PBISApps, the same organization that offers PBIS Assessment. PBISApps records each instance of school user generation of reports within the data system and these instances of report generation were analyzed. School-Wide Information System (SWIS) To explore potential indirect and simple measures of school team data use at Tier 1, school user access to reports about student discipline referrals in the School-Wide Information System (SWIS; May et al., 2013) were analyzed. In year 1 of the study, 180 schools entered referral data into SWIS and agreed to share their data for research and evaluation purposes, there was an average of 1.08 referrals per 100 students per day (SD = 1.86), In year 2, 205 schools entered referral data into SWIS and reported an average of 1.25 referrals per 100 students per day (SD = 2.24), Finally, in year 3 of the study, 220 schools entered referral data into SWIS and reported an average of 1.29 referrals per 100 students per day (SD = 2.59). As of March 2021, the PBISApps organization reported that over 9,000 schools subscribed to SWIS in the U.S. and abroad. Two groups or types of reporting options were selected as potential indirect measures of Tier 1 data use. Core reports disaggregate referral data for a specified date range across a set of seven features (i.e., average referrals per day per month, referrals by time, referrals by location, referrals by day of week, referrals by problem behavior, referrals by grade, and referrals by student). These collectively provide an overall indicator of social climate in the 71 school as well as potential contexts or behaviors that may be negatively impacting the social climate and learning. Schools that subscribe to SWIS commit to maintaining a PBIS leadership team that will analyze SWIS reports at least monthly. The core reports are the most likely to be generated, analyzed, and shared during monthly meetings and were analyzed as an indirect measure of data use at Tier 1. Drill down reports are intended to allow users to customize the data filters (e.g.,. cafeteria referrals only, only referrals for classroom defiance from male students) and disaggregate (e.g., show referrals by time of day, show referrals by perceived motivation) referral data to a level of precision that would allow for problem-solving. Drill down reports are most frequently used to (a) determine whether a potential problem identified in the core and additional reports meet the school’s criteria as a problem, (b) precisely identify the problem at a level of precision that the team can efficiently and effectively solve, and (c) assess a previously identified problem or potential problem to determine effectiveness of previous decisions. The generation, analysis, and sharing of drill down reports may vary month-by-month, but each change in filters and type of disaggregation are counted as one event of access. School generation of drill-down reports in SWIS were also analyzed as an indirect measure of data use at Tier 1. ________________ 1 While not analyzed for this study, several additional reports in SWIS provide disaggregation of referral data in a variety of ways that contribute to understanding the core reports and overall health of PBIS and the social climate. These additional reports (e.g., multi-year, staff, racial/ethnic equity, year-end) are often less sensitive to monthly changes and are therefore recommended/likely to be generated, analyzed, and shared less frequently or with only specific audiences. 72 Given the limited research on using generation of reports as an indicator of data use, four metrics were developed and applied first to core SWIS reports and then to drill down reports across the three years of the study. First, the count of months (range 0-12) that the school generated at least one SWIS (core, drill down) report was developed as an indicator of consistent use of data with minimal criteria. Second, the median number of SWIS (11 core, 9.5 drill down) reports was identified and a subsample of schools was developed to include only months when at least the median number of reports (range 0-12) were generated by a school user. This configuration was intended to produce a more stringent criteria of report access. Third, the average number of reports generated per month (out of a 12-month calendar year) was developed as a continuous measure of data use across the year. This configuration was intended to be more sensitive to the frequency of reports during a calendar month. Fourth, the average number of reports per school week (i.e., 36 weeks) was similarly developed to indicate level of use, considering the typical number of weeks that school is in session. Table 2.5 provides descriptive statistics for the metrics of SWIS Core reports that were examined as potential indicators of Tier 1 data use. Table 2.6 provides descriptive statistics for the metrics of SWIS Drill Down reports that were examined as potential indicators of Tier 1 data use. These metrics were examined as potential indicators and finally the average count of core reports generated per month was selected as the measure of PBIS data use at Tier 1 for the final analysis of Tier 1 PBIS data use, fidelity, and sustainability. 73 Table 2.5 Descriptive Statistics for Generation of SWIS Core Reports for Three Years for All School Types Variables N M SD Year 3 (2018-2019) Count Months Generating 1+ Core Reports 298 7.82 2.81 Count Months Generating 11+ Core Reports a 298 4.07 3.03 Avg Count Core Reports Generated Per Month b 298 12.00 12.20 Avg Count Core Reports Generated Per Week c 298 4.00 4.07 Year 2 (2017-2018) Count Months Generating 1+ Core Reports 266 7.70 2.76 Count Months Generating 11+ Core Reports a 266 3.84 3.09 Avg Count Core Reports Generated Per Month b 266 10.89 10.65 Avg Count Core Reports Generated Per Week c 266 3.63 3.55 Year 1 (2016-2017) Count Months Generating 1+ Core Reports 224 7.53 2.84 Count Months Generating 11+ Core Reports a 224 3.88 3.08 Avg Count Core Reports Generated Per Month b 224 10.26 9.37 Avg Count Core Reports Generated Per Week c 224 3.42 3.12 Note. Schools were included if they met inclusion for one or more research question. a Median split across sample for SWIS Drill Down was 9.5 reports per month. b Total reports generated across the year divided by the calendar months (12). c Total reports generated across the year divided by the average number of school weeks (36). 74 Table 2.6 Descriptive Statistics for Generation of SWIS Drill Down Reports for Three Years for All School Types Variables N M SD Year 3 (2018-2019) Count Months Generating 1+ Drill Down Reports 283 7.11 3.35 Count Months Generating 9.5+ Drill Down Reports a 226 4.46 2.93 Avg Count Drill Down Reports Generated Per Month b 283 12.63 20.60 Avg Count Drill Down Reports Generated Per Week c 283 4.21 6.87 Year 2 (2017-2018) Count Months Generating 1+ Drill Down Reports 247 6.60 3.29 Count Months Generating 9.5+ Drill Down Reports a 172 4.34 2.94 Avg Count Drill Down Reports Generated Per Month b 247 9.79 15.28 Avg Count Drill Down Reports Generated Per Week c 247 3.26 5.09 Year 1 (2016-2017) Count Months Generating 1+ Drill Down Reports 206 6.32 3.17 Count Months Generating 9.5+ Drill Down Reports a 146 4.32 2.80 Avg Count Drill Down Reports Generated Per Month b 206 8.77 11.34 Avg Count Drill Down Reports Generated Per Week c 206 2.92 3.78 Note. Schools were included if they met inclusion for one or more research question. a Median split across sample for SWIS Drill Down was 9.5 reports per month. b Total reports generated across the year divided by the calendar months (12). c Total reports generated across the year divided by the average number of school weeks (36). 75 Check-In Check-Out School-Wide Information System (CICO-SWIS) To explore potential indirect measures of school team data use at advanced tiers, school user access to reports about student points earned on a daily progress report in the Check-In Check-Out School-Wide Information System (CICO-SWIS; May et al., 2008) were analyzed. CICO-SWIS was originally developed to align with the CICO intervention but schools anecdotally report using the system for a wide range of interventions that rely on a daily progress report. School users access reports in CICO-SWIS by (a) navigating to a menu of report options, (b) choosing from the five report types, (c) adjusting specific report settings (e.g., date range), and then (d) clicking the “Generate” button. As of March 2021, the PBISApps organization reported that over 3,600 schools subscribed to CICO- SWIS in the U.S. and abroad. In the first year of the study, a total of 72 schools reported student intervention data in CICO-SWIS and agreed to share their data for research and evaluation purposes. In year 1 there were an average of 9.51 students enrolled in CICO- SWIS per month (SD = 8.35), an average of 41.81 days of point data per enrolled student (SD = 24.46), and an average daily score of 78% per student (SD = 23%). In year 2, a total of 97 schools reported student intervention data in CICO-SWIS with an average of 8.80 students enrolled per month (SD = 8.62), an average of 41.43 days of data per student, and an average daily score of 79% of points per student (SD = 22%). In year three of the study, a total of 144 schools reported student intervention data in CICO-SWIS with an average of 9.22 students enrolled per month (SD = 11.44), an average of 39.00 days of data per student (SD = 24.75) and an average daily score of 81% of points earned per student (SD = 19%). According to a recent evaluation brief (Conley et al., 2018), schools using CICO- SWIS identified a median of 5 school-wide or behavioral expectations (range = 3 - 5) and 8 76 opportunities to earn points each day (range = 2 - 22) on their student daily progress report cards. The default school goal is 80% of points (range = 60% - 90%) although CICO-SWIS allows for individual student goals to vary. The median percent of enrolled students that were enrolled into CICO-SWIS was 3% (range = 0% - 29%) and descriptively varied by school size (i.e., total enrollment) and rates of discipline referrals. Core CICO-SWIS reports disaggregate student point data for a specified date range either across all participating students or for individual students. These reports collectively provide an overall indicator of effectiveness of the intervention in supporting students in meeting behavioral expectations. Schools that subscribe to CICO-SWIS commit to maintaining a PBIS or Tier 2 leadership team that will analyze CICO-SWIS reports at least twice monthly. While CICO-SWIS is primarily designed for monitoring student point data from the Tier 2 Check-In Check-Out (CICO) intervention, it may be used for any intervention that similarly relies on a standardized daily progress report (point card). The procedures used to develop 4 metrics of Tier 1 data use were repeated for the CICO-SWIS data to represent data use at advanced tiers. First, the count of months in the year when at least one CICO-SWIS report was generated. Second, the count of months that the median count of CICO-SWIS (10.25) reports were generated. Third, the average number of CICO-SWIS reports generated per month was analyzed. Finally, the average number of reports generated per school week was calculated. Table 2.7 provides descriptive statistics for the metrics of CICO-SWIS reports that were examined as potential indicators of advanced tier data use. All four metrics were examined as potential indicators and the average count of CICO-SWIS reports generated per week was selected as the final indicator of advanced tier data use. 77 Table 2.7 Descriptive Statistics for Generation of CICO-SWIS Reports for Three Years for All School Types Variables N M SD Year 3 (2018-2019) Count Months Generating 1+ CICO Reports 162 5.97 3.17 Count Months Generating 10.25+ CICO Reports a 100 4.55 2.81 Avg Count CICO Reports Generated Per Month 162 10.73 18.53 Avg Count CICO Reports Generated Per Week b 162 3.58 6.18 Year 2 (2017-2018) Count Months Generating 1+ CICO Reports 112 6.61 3.05 Count Months Generating 10.25+ CICO Reports a 89 4.13 2.64 Avg Count CICO Reports Generated Per Month 112 10.86 13.79 Avg Count CICO Reports Generated Per Week b 112 3.62 4.60 Year 1 (2016-2017) Count Months Generating 1+ CICO Reports 88 6.61 3.41 Count Months Generating 10.25+ CICO Reports a 66 4.74 2.95 Avg Count CICO Reports Generated Per Month 88 12.07 15.27 Avg Count CICO Reports Generated Per Week b 88 4.02 5.09 Note. Schools were included if they met inclusion for one or more research question. a Median split across sample for SWIS Drill Down was 11 reports per month. b Total reports generated across the year divided by the average number of school weeks (36). 78 Procedures Extant data examined in this study were collected from three sources over two phases. School demographic data, both measures of PBIS sustainability, and year 3 PBIS fidelity data were collected by an independent research group within the University of Oregon for a separate longitudinal study of PBIS sustainability. Additional fidelity data for the two years prior to the original study and the measures of PBIS data use were requested by the author from the PBISApps organization and were merged with the original study data. Original School Demographic, PBIS Sustainability, and PBIS Fidelity Variables The two measures of PBIS sustainability, the SUBSIST and ALTITUDE were collected separately by an independent research group at the University of Oregon during the 2018-2019 school year using the online survey tool Qualtrics. Surveys were completed by school respondents who self-identified as representatives of the school’s PBIS implementation efforts. Respondents were recruited from state leaders affiliated with the Center on PBIS (Kittelman, Mercer, McIntosh, & Nese, 2021) at the request of the researchers. Demographic data were collected from the NCES database by the PBISApps organization and represent school information from the 2016-2017 school year. PBIS fidelity data were collected in the PBIS Assessment data system. Researchers then requested school demographic and PBIS fidelity data across tiers for the 2018-2019 school year from the PBISApps database and were merged with PBIS sustainability as part of the original longitudinal study by McIntosh, Mercer, Horner, et al. (2018). Data were identifiable by a research ID contained in the PBISApps database. 79 Additional PBIS Fidelity and PBIS Data Use Variables Once original sustainability data were provided by the original research team at the University of Oregon, the author requested a separate dataset from the PBISApps organization for the specific sample of schools in the original study using a research identification number. The additional data included the PBIS fidelity data collected as part of the original study for the two years prior (2016-2017 and 2017-2018) as well as PBIS Data Use variables from the SWIS and CICO-SWIS data systems for all three years of the study. These variables included raw numbers of counts of reports generated for each individual type of report available in the two data systems. Data Cleaning and Preparation Once both sets of data were received, the data were merged using a research identifier as the matching variable. Once merged, the data were cleaned and organized to match the research questions identified for this study. A simplified measure of school type was generated to allow for examination of variations by school type (elementary, secondary, other, all school types). PBIS fidelity data for Tier 1 were consolidated across the three measures (SET, TFI-Tier 1, BoQ) using the cascading logic procedure implemented by Nese et al. (2016) and all fidelity scores were standardized to use the total ratio scores for each tier across all three years. The individual counts of reports were recalculated into the consolidated metrics previously identified for SWIS Core reports, SWIS Drill Down reports, and CICO-SWIS reports including count of months with any reports generated, count of months that the median number of reports were generated, the average number of reports generated per month, and the average number of reports generated per school week. 80 Data Analyses Four types of statistical procedures were used to analyze PBIS sustainability, fidelity, and data use across tiers of support including Spearman’s rho correlations, multiple linear regression, partial correlations, and Kendall’s Tau-U-b correlations. All analyses were completed using the IBM SPSS Statistics for Windows, Version 26.0 (IBM, 2019). A Bonferroni adjustment procedure was applied to several analyses to better control for multiple comparisons. Determination of Bonferroni adjustment use was based on recommendations from the literature for specific analytic procedures (e.g., Spearman’s rho correlations, multiple linear regression). When appropriate, analyses were also conducted to determine the impact of school type and whether it was appropriate to report the aggregated results, provide disaggregated results, or report only results for specific school type(s) with sufficient sample size(s). Research Question 1 To examine the relation between PBIS Tier 1 fidelity and PBIS Tier 1 sustainability for research question one, both correlations and regressions were conducted. RQ1. Spearman’s Rho Correlations Given the non-normality of the fidelity and sustainability scores, Spearman’s rho non-parametric correlations using list-wise deletion were conducted to identify the strength and direction of relation across the different variables (Mukaka, 2012; Onwuegbuzie & Daniel, 1999). Specifically, correlations between (a) PBIS Tier 1 fidelity overall scores from the SET, BoQ, or TFI-T1 in year 1 (2016-2017), year 2 (2017-2018), and year 3 (2018-2019); (b) the overall SUBSIST score from year 3 (2018-2019); and finally, (c) the individual factor scores on the SUBSIST (School Priority, School Team Use of Data, 81 District Priority, District Capacity Building) were analyzed. A Bonferroni adjustment procedure was applied to address concerns of multiple testing (Armstrong, 2014). RQ1. Multiple Linear Regression To further examine relation between PBIS Tier 1 fidelity and PBIS Tier 1 sustainability and to determine whether Tier 1 fidelity scores predicted scores of Tier 1 sustainability (potential to sustain core elements of PBIS over time) on the SUBSIST, multiple linear regressions were conducted. Multiple linear regression allows for examination of a set of predictor variables, in this study PBIS fidelity scores, on a dependent variable, in this study the overall or factor-level PBIS sustainability score (Uyanik & Güler, 2013; Vesey et al., 2011). Specifically, regressions were conducted to determine whether PBIS Tier 1 fidelity overall scores from the SET, BoQ, or TFI-T1 in year 1 (2016-2017), year 2 (2017-2018), and year 3 (2018-2019)would predict overall Tier 1 PBIS sustainability on the SUBSIST measure and what value individual years of fidelity contribute. Research Question 2 To examine the relation between PBIS fidelity across tiers and PBIS sustainability at advanced tiers of support for research question two, both correlations and regressions were conducted. RQ2. Spearman’s Rho Correlations Given the non-normality of the fidelity and sustainability scores, Spearman’s rho non-parametric correlations using list-wise deletion were conducted to identify the strength and direction of relation across the different variables (Mukaka, 2012; Onwuegbuzie & Daniel, 1999). Specifically, correlations between (a) PBIS Tier 1, Tier 2, and Tier 3 fidelity 82 overall scores from the SET, BoQ, or TFI-T1 in year 1 (2016-2017), year 2 (2017-2018), and year 3 (2018-2019); (b) the overall ALTITUDE score from year 3 (2018-2019); and finally, (c) the individual factor scores on the ALTITUDE (Tier 2 and Tier 3 General, Tier 2 Specific, Tier 3 Specific) were analyzed. A Bonferroni adjustment procedure was applied to address concerns of multiple testing (Armstrong, 2014). RQ2. Multiple Linear Regression To further examine relation between PBIS fidelity across tiers and PBIS sustainability at advanced tiers of support and to determine whether PBIS fidelity scores predicted scores of advanced tier sustainability (potential to sustain core elements of PBIS over time) on the ALTITUDE, multiple linear regressions were conducted. Multiple linear regression allowed for examination of relation between a set of predictor variables or PBIS fidelity scores, on a dependent variable, the PBIS sustainability score (Uyanik & Güler, 2013; Vesey et al., 2011). Specifically, regressions were conducted to determine whether PBIS Tier 1, Tier 2, and Tier 3 fidelity overall scores from the SET, BoQ, TFI-T1, TFI-T2, or TFI-T3 in year 1 (2016-2017), year 2 (2017-2018), and year 3 (2018-2019) would predict overall advanced tier PBIS sustainability on the ALTITUDE measure and what value individual years of fidelity contribute. Research Question 3 To examine the relation between PBIS Tier 1 sustainability on the SUBSIST and PBIS sustainability at advanced tiers on the ALTITUDE for research question three, two types of correlations were conducted. RQ3. Spearman’s Rho Correlations Given non-normality in the sustainability scores, Spearman’s rho non-parametric 83 correlations, using list-wise deletion, were conducted to identify the strength and direction of relation across the different variables (Mukaka, 2012; Onwuegbuzie & Daniel, 1999). Specifically, correlations between the (a) overall SUBSIST score from year 3 (2018-2019), (b) individual factor scores on the SUBSIST (School Priority, School Team Use of Data, District Priority, District Capacity Building), (c) overall ALTITUDE score from year 3 (2018-2019), and finally, (d) individual factor scores on the ALTITUDE (Tier 2 and Tier 3 General, Tier 2 Specific, Tier 3 Specific) were analyzed. A Bonferroni adjustment procedure was applied to address concerns of multiple testing (Armstrong, 2014). RQ3. Partial Correlations When results of the Spearman’s rho were analyzed, multicollinearity between the overall scores and factor-level scores across both the SUBSIST and ALITITUDE were detected, and it was determined that nonparametric correlations for some relations between overall and individual variables would be skewed. Further analysis of partial correlations allowed pairs of variables to be examined individually while controlling for related variables that may inflate the correlation coefficient and cause Type I errors in interpretation of results (Aloe & Thompson, 2013). Controlled variables were determined uniquely for each pairing, for example, when examining the relation between the School Priority factor on the SUBSIST measure of Tier 1 PBIS sustainability and the Tier 2 Specific factor on the ALTITUDE measure of advanced tier PBIS sustainability, all other SUBSIST factors (School Team Use of Data, District Priority, District Capacity) and ALTITUDE factors (Advanced Tier General, Tier 3 Specific) scores were controlled. A Bonferroni adjustment procedure was also applied to address concerns of multiple testing (Armstrong, 2014). 84 Research Question 4 To examine the relation between school access of Tier 1 progress monitoring data (SWIS), PBIS Tier 1 fidelity (SET, BoQ, TFI-T1), and Tier 1 sustainability (SUBSIST) for research question four, correlations were conducted. RQ4. Kendall’s Tau-b Correlations For question four, Kendall’s tau-b was selected due to the non-normal distributions, dissimilar ranking (i.e., inclusion of continuous and ordinal variables with different score ranges), and the non-linear relations between the various sustainability, fidelity, and data use variables (O'Gorman & Woolson, 1995; Xu et al., 2013). Specifically, correlations were conducted for (a) patterns of school access to several SWIS reports in year 1 (2016- 2017), year 2 (2017-2018), and year 3 (2018-2019); (b) PBIS Tier 1 fidelity overall scores from the SET, BoQ, or TFI-T1 in year 1 (2016-2017), year 2 (2017-2018), and year 3 (2018-2019); (c) the overall SUBSIST score from year 3 (2018-2019), and finally, (d) the individual factor scores on the SUBSIST (School Priority, School Team Use of Data, District Priority, District Capacity Building) were analyzed. The analyses were conducted in phases. First, Kendall’s tau-b correlations were first conducted between PBIS sustainability at Tier 1 (SUBSIST) overall and by factor, fidelity across tiers for year 3 (2018-2019) only, and eight exploratory metrics of Tier 1 data use using SWIS core and drill down reports for year 3 (2018-2019) only. Using these results, one final measure of data use (average SWIS core reports generated per month) was selected for the subsequent analysis between the SUBSIST overall and School Data Use factor score in year 3 and three years of both PBIS fidelity and data use at Tier 1. The final decisions for selection of data use variables at Tier 1 and advanced tiers, were based on 85 three considerations: (a) larger significance and size (level) of the correlation coefficient; (b) a preference for continuous variables; (c) larger sample size, specifically for the metric with more stringent criteria; and finally (d) alignment with theoretical model and literature on use of data (e.g., monthly Tier 1 school team meetings). Research Question 5 To examine the relation between school access of advanced tier student behavior data from CICO-SWIS, PBIS fidelity across tiers (SET, BoQ, TFI-T1, TFI-T2, TFI-T3), and advanced tier sustainability on the ALTITUDE for research question five, correlations were conducted. RQ5. Kendall’s Tau-b Correlations For question five, Kendall’s tau-b was selected due to the non-normal distributions, dissimilar ranking (i.e., inclusion of continuous and ordinal variables with different score ranges), and the non-linear relations between the various sustainability, fidelity, and data use variables (O'Gorman & Woolson, 1995; Xu et al., 2013). Specifically, correlations were conducted for (a) patterns of school access to several CICO-SWIS reports in year 1 (2016-2017), year 2 (2017-2018), and year 3 (2018-2019); (b) PBIS fidelity across tires overall scores from the SET, BoQ, TFI-T1, TFI-T2, or TFI-T3 in year 1 (2016-2017), year 2 (2017-2018), and year 3 (2018-2019); (c) the overall ALTITUDE score from year 3 (2018-2019), and finally, (d) the individual factor scores on the ALTITUDE (Tier 2 and Tier 3 General, Tier 2 Specific, Tier 3 Specific) were analyzed. The analyses were conducted in phases. First, Kendall’s tau-b correlations were first conducted between PBIS sustainability at advanced tiers (ALTITUDE) overall and by factor, fidelity scores across tiers for year 3 (2018-2019) only, and four exploratory metrics 86 of advanced tier data use using CICO-SWIS reports for year 3 (2018-2019) only. Using these results, one final measure of data use (average count CICO-SWIS reports generated per week) was selected for the subsequent analysis between the SUBSIST overall and School Team Use of Data factor score in year 3 and three years of both PBIS fidelity and data use at Tier 1. The final decisions for selection of data use variables at Tier 1 and advanced tiers, were based on three considerations: (a) larger significance and size (level) of the correlation coefficient; (b) a preference for continuous variables; (c) larger sample size, specifically for the metric with more stringent criteria; and finally (d) alignment with theoretical model and literature on use of data (e.g., twice monthly advanced tier team meetings). 87 CHAPTER III RESULTS To examine the associations between sustainability, fidelity, and data use in PBIS, several correlational analyses and multiple linear regressions were conducted. For research question 1, relations between Tier 1 PBIS sustainability and Tier 1 PBIS fidelity across three years were examined. For research question 2, relations between advanced tier PBIS sustainability and PBIS fidelity for all tiers across three years were examined. For research question 3, relations between factors of Tier 1 sustainability and advanced tier sustainability were examined and reported for both all grade levels as well as the subsample of elementary schools. For research question 4, relations between generation of Tier 1 student outcome reports, fidelity, and sustainability were examined. For research question 5, relations between generation of advanced tier student outcome reports, fidelity, and sustainability were examined. For both research questions 4 and 5 the results varied by grade level and only results for the elementary schools are reported, given the smaller sample sizes for secondary and other types of schools in the sample. Research Question 1. To what extent is PBIS Tier 1 fidelity related to factors predicting sustainability of PBIS at Tier 1 (as measured by the SUBSIST)? To examine relations between the fidelity and sustainability of Tier 1 PBIS implementation, two types of analyses were conducted. First, Spearman’s rank-order correlations were analyzed to identify patterns of relations across the variables. Table 3.1 presents these correlations. All relations between Tier 1 PBIS sustainability using the SUBSIST measure (overall score and four individual factors) and Tier 1 PBIS fidelity scores using the SET, TFI, and BoQ were significantly and positively correlated. 88 Tier 1 PBIS fidelity at year 3 was moderately correlated with factors of Tier 1 sustainability including the overall SUBSIST score (rSpearman = .43, p < .007), the School Priority factor (rSpearman = .43, p < .007), the School Team Use of Data factor (rSpearman = .45, p < .007), the District Priority factor (rSpearman = .34, p < .007), and the District Capacity Building factor (rSpearman = .30, p < .007). Tier 1 PBIS fidelity at year 2 was moderately and positively correlated with the overall Tier 1 sustainability score (rSpearman = .34, p < .007) as well as the two school-level factors of School Priority (rSpearman = .34, p < .007)and School Team Use of Data (rSpearman = .38, p < .007). Correlations were positive but small between year 2 fidelity and district-level factors of Tier 1 sustainability as well as for all year 1 (2016-2017) Tier 1 fidelity and Tier 1 sustainability in year 3 (2018-2019). Multiple linear regressions using PBIS Tier 1 fidelity to predict PBIS Tier 1 sustainability were conducting using each year’s overall Tier 1 fidelity score as a predictor of the SUBSIST Tier 1 PBIS sustainability total ratio score across factors. Initially, the regressions were run for 294 schools of all types (e.g., elementary) that had reported PBIS Tier 1 fidelity across all three years. A Bonferroni adjustment procedure was applied to results to control for Type 1 error due to multiple predictors (Mundfrom et al, 2006). Results are presented in Table 3.2 and indicate that PBIS Tier 1 fidelity scores predict 46% of the variance in PBIS Tier 1 sustainability scores, but that year 3 (2018-2019) Tier 1 PBIS fidelity was the only significant predictor within the model (b = 0.21, t(290) = 7.03, p < .001). Correlations and regressions were further examined by school type and were found to support the original results. 89 Research Question 2. To what extent is PBIS Tier 1, Tier 2 and Tier 3 fidelity related to factors predicting sustainability of PBIS at advanced tiers (as measured by the ALTITUDE)? To examine relations between the fidelity and sustainability of advanced tier PBIS implementation, two types of analyses were conducted. First, Spearman’s rank-order correlations were analyzed to identify patterns of relations across the variables for the 291 schools that reported PBIS implementation fidelity for all three years of the study. Fidelity across tiers for year 3 (2018-2019) were generally significant and moderately-to strongly, positively correlated with overall advanced tier PBIS sustainability as well as the Tier 2 and Tier 3 General and Tier 2 Specific factors. Tier 1 fidelity in year 3 was strongly and positively correlated with the overall ALTITUDE score (rSpearman = .50, p < .002) and Tier 2 and Tier 3 General factor (rSpearman = .56, p < .002) and moderately positively correlated with the Tier 2 Specific factor and Tier 3 Specific factor scores. Tier 2 fidelity in year 3 was strongly and positively correlated with the overall ALTITUDE (rSpearman = .57, p < .002), Tier 2 and Tier 3 General factor (rSpearman = .60, p < .002), and the corresponding Tier 2 Specific factor (rSpearman = .53, p < .002) with moderate correlations with the Tier 3 Specific factor score. Tier 3 fidelity in year 3 of the study (2018-2019) was strongly and positively correlated with the overall ALTITUDE (rSpearman = .58, p < .002), Tier 2 and Tier 3 General factor (rSpearman = .55, p < .002), and corresponding Tier 3 Specific factor (rSpearman = .58, p < .002) scores as well as moderately correlated with the Tier 2 Specific factor score. For year 2 (2017-2018) PBIS fidelity there were moderate positive correlations across the overall and individual factor sustainability scores except for Tier 1 fidelity, 90 which had a small positive correlation with Tier 3 Specific sustainability and Tier 2 fidelity which had a large positive correlation with the overall ALTITUDE score. Tier 1 fidelity in year 2 was strongly and positively correlated with both the overall ALTITUDE (rSpearman = .53, p < .002) and Tier 2 and Tier 3 General factor (rSpearman = .55, p < .002) scores with moderate correlations with tier-specific factors of the ALTITUDE. Tier 2 fidelity in year 2 was strongly correlated with both the overall ALTITUDE (rSpearman = .51, p < .002) and the Tier 2 Specific factor (rSpearman = .50, p < .002) scores with moderate correlations with the Tier 2 and Tier 3 General and Tier 3 Specific scores. Tier 3 fidelity in year 2 was strongly and positively correlated with both the overall ALTITUDE (rSpearman = .52, p < .002) and the Tier 3 Specific factor (rSpearman = .56, p < .002) score. For year 1 (2016-2017), Tier 1 fidelity was strongly, positively correlated with the Tier 2 and Tier 3 General factor (rSpearman = .50, p < .002), and moderately correlated with the overall ALTITUDE and Tier 2 Specific factor score of sustainability. There was a small, positive correlations between Tier 1 fidelity in year 1 with the Tier 3 Specific factor score. Tier 2 fidelity for year 1 results indicated large positive correlations with the Tier 2 Specific factor (rSpearman = .50, p < .002) score and moderate positive correlations with the overall ALTITUDE, Tier 2 and Tier 3 General factor, and Tier 3 Specific factor scores. Tier 3 fidelity for year 1 results indicated moderate positive correlations with the overall ALTITUDE as well as the Tier 2 and Tier 3 General and Tier 2 Specific factor scores and a large positive correlation with the Tier 3 Specific factor score (rSpearman = .53, p < .002). Table 3.3 presents all correlations between PBIS fidelity across tiers for the three years of the study and the year 3 advanced tier PBIS sustainability overall and individual factor scores of the ALTITUDE. 91 Multiple linear regression was then conducted to determine whether PBIS fidelity scores across tiers for all three years predict PBIS advanced tier sustainability for the 68 schools that reported PBIS fidelity across tiers and completed the advanced tier PBIS sustainability measure (ALTITUDE). A Bonferroni adjustment procedure was applied to results to control for Type 1 error due to multiple predictors (Mundfrom, et al, 2006). Results indicated that the model explained 55% of the variance in overall PBIS advanced tier sustainability but there were no statistically significant individual predictors of PBIS fidelity within the model either before or after applying the Bonferroni adjustment procedure. The regression model was revised to include only year 3 PBIS fidelity across tiers to predict overall advanced tier PBIS sustainability. This model, presented in Table 3.4, included 282 schools, and explained 24% of variance with two statistically significant positive predictors including Tier 1 PBIS fidelity for year 3 (b = 0.23, t(278) = 2.82, p = .005) and Tier 2 PBIS fidelity (b = 0.30, t(278) = 4.38, p < .001). Regressions were further examined for the subsample of 174 elementary schools to identify variance by school type and are presented in Table 3.5. For elementary schools, year 3 PBIS fidelity across tiers explained 22% of the variance of overall advanced tier PBIS sustainability, but only Tier 2 PBIS fidelity was a significant, positive predictor (b = 0.28, t(170) = 3.50, p = .001). Research Question 3. To what extent are factors of sustainability of PBIS at Tier 1 (as measured by the SUBSIST) related to factors of sustainability of PBIS at advanced tiers (as measured by the ALTITUDE)? To examine relations between the factors of sustainability for Tier 1 on the SUBSIST and the more recently developed ALTITUDE measure of advanced tier sustainability of PBIS implementation, two types of analyses were conducted. First, 92 Spearman’s rank-order correlations were analyzed to identify patterns of relations across the variables for the 534 schools that completed both the SUBSIST and ALTITUDE surveys in year 3 of the study (2018-2019). Table 3.6 (below the diagonal) presents the nonparametric correlations between overall and individual factors of sustainability across PBIS tiers. All relations between Tier 1 PBIS sustainability (SUBSIST overall and four factor scores) and advanced tier PBIS sustainability (ALTITUDE overall and three factor scores) were statistically significant and positively correlated. The Tier 1 Team Use of Data and Tier 1 District Priority factors of the SUBSIST were moderately positively correlated with the Tier 3 Specific factor of the ALTITUDE. All other relations were strongly positively correlated. Next, partial correlations were conducted to control for the multicollinearity between individual and overall factors. Table 3.6 (above the diagonal) presents the partial correlations. Each individual relation was examined separately while controlling for non- target variables. The overall Tier 1 and advanced Tier scores were not analyzed within the partial correlations since it would not be appropriate to control for individual factors that contribute to the overall score. For example, when analyzing the relation between overall Tier 1 PBIS sustainability on the SUBSIST and Tier 2 Specific factor on the ALTITUDE, I controlled for the Advanced Tier General factor and Tier 3 Specific factor on the ALTITUDE. Appendix I provides the full syntax and results for specific procedures for each correlation. Results of the partial correlations indicate that, when controlling for other advanced tier PBIS sustainability factors, the overall Tier 1 PBIS sustainability score is significantly positively correlated with the Advanced Tier (Tier 2 and Tier 3) General 93 factor (rpartial = .34, p < .001), and the Tier 2 Specific factor (rpartial = .32, p < .001), but was significantly and negatively correlated with the Tier 3 Specific factor (rpartial = -.14, p = .001). The Tier 1 School Priority sustainability factor, when controlling for other individual Tier 1 and advanced tier PBIS sustainability factors, was positively correlated with both the overall advanced tier PBIS sustainability score (rpartial = .29, p < .001) and the Advanced Tier General factor score (rpartial = .28, p < .001) but was not significantly correlated with the Tier 2 Specific factor or Tier 3 Specific factor once the Bonferroni adjustment procedure was applied. The Tier 1 PBIS School Team Use of Data factor, when controlling for other Tier 1 and advanced tier PBIS sustainability factors, was not significantly related to the overall advanced tier PBIS sustainability or any of the individual factors. The Tier 1 PBIS District Priority factor, when controlling for other Tier 1 and advanced tier PBIS sustainability factors, was positively correlated with overall advanced tier PBIS sustainability (rpartial = .18, p < .001), but not with the individual factors (Advanced Tier General, Tier 2 Specific, Tier 3 Specific). The Tier 1 PBIS District Capacity factor was significantly and positively correlated with both the overall advanced tier PBIS sustainability score (rpartial = .14, p = .001) and the Tier 2 Specific factor (rpartial = .18, p < .001) but was not significantly correlated with the Advanced Tier General or Tier 3 Specific factors. Research Question 4. To what extent is access of progress monitoring data about Tier 1 student behavior patterns related to (a) fidelity of PBIS at Tier 1 and (b) sustainability of PBIS at Tier 1 (as measured by the SUBSIST)? To examine the relations between Tier 1 PBIS sustainability, fidelity across three 94 years, and data use (SWIS) across three years, Kendall’s tau-b correlations were analyzed. Kendall’s tau-b was selected due to the non-normal distributions of variables, dissimilar ranking (i.e., inclusion of continuous and ordinal variables with different score ranges), and the non-linear relations between the various sustainability, fidelity, and data use variables. Additionally, the Bonferroni adjustment procedure applied to the previous research questions was not recommended for use with the Kendall’s tau-b correlations within the literature or by methodologists consulted during this study. The analyses were conducted in phases. First correlations were examined between Tier 1 fidelity, sustainability (SUBSIST), and the eight exploratory indirect metrics of Tier 1 data use using school generation of SWIS core and drill down reports for year 3 (2018- 2019) only. Results were analyzed by school type to determine whether variations existed for elementary, secondary, and other school types. There were no significant correlations between Tier 1 access to student outcome data and fidelity or sustainability for secondary schools or for other school types so only elementary schools were reported. Results of the Kendall’s tau-b correlations for the exploratory phase indicated that for elementary schools there were no significant correlations between any of the eight indirect measures of Tier 1 data use metrics and Tier 1 sustainability (overall or by factor). There were significant correlations between all four of the Tier 1 data use metrics using access of SWIS Core reports and fidelity of PBIS implementation, but not with indirect measures using access of SWIS drill down reports. Results indicated there were small significant correlations between both average reports generated per month and average reports per school week with Tier 1 fidelity (n = 162, τb = .16, p = .003) and Tier 2 PBIS fidelity (n = 136, τb = .23, p < .001). The final indirect measure of data use selected was 95 average number of SWIS core reports generated per month based on three considerations: (a) larger significance and size (level) of the correlation coefficient (correlations between Tier 1 fidelity and both average reports per month and per school week were identical); (b) a preference for continuous variables (both average reports per month and per week were continuous, although there were different ranges); (c) larger sample size, specifically for the metric with more stringent criteria (identical for average reports per month and per week); and (d) alignment with literature and theory about data use at Tier 1 (literature and the SWIS readiness requirements indicate Tier 1 school teams meet monthly to review data). Table 3.7 presents the exploratory correlations. Next, a Kendall’s tau-b correlation was run to determine the relation between overall Tier 1 sustainability as well as the School Data Use factor of the SUBSIST in year 3, three years of Tier 1 fidelity, and three years of Tier 1 data use (average core SWIS reports generated per month). There were no significant correlations between Tier 1 sustainability overall or the School Data Use factor and Tier 1 data use. Results indicated there were small positive correlations between the indirect measure of Tier 1 data use in year 3 and Tier 1 fidelity in year 3 (n = 162, rτb = .16, p = .003) and year 2 (n = 143, rτb = .16, p < .007). Additionally, there was a small positive correlation between Tier 1 data use in year 1 and Tier 1 fidelity in year 1 (n = 102, rτb = .24, p = .001). Table 3.8 presents the correlation coefficients between Tier 1 sustainability, Tier 1 fidelity, and Tier 1 data use. Research Question 5. To what extent is access of progress monitoring data about student behavior related to (a) fidelity of PBIS at Tiers 2 and 3 and (b) sustainability of PBIS at advanced tiers (as measured by the ALTITUDE)? To examine the relations between PBIS sustainability at advanced tiers, fidelity for 96 all tiers across three years, and data use at advanced tiers (CICO-SWIS) for three years, correlations were analyzed. Kendall’s tau-b was selected due to the non-normal distributions of variables, dissimilar ranking (i.e., inclusion of continuous and ordinal variables with different score ranges), and the non-linear relations between the various sustainability, fidelity, and data use variables. The analyses were conducted in phases. First, correlations were examined between sustainability (SUBSIST), fidelity for all tiers, and the four exploratory metrics of advanced tier data use from CICO-SWIS reports for year 3 (2018-2019) only. Results were analyzed by school type to determine whether variations existed for elementary, secondary, and other school types. There were no significant correlations between Tier 1 data use and fidelity or sustainability for secondary schools and varying results for the small (ns = 9 - 29) subsample of other school types, so only results for elementary schools were reported. Results of the Kendall’s tau-b correlations for the exploratory phase indicated that for elementary schools, there were no significant correlations between any of the four advanced tier data use metrics and advanced tier sustainability (overall or by factor). There were significant correlations between three of the four Tier 1 data use metrics and fidelity of PBIS implementation, a moderate or medium correlation between the count of months that a school generated at least 10.25 reports (n = 54, rτb = .35, p = .001). There were small, significant, and identical correlations between the count of months that one or more CICO reports were generated and average reports per school week with Tier 2 fidelity (n = 84, rτb = .27, p = .001). The final measure of data use selected was average number of SWIS core reports generated per week (total reports divided by 36 school weeks). The final measure of data use was based on three considerations: (a) larger significance and size (level) of the 97 correlation coefficient (medium correlations between Tier 2 fidelity and count of months where median number of reports were generated, both count of months with any reports and average reports generated per school week yielded small but significant correlations) ; (b) a preference for continuous variables (both average reports per month and per week were continuous, although there were different ranges but only average reports per school week was significantly correlated with Tier 2 fidelity); (c) larger sample size, specifically for the metric with more stringent criteria (only 54 schools included in the count of months where the median number of reports were generated, compared to 84 for all other metrics); and (d) alignment with literature and theory about data use at Tier 1 (literature and the CICO-SWIS readiness requirements indicate Tier 2 school teams meet twice monthly or approximately every two weeks to review data). Table 3.9 presents the exploratory correlations. Next, a Kendall’s tau-b correlation was run to determine the relation between overall advanced tier sustainability as well as the Tier 2 Specific factor (i.e., the variable most closely associated with CICO-SWIS data) of the SUBSIST in year 3, three years of Tier 2 and Tier 3 (advanced tier) fidelity, and three years of advanced tier data use (average CICO reports generated per week). Results indicated there were small, positive correlations between CICO report generation per week in Year 1 of the study with advanced tier sustainability overall (n = 63, rτb = .21, p < .05) and the Tier 2 Specific factor (n = 63, rτb = .20, p < .05) in Year 3 of the study. There were small, positive correlations between the indirect measure of advanced tier data use in Year 3 and Tier 2 fidelity in the same year (Year 3; n = 84, rτb = .21, p < .01), the prior year (Year 2; n = 65, rτb = .28, p < .01), and two years prior (Year 3; n = 45, rτb = .21, p < .05). There were also small, positive 98 correlations between frequency of access to CICO reports in Year 2 (2017-2018) with Tier 2 fidelity in Year 3 (n = 65, τb = .27, p < .01) and Year 2 (n = 55, rτb = .28, p < .01). Finally, there were small, positive correlations between the indirect measure of advanced tier data use in Year 1 with all Tier 2 PBIS fidelity in Year 3 (n = 54, rτb = .29, p < .01), Year 2 (n = 46, rτb = .23, p < .01), and Year 1 (n = 39, rτb = .26, p < .05). Table 3.10 presents the correlation coefficients between advanced tiers of PBIS sustainability, fidelity, and data use. Table 3.1 Correlations (Spearman’s Rho) for Tier 1 PBIS Sustainability and Tier 1 PBIS Fidelity Across Three Years for All School Types Variables 1 2 3 4 5 6 7 1. Tier 1 Sust. Overall Yr 3 - 2. Tier 1 Sust. Sch. Priority Factor Yr 3 .88** - 3. Tier 1 Sust. Sch. Team Use of Data Factor Yr 3 .81** .74** - 4. Tier 1 Sust. Dist. Priority Factor Yr 3 .89** .68** .62** - 5. Tier 1 Sust. Dist. Capacity Factor Yr 3 .85** .67** .62** .66** - 6. Tier 1 Fidelity Yr 3 (2018-2019) .43** .43** .45** .34** .30** - 7. Tier 1 Fidelity Yr 2 (2017-2018) a .34** .34** .38** .28** .23** .57** - 8. Tier 1 Fidelity Yr 1 (2016-2017) b .25** .27** .29** .17** .16** .43** .63** Note. N = 291. a Year 2 fidelity scores for schools that reported PBIS Tier 1 fidelity for at least Year 2 and Year 3. b Year 3 fidelity scores for schools that reported PBIS Tier 1 fidelity for all three years of the study. *p < .05 **p < .007 (Bonferroni adjustment) 99 Table 3.2 Regression Coefficients of Tier 1 PBIS Fidelity on Tier 1 PBIS Sustainability Overall in Year 3 (2018-2019) For All School Types Variable b (SE) SE Constant .30** .06 1. Tier 1 PBIS Fidelity Year 3 (2018-2019) .51** .07 2. Tier 1 PBIS Fidelity Year 2 (2017-2018) .10 .08 3. Tier 1 PBIS Fidelity Year 1 (2016-2017) -.01 .04 R2 .21** Note. N = 294. *p < .05 **p > .017 (Bonferroni adjustment) 100 Table 3.3 Correlations (Spearman’s Rho) for Advanced Tier PBIS Sustainability and Fidelity Across Tiers for Three Years for All Types Variables 1 2 3 4 5 6 7 8 9 10 11 12 1. Adv Tier Sust. Overall Yr 3 (2018-2019) - 2. Adv Tier Sust. Tier 2/3 Gen Factor Yr 3 .88** - 3. Adv Tier Sust. Tier 2 Specific Factor Yr 3 .89** .79** - 4. Adv Tier Sust. Tier 3 Specific Factor Yr 3 .92** .69** .71** - 5. Tier 1 Fidelity Yr 3 (2018-2019) .50** .56** .44** .40** - 6. Tier 2 Fidelity Yr 3 (2018-2019) .57** .60** .53** .43** .65** - 7. Tier 3 Fidelity Yr 3 (2018-2019) .58** .55** .40** .58** .45** .58** - 8. Tier 1 Fidelity Yr 2 (2017-2018) a .53** .55** .47** .43** .65** .58** .59** - 9. Tier 2 Fidelity Yr 2 (2017-2018) a .51** .49** .50** .38** 62** .75** .48** .65** - 10. Tier 3 Fidelity Yr 2 (2017-2018) a .52** .42** .35** .56** .38** .47** .72** .67** .53** - 11. Tier 1 Fidelity Yr 1 (2016-2017) b .42** .50** .39** .29** .56** .53** .46** .56** .42** .41** - 12. Tier 2 Fidelity Yr 1 (2016-2017) b .49** .44** .50** .38** .44** .58** .47** .48** .69** .54** .44** - 13. Tier 3 Fidelity Yr 1 (2016-2017) b .49** .42** .32* .53** .33* .44** .71** .47** .34** .80** .43** .55** Note. N = 65. a Year 2 fidelity scores for schools that reported PBIS Tier 1 fidelity for at least Year 2 and Year 3. b Year 3 fidelity scores for schools that reported PBIS Tier 1 fidelity for all three years of the study. *p < .05. **p < .002 (Bonferroni adjustment) 101 Table 3.4 Regression Coefficients of Year 3 (2018-2019) PBIS Fidelity on Overall Advanced Tier PBIS Sustainability for All School Types Variable b SE Constant .16* .07 Tier 1 Fidelity Year 3 .23** .08 Tier 2 Fidelity Year 3 .30** .07 Tier 3 Fidelity Year 3 .09* .04 R2 .24** Note. N = 282. *p < .05 **p < .017 (Bonferroni adjustment) Table 3.5 Regression Coefficients of Year 3 (2018-2019) PBIS Fidelity in on Overall Advanced Tier PBIS Sustainability for Only Elementary Schools Variable b SE Constant .16 .10 Tier 1 Fidelity Year 3 .30* .13 Tier 2 Fidelity Year 3 .28** .08 Tier 3 Fidelity Year 3 .05 .05 R2 .22** Note. N = 174. *p < .05 **p < .017 (Bonferroni adjustment) 102 Table 3.6 Spearman’s Rho and Partial Correlations for PBIS Sustainability Across Tiers in Year 3 (2018-2019) for All School Types Variables 1 2 3 4 5 6 7 8 9 1. Tier 1 Sust. Overall - .78** .82** .79** .82** .34** .32** -.14** (614) (614) (612) (612) - (537) (537) (537) 2. Tier 1 Sust. Sch. Priority Factor .88** - .62** .27** -.06 .29** .28** .06 -.11 (610) (610) (610) (610) (534) (534) (534) 3. Tier 1 Sust. Sch. Team Use of Data .86** .81** - -.01 .34** .00 .01 .03 -.05 Factor (610) (610) (610) (534) (534) (534) 4. Tier 1 Sust. Dist. Priority Factor .90** .72** .67** - .44** .28** .03 .03 .09* (610) (610) (534) (534) (534) 5. Tier 1 Sust. Dist. Capacity Factor .87** .68** .69** .71** - .14** -.01 .18** -.06 (610) (534) (534) (534) 6. Adv Tier Sust. Overall .71** .68** .61** .62** .59** - - - - 7. Adv Tier Sust. Tier 2/3 General Factor .73** .73** .64** .62** .60** .91** - - .27** (537) 8. Adv Tier Sust. Tier 2 Specific Factor .73** .69** .64** .63** .62** .92** .82** - .42** (537) 9. Adv Tier Sust. Tier 3 Specific Factor .55** .52** .46** .50** .44** .92** .73** .76** - Note. N = 534. Nonparametric zero-order correlations are reported below the diagonal. Partial correlations between ALTIUDE and SUBSIST scores are reported above the diagonal with sample sizes reported in parentheses below correlation coefficients. *p < .05 **p < .004 (Bonferroni adjustment 103 Table 3.7 Correlations (Kendall’s Tau-b) for Tier 1 PBIS Sustainability, Fidelity, and Exploration of Indirect Metrics of Data Use in Year 3 (2018-2019) for Elementary Schools Variables 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 1. Tier 1 PBIS Sust. Overall - 2. Tier 1 Sust. Sch. Priority .68** Factor (374) - 3. Tier 1 Sust. Sch. Team .65** .57** Use of Data Factor (374) (374) - 4. Tier 1 Sust. Dist. Priority .71** .51** .46** Factor (372) (372) (372) - 5. Tier 1 Sust. Dist. .69** .49** .48** .52** Capacity Factor (372) (372) (372) (370) - 6. Tier 1 Fidelity .33** .32** .37** .24** .23** (324) (324) (324) (322) (323) - 7. Tier 2 Fidelity .20** .22** .22** .15** .11* .44** (280) (280) (280) (278) (279) (282) - 8. Tier 3 Fidelity .11* .16** .12* .06 .05 .28** .46** (275) (275) (275) (274) (275) (288) (290) - 9. Count Months Generating -.01 -.04 .08 .01 -.03 .16* .24** .09 1+ Core Reports (159) (159) (159) (157) (159) (162) (136) (72) - -.04 -.02 .01 -.03 -.08 .12 .21** .11 .56** 10. Count Months Generating (138) (138) (138) (136) (138) (141) (120) (67) (152) - 11+ Core Reports (151) (151) (151) (149) (151) (153) (128) (69) (165) (147) (165) (165) (166) (132) (166) 104 Table 3.7 (continued) Variables 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 11. Avg Count Core Reports .02 .01 .07 .02 -.02 .16** .23** .17* .62** .80** Generated Per Mo (159) (159) (159) (157) (159) (162) (136) (72) (177) (152) - 12. Avg Count Core Reports .02 .01 .07 .02 -.02 .16** .23** .17* .62** .80** 1.00** Generated Per Wk (159) (159) (159) (157) (159) (162) (136) (72) (177) (152) (177) - 13. Count Months Generating .07 .01 .11 .10 .04 .09 .08 .02 .34** .19** .29** .29** 1+ Drill Down Reports (151) (151) (151) (149) (151) (153) (128) (69) (165) (147) (165) (165) - 14. Count Months Generating .04 .05 .02 .05 .02 .09 .04 -.03 .26** .19** .21** .21** .62** 11+ Drill Down Reports (119) (119) (119) (117) (119) (123) (102) (54) (132) (124) (132) (132) (132) - 15. Avg Count Drill Down .01 -.03 .05 .05 -.01 .03 .05 -.05 .25** .16* .27** .27** .70** .82** Reports Generated Per (151) (151) (151) (149) (151) (153) (128) (69) (165) (147) (165) (165) (166) (132) - Mo 16. Avg Count Drill Down .01 -.03 .05 .05 -.01 .03 .05 -.05 .25** .16** .27** .27** .70** .82** 1.00** Reports Generated Per Wk Note. Correlation sample sizes are reported in parentheses beneath regression coefficients for each model. *p < .05 **p < .01 105 Table 3.8 Correlations (Kendall’s Tau-b) for Tier 1 PBIS Sustainability in Year 3 (2018-2019), Fidelity Across Three Years, and Tier 1 PBIS Data Use (Average Count of SWIS Core Reports Generated per Month) Across Three Years for Only Elementary Schools Variables 1 2 3 4 5 6 7 1. Tier 1 PBIS Sust. Overall - 2. Tier 1 Sust. Sch. Team Use of .65** Data Factor (374) - 3. Tier 1 Fidelity Year 3 .33** .37** (324) (324) - 4. Tier 1 Fidelity Year 2 .25** .30** .47** - (236) (236) (250) 5. Tier 1 Fidelity Year 1 .14** .19** .30** .47** (198) (198) (208) (208) - 6. Avg Count Core Reports .02 .07 .16** .16** .16* Generated Per Mo Year 3 (159) (159) (162) (143) (115) - 7. Avg Count Core Reports -.01 .04 .13* .10 .12 .43** Generated Per Mo Year 2 (152) (152) (150) (136) (115) (157) - 8. Avg Count Core Reports .06 .11 .16* .16* .24** .41** .50** Generated Per Mo Year 1 (136) (136) (129) (114) (102) (130) (138) Note. Correlation sample sizes are reported in parentheses beneath regression coefficients for each model. *p < .05 **p < .01 106 Table 3.9 Correlations (Kendall’s Tau-b) for Advanced Tier PBIS Sustainability, PBIS Fidelity Across Tiers, and Indirect Metrics of Advanced Tier PBIS Data Use in Year 3 (2018- 2019) for Only Elementary Schools Variables 1 2 3 4 5 6 7 8 9 10 1. Adv Tier Sust. Overall - 2. Adv Tier Sust. Tier 2/3 .74** General Factor (380) - 3. Adv Tier Sust. Tier 2 .76** .64** Specific Factor (375) (375) - 4. Adv Tier Sust. Tier 3 .76** .53** .56** - Specific Factor (329) (329) (328) 5. Tier 1 Fidelity .28** .31** .29** .18** (329) (329) (325) (282) - 6. Tier 2 Fidelity .30** .29** .32** .21** .44** (287) (287) (284) (245) (282) - 7. Tier 3 Fidelity .25** .26* .20** .25** .28** .46** (178) (178) (175) (166) (178) (178) - 8. Count Months Generating .09 .14 .09 .08 .04 .27** .03 1+ CICO Reports (98) (98) (97) (82) (96) (84) (56) - 9. Count Months Generating .12 .13 .18* .12 .10 .35** .17 .53** 10.25+ CICO Reports (65) (65) (64) (53) (62) (54) (36) (67) - 10. Avg Count CICO Reports .08 .11 .08 .03 .04 .21** .03 .69** .76** Generated Per Mo (98) (98) (97) (82) (96) (84) (56) (102) (67) - 11. Avg Count CICO Reports .08 .11 .08 .03 .04 .21** .03 .69** .76** 1.00** Generated Per Wk (98) (98) (97) (82) (96) (84) (56) (102) (67) (102) Note. Correlation sample sizes are reported in parentheses beneath regression coefficients for each model. *p < .05 **p < .01 107 Table 3.10 Correlations (Kendall’s Tau-b) for Advanced Tier PBIS Sustainability in Year 3 (2018- 2019), Advanced Tier PBIS Fidelity Across Three Years, and Advanced Tier PBIS Data Use (Average Count of CICO-SWIS Reports Generated Per Week) Across Three Years for Only Elementary Schools Variables 1 2 3 4 5 6 7 8 9 10 1. Adv Tier Sust. Overall - 2. Adv Tier Sust. Tier 2 .76** - Specific Factor (375) 3. Tier 2 Fidelity Year 3 .30** .32** - (287) (284) 4. Tier 2 Fidelity Year 2 .33** .31** .59** (164) (163) (167) - 5. Tier 2 Fidelity Year 1 .28** .24** .43** .60** (104) (104) (106) (106) - 6. Tier 3 Fidelity Year 3 .25** .20** .46** .37** .35** - (178) (175) (178) (109) (75) 7. Tier 3 Fidelity Year 2 .24** .10 .28** .37** .42** .46** (80) (80) (81) (78) (62) (81) - 8. Tier 3 Fidelity Year 1 .21* .05 .14 .04 .32** .49** .58** (50) (50) (51) (50) (50) (51) (51) - 9. Avg Count CICO Reports .08 .08 .21** .28** .21* .03 -.10 -.25 Generated Per Wk Year 3 (98) (97) (84) (65) (45) (56) (36) (21) - 10. Avg Count CICO Reports .13 .11 .27** .28** .11 .05 -.12 -.13 .48** Generated Per Wk Year 2 (76) (76) (65) (55) (45) (43) (32) (21) (72) - 11. Avg Count CICO Reports .21* .20* .29** .23* .26* .04 -.19 -.13 .55** .66** Generated Per Wk Year 1 (63) (63) (54) (46) (39) (38) (29) (21) (57) (58) Note. Correlation sample sizes are reported in parentheses beneath regression coefficients for each model. *p < .05 **p < .01 108 CHAPTER IV DISCUSSION General Discussion This study proposed to answer five primary research questions. The first two research questions focused on examining associations between measures of PBIS fidelity reported by schools over a three-year period (implementation history) and measures of PBIS sustainability reported during the third year across tiers of support. The third research question focused on examining associations between a measure of PBIS sustainability at Tier 1 and a newly developed measure of PBIS sustainability at advanced tiers (Tier 2, Tier 3). Finally, questions four and five focused on examining potential indirect measures of data use across tiers of PBIS, as well as associations between three years of data use and implementation fidelity with sustainability of PBIS across tiers. Sustainability, fidelity, and data use are constructs identified in literature about school improvement and implementation of evidence-based innovations, including the innovation of focus for this study, the PBIS framework. Research on the fidelity, sustainability (indicators of potential to sustain implementation), and sustained implementation of PBIS is emerging, especially at Tier 1, but this study attempted to examine historical indicators of implementation fidelity and data use that preceded and included the school year that sustainability measures were collected. The construct of data use at a system or school-wide level has yet to be operationally defined in a way that would allow for a direct measure to be established for purposes of evaluation or research. Comprehensive understanding of data use may include procedures for measurement (data) prioritization, frequency of analysis, organization of data (e.g., charts, graphs), procedures 109 for sharing with stakeholders, and many other aspects of the data and decision systems. Improved understanding of data use would allow school PBIS leadership teams, district leaders, and researchers to establish data use guidelines and criteria that will maximize use of resources (e.g., staff time, data system adoption) and improve the efficiency of decision- making about implementation fidelity, outcomes, capacity, and sustainability of PBIS and the specific practices and systems embedded within the framework. PBIS Fidelity and Sustainability For research questions one and two, the original hypothesis was that there will be a positive relation between Tier 1 fidelity across three years and Tier 1 sustainability scores in year 3, as well as a positive relation between Tier 2 and Tier 3 fidelity and advanced tier sustainability. For research question one that focused on these relations for only Tier 1 PBIS implementation, results of the Spearman’s rho correlations indicate that relations between Tier 1 fidelity across the three years of the study were positively associated with the overall and individual factors of Tier 1 PBIS sustainability on the SUBSIST measure. The relations with Tier 1 PBIS sustainability scores were comparatively highest for fidelity in year 3, the concurrent year that sustainability was collected, slightly smaller for fidelity in year 2, and smallest for fidelity in year 1. Correlations between fidelity in year 3 (2018- 2019) and overall SUBSIST scores in year 3 as well as individual sustainability factors (School Priority, School Team Use of Data, District Priority, District Capacity Building) were moderate (rSpearman = .34 - .45). Tier 1 fidelity in year 2 (2017-2018) was moderately positively correlated with the overall and school-level factors of the SUBSIST. All other correlations between year 1 and year 2 fidelity and sustainability, while significant, were small (rSpearman = .16 - .29). Similarly, when multiple linear regression was used to 110 determine whether Tier 1 fidelity across years predicted Tier 1 PBIS sustainability, the model explained 21% of variance in overall SUBSIST scores, but only Tier 1 fidelity in year 3 was a significant predictor within the model. Examining advanced tiers yielded more complex results, given the inclusion of all nine PBIS fidelity variables (three tiers X three years). The results of the initial Spearman’s rho correlations indicated large positive correlations between the overall ALTITUDE score and fidelity across tiers in year 2 and year 3 and moderate correlations between overall ALTITUDE score and fidelity across tiers in year 1. Correlations between individual factors of the ALTITUDE and fidelity were also positive at moderate and large levels. Correlations indicated that fidelity for all three tiers across three years was not a useful model in identifying predictors of advanced tier sustainability overall. For the aggregated sample of schools across types, both Tier 1 and Tier 2 fidelity in year 3 were significant predictors of advanced tier sustainability but when disaggregated by school type the results varied. For elementary schools, which made up most of the sample, only Tier 2 fidelity in year 3 was a significant predictor of advanced tier sustainability. Given these results, the general hypothesis of positive relations between fidelity and sustainability was supported for research questions 1 and 2. However, inclusion of prior years of overall fidelity scores did not provide a clear contribution toward better understanding factors of PBIS sustainability, likely because the variance of fidelity in the final year (Year 3) was most closely related to variance in sustainability for the concurrent (same) year. Relation between fidelity and advanced tier sustainability may require further examination given the strong correlations between tier-specific measures (e.g., Tier 2 Specific sustainability and 111 Tier 2 fidelity across years). PBIS Sustainability For research question three, the original hypothesis was that there would be a positive relation between factors of Tier 1 sustainability and factors of advanced tier sustainability in year 3. Results of the initial Spearman’s rho correlations indicated strong correlations between Tier 1 and advanced tier sustainability of PBIS overall and between the various factors of sustainability. However, due to multicollinearity within factors of the measure, it was appropriate to further analyze the results while controlling for the influence of either overall scores or specific factors within the measures (selected individually for each comparison). The results of the partial correlations revealed that there were moderate correlations between overall Tier 1 sustainability on the SUBSIST and both the Advanced Tier General and Tier 2 Specific factors of the ALTITUDE. Interestingly there was a small significant negative correlation between overall Tier 1 sustainability on the SUBSIST and the Tier 3 Specific factor of the ALTITUDE. There were small positive correlations between the Tier 1 School Priority factor of the SUBSIST and the overall advanced tier sustainability on the ALTITIDUE and the Advanced Tier General factor, but not with the Tier 2 Specific or Tier 3 Specific factors. Curiously, there were no significant correlations between Tier 1 School Team Use of Data on the SUBSIST and advanced tier sustainability (overall or individual factors) on the ALTITUDE. There was a small positive correlation between the District Priority factor of the SUBSIST and the overall advanced tier sustainability score on the ALTITUDE, but no significant correlations with any of the individual factors. Finally, there were two small positive correlations between the District Priority factor of the 112 SUBSIST and both the overall advanced tier sustainability score and the Tier 2 Specific factor on the ALTITUDE. Given these results, the general hypothesis of positive relations between Tier 1 and advanced tier PBIS sustainability was partially supported, but once the use of partial correlations was introduced, only specific relations were significant. Additionally, the small but significant negative correlation between overall Tier 1 sustainability and the Tier 3 Specific factor of advanced tier sustainability bears further examination. PBIS Data Use, Fidelity, and Sustainability For research questions four and five, the original hypothesis was that there would be modest positive associations between indirect measures of data use across tiers, sustainability across tiers, and fidelity across tiers of support. This hypothesis was based on results from previous studies that pointed to school and team use of data as a predictor of PBIS Tier 1 sustainability in schools (McIntosh et al., 2015; McIntosh et al., 2013). This study also sought to expand on the use generation of reports about student discipline as an indirect measure of data use (Kittelman et al., 2019). Results of the exploratory Kendall’s tau-b correlations guided in the selection of the measure of data use at Tier 1 (average SWIS core reports generated per month) and advanced tiers (average CICO-SWIS reports generated per week). The subsequent results analyzing three years of data use, three years of fidelity, and the overall sustainability scores indicated a small positive relation between the indirect measure of Tier 1 data use in year 3 and Tier 1 fidelity in year 2 and year 3, but no relations were found between use of data and sustainability at Tier 1. For advanced tiers of support there were small and moderate correlations found between use of data and Tier 2 fidelity in both year 2 and year 3. Small correlations were also found between generation of 113 advanced tier reports in year 1 with overall and Tier 2 Specific advanced tier sustainability. Given these results, the hypotheses were not clearly supported, at least not at a socially relevant scale of importance. Kittelman et al. (2019) found modest but significant positive correlations between school generation of discipline data reports and the Team Use of Data subscale of the SUBSIST across stages of implementation. The contrasting results of the present study to may be due to the inclusion of additional variables within the analyses, including multiple variations of report generation in the exploration phase and multiple years of the same measure for the second phase. Further replication is needed, but there may be a need for more sophisticated methods to measure school use of data across tiers of the PBIS framework to better represent the nuances of school use of data for decisions related to PBIS fidelity and sustainability. Limitations Several limitations should be considered when interpreting results of this study and conducting future research using the measures of PBIS sustainability, fidelity, and especially data use. Two broad categories of limitations include (a) measurement limitations and (c) sample sizes. Measurement Limitations A major limitation for this study is related to the measures used to represent the constructs of sustainability, fidelity, and data use in PBIS. Several of the measures were self-reports by one or more representatives of the school, so the accuracy of scores is dependent on the competence and honesty of the respondents. The two self-reported measures of PBIS sustainability (SUBSIST, ALTITUDE) were completed by individuals who self-identified as representatives who were knowledgeable about one or more school’s 114 implementation efforts. While the recruitment and survey materials provided instructions on respondent criteria, there was no external validation to confirm that the respondent (a) was knowledgeable about school PBIS implementation efforts and (b) accurately represented those implementation efforts. PBIS fidelity measures were retrieved from the PBIS Assessment data system, which does not monitor the training or competence of the evaluators or accuracy of the data. One of the PBIS fidelity measures (SET) was originally designed as an external evaluation by someone trained in administrating the tool, but the other two measures (TFI, BoQ) are designed to be conducted by the PBIS team and with an external PBIS coach if possible. Instructions and training materials are publicly available via the PBISApps website to guide school use of the fidelity measures. A further limitation of this study’s use of fidelity measure was the use of multiple measures of Tier 1 PBIS fidelity to increase the sample size. While there was a precedence for using a cascading logic to consolidate these measures, the procedure limited the analysis of specific subscales of fidelity that would have been available if only one measure (e.g., TFI) were selected. Given the breadth of the constructs explored in this study (sustainability, fidelity, data use), the increased precision of fidelity subscales may have allowed for expanded analyses of relations with factors of sustainability and data use. The variables to examine PBIS data use were based on reports accessed from two specific data systems, were not self-reports but were highly simplified, distal indicators of a broad and complex construct. A review of literature indicates few efforts to identify simple and direct measures of data use by school PBIS teams. The research on the Team Initiated Problem Solving (TIPS) model measured team activities through observation of team 115 meetings and analysis of permanent meeting products (i.e., meeting minutes) over time (Algozzine et al., 2016). While comprehensive in examining team procedures, the procedures of analysis (aggregation and disaggregation) were not explicitly identified. Additionally, the resources required to examine student team meetings at greater depth are both labor and cost intensive. While attempting to validate findings from an early PBIS sustainability study, Kittelman et al (2019) explored a simpler though less direct measure of school data use, the count of months that reports about discipline referrals in SWIS were generated. This study attempted to expand on this simpler indicator of data use by examining two types of SWIS reports (Core, Drill Down) and CICO-SWIS reports. The generation of reports were organized into four potentially useful formats related to count of months and average reports accessed to determine whether an ideal measure of data use at Tier 1 and advanced tiers (specifically Tier 2) could be identified. The average count of reports generated (per month or week) and the count of months that reports were generated (any report or above the median across schools) indicate only that a school user associated with the school clicked on a report available in SWIS or CICO-SWIS. Without additional measures (e.g., meeting minutes, communications between school staff), it is not possible to verify the quality of training received by the user to select and interpret the reports appropriately or the extent that reports accessed were connected to decision making. Future research should include both measures of access and permanent products or observations of the team actions following the access of reports. Secondly, the use of the CICO-SWIS data system is primarily recommended for standardized tier 2 interventions. While schools anecdotally report using CICO-SWIS for a broader range of interventions, there is no indicator within CICO-SWIS 116 of whether the point card is standardized or individualized. Ideally, this information would be collected for future or research and separate measures of individualized Tier 3 data use will be identified. Within the existing measures it may also be useful to analyze the specific types of reports in SWIS (e.g., referrals by location, referrals by problem behavior) and CICO-SWIS (e.g., average daily points per student) to determine whether patterns of use for specific reports vary or are differentially associated with problem-solving and decision making. For example, what differences exist in decisions made by teams that use a few reports frequently and teams that use many of the standardized and custom reporting options less frequently, possibly on a rotating basis. Future research should also analyze changes in actual data (discipline referral rates, percent of daily points earned) to examine relations between use of data (specific or general) and changes in student outcomes. Sample Size Limitations Another area of limitation of the study is related to sample size. The sample sizes varied for each question, especially questions that analyzed measures of Tier 3 supports and data collected in year 1 (2016-2017) of the study. Further disaggregating results by school type created extremely small sample sizes and, in some cases, made certain analyses (e.g., multiple regressions, correlations for secondary schools only) inappropriate. To compensate for the small sample sizes, research questions three, four, and five included use of pairwise deletion within the analyses to maximize the information across the sample, but further limits interpretation of results since schools with one year of data were compared with schools that reported two or three years of data. Another method employed to retain adequate sample sizes was the consolidation of three separate PBIS fidelity measures (SET, TFI, BoQ) into a single measure of overall fidelity at Tier 1. 117 Implications for Future Research Future research on sustainability, fidelity, and data use is needed to improve our understanding of factors that impact a school’s sustained implementation of PBIS with sufficient fidelity to realize valued academic and social benefits for students. Data Use and Implementation Science The goal of implementation science is to understand the variables that improve speed, efficiency, and durability of the processes used by schools and other organizations to adopt promising and evidence-based innovations for improvement purposes. Results of the present study expose a series of inter-related relations between implementation fidelity, factors of implementation sustainability, and data use by implementation teams. These inter-relations may encourage future research to focus on the design of instruments that comprehensively measure constructs of implementation including fidelity, sustainability, and use of data by leadership teams. One smaller step toward this goal is to examine the functions that data systems (e.g., measures, collection procedures) and decision systems (e.g., sharing, problem identification, solution development, action planning) serve in implementation activities across different fields including social behavior, academic and professional growth, healthcare, business, and social justice. For example, implementation scientists are encouraged to examine relations between variables of data use (e.g., teaming, collection, analysis, sharing) and an organization’s capacity to scale the innovation, maintain adequate fidelity of implementation, monitor adaptations to the innovation, increase social/cultural acceptability of the innovation, or sustain use of the innovation over time (especially when known barriers are present). Identifying these relations with precision will allow developers of new innovations to embed the most critical variables of 118 data use into standard procedures for data use within the implementation protocol. Additionally, researchers will have the information needed to build valid and reliable implementation measures that are robust across common barriers encountered and include the necessary weights or modifications to be useful for any given stage of implementation. School and District Data Use Across Stages of PBIS Implementation PBIS evidence has focused on the school as the unit of analysis, but district influence on school implementation and improvement (outcomes) has been recognized and recommended as a separate or inter-related unit of analysis (George et al., 2018; Sugai & Horner, 2019). Future research on use of data within PBIS schools should also target the shared and unique information needed by different stakeholders at the school and district levels at given stages of implementation. Table 4.1 presents a set of implementation questions that school and district leadership teams should ask at each stage. Measures of implementation and outcomes should be designed to guide teams in answering these question as well as the related decision or action taken by the team in response. While questions may be similar for school and district leadership teams, the measures that answer questions and guide decisions may vary from school-to-school or between school and district. For example, school capacity to implement the CICO intervention across all (and only) students who are likely to benefit may rely on a variety of universal screeners or matching criteria while the district capacity to support school implementation of CICO may focus on number of individuals in the district with the expertise and time to train school staff or coach school teams. Research to develop or improve existing measures should focus on identifying indicators shared across types of decisions and prioritizing indicators that are unique to only one type of decision. 119 Table 4.1 Questions to Guide PBIS Decisions by Implementation Stage Implementation Type of Stage Decision Data Use Question Exploration Readiness What are the immediate and long-term needs we need to meet (or improve) within PBIS? What innovations (e.g., practices, interventions) are available? What is the evidence base? What is the fit of the innovation to the need and context? What core elements of the innovation are in place, partially in place, and not in place? Installation and Fidelity What is our current stage/status of implementation? Initial Prioritization What activities are complete, in process, and ready to start? Implementation Planning What are the next activities to attend to? Sustainability Have past activities/improvements been sustained? Impact What initial impact has implementation had on student social, emotional, and behavioral indicators? Full Fidelity Which core elements are consistently in place? Which Implementation elements may benefit from further attention (e.g., training, adaptation)? Outcomes What distal impact has implementation had on student social, emotional, and behavioral indicators? Contextual Fit Do stakeholder groups generally agree that current PBIS practices are aligned to local values? Are there differences reported across groups? Capacity Is the innovation reaching all (or only) students who are likely to benefit? Is it feasible to increase access if needed? Sustainability Is implementation likely to sustain for at least 5 years, even if common barriers (e.g., turn-over, new initiatives) arise? Adaptation What adaptations have been made to increase fidelity, outcomes, fit, capacity, or sustainability? How have adaptations directly and indirectly impacted fidelity, outcomes, fit, capacity, and sustainability? 120 Another potentially fruitful area for future research is the design of measures that incorporate specific areas of PBIS adaptation. During the early stage of PBIS implementation, adaptations often focus on addressing barriers to move toward the next stage of implementation and address barriers or concerns from stakeholders. Early adaptations by newly trained leadership teams often require high levels of support from a technical assistance provider (e.g., coach, trainer) to ensure that adaptations will not degrade fidelity or capacity of the innovation to meet the original need. During (or perhaps beyond) full implementation the focus is on embedding PBIS deeply into the routines and values of the organization and general competency of implementers and leadership teams has been established. This competency allows for broader adaptations to be folded into the PBIS framework. As previously mentioned, research groups have offered companion guides to the TFI, encouraging schools to weight the original PBIS-general fidelity items with criteria or follow-up questions related to specific initiatives that fit to or align with the goal of improving student social, emotional, and behavioral outcomes. One example is the Interconnected System Framework Action Planning Companion Guide to SWPBIS-Tiered Fidelity Inventory (Barrett et al., 2016), which provides enhanced criteria for scoring key items on the standard TFI to specifically address student mental health within PBIS. Schools that previously scored Team Operating Procedures as fully in place using the standard criteria may score the same item as partially in place using the enhanced criteria and identify this as an area for improvement to target. Much work remains to understand the role and effectiveness of these companion guides within PBIS decision making. The potential benefit of using these types of companion guides is that the measure becomes 121 flexible to local adaptations and improvement priorities over time, but further research is needed to establish indicators that a school or district is ready for any (or a specific) adaptation and the impact of adaptations on implementation of the original core elements of PBIS. Future companion guides may target areas of common areas of adaptation including restorative practices, trauma-informed practices, or family engagement. A potential barrier to the use of companion guides is the need to manage or consolidate enhancements over time as target areas expand or shift. For example, a district may introduce a companion guide to embed restorative practices within PBIS for three years to schools at full implementation. At the end of three years the district may identify that several, but not all schools have successfully embedded restorative practices and are interested in next targeting family engagement into their PBIS framework and would like a similar companion guide to use with their existing fidelity measure (e.g., TFI). These schools will want to monitor and sustain improvement of restorative practices as they shift towards increasing family engagement but administering multiple companion guides will likely reduce team efficiency. Consolidating multiple companion guides into the original measure may initially seem feasible but would evolve into an ever-growing set of criteria and the need to regularly manage and revise, eventually leading back to reduced efficiency. This scenario would become further complicated if significant barriers to PBIS occurred (e.g., administrative turn-over, budget reduction) and resulted in the need for the school or district to simplify implementation efforts or even move back to an earlier implementation stage and cut down on the enhanced criteria. Further research should focus on studying a wide range of adaptations to PBIS, and the indicators needed to drive implementation decisions about adaptations over time, especially during periods when major barriers to 122 sustained implementation occur (e.g., budget reduction, world-wide pandemic). Implications for Practice While the results of this study were modest and further research is needed to guide efforts toward use of data to sustain fidelity of PBIS implementation over time, the results offer implications for school and district leaders engaged in PBIS implementation efforts. School Administrators and PBIS Leadership Teams All school personnel and many other stakeholders (e.g., district personnel, students, family members) are involved in PBIS implementation activities, but school administrators and decision-making teams are intended to represent the broader stakeholder group(s) and provide the leadership and organization for PBIS and related practices that establish the competency. For example, PBIS school leaders influence the selection of behavioral practices and data sources to guide decisions, the organization and presentation of those data to different stakeholder groups and choose procedures for translating data into information that will drive decisions. Two implications of research on fidelity, sustainability, and data use in PBIS are the use of implementation focused (vs outcome) measures and the use of decision guidelines or rules that integrate various data sources. Use of Implementation-Focused Measures Measures that focus on features of implementation (e.g., fidelity, sustainability, capacity), rather than outcomes have been historically viewed as research or accountability tools, providing documentation of treatment integrity with little application to iterative decision cycles. Over the last decade, the field of education has sought to expand the role, format, and functions of implementation measures, especially measures of implementation fidelity. Beyond external accountability, implementation-focused measures are collected 123 for: (a) shared internal accountability to sustain adopted practices, (b) formative assessment of need and readiness for an innovation or practice, (c) guidance during the multi-year stages of implementing complex educational innovations, and (d) ongoing guidance in efforts to sustain core elements of the innovation while making adaptations to improve contextual fit and local effectiveness. In addition to measuring implementation fidelity or alignment to core elements of the target innovation, implementation-focused measures may also be modified to also include measurement of other constructs like capacity and sustainability. Capacity measures serve to evaluate the scale or availability of the innovation to meet the intended recipients or organization needs. For example, school leaders may use referral patterns and student interviews to determine that the CICO intervention is beneficial to some recipients but has been over-delivered to students who are less likely to benefit from the intervention based on behavioral patterns and under-delivered to students who are likely to benefit. Measurement of sustainability evaluates the potential for future use (and benefit) of the innovation. For example, school leaders may use fidelity measures and staff survey results to determine that staff perceptions of PBIS are not favorable and that PBIS is viewed as a fad innovation that will soon fade if the school team does not increase visibility and direct relevance to implementer routines. Over time, the addition of items (or modifications to existing items) of implementation-focused measures may add a modest amount of effort to the completion of implementation measures, but gaining this information would allow the school PBIS team to intentionally address specific staff concerns early, possibly by increasing transparency and staff involvement. Monitoring features of fidelity, capacity, and sustainability will increase the ability of school decision makers to confidently and 124 efficiently make implementation decisions, allocate local resources, and advocate for resources (e.g., training, materials, staff) when needed. Comprehensive Decision Guidelines for Data Use and Sharing A second implication of this study for school PBIS leaders is the need for decision guidelines to improve the use of data for decision making within iterative decision cycles. Too often in education, data collection is seen as a task performed by schools but delivered to the district, state, and federal levels for accountability and decision making at broad scales. The theoretical model of PBIS School Team Data Use identifies decision rules and guidelines as a set of pre-determined indicators used by school and district teams for data to guide decisions targeting school improvement. Decision guidelines allow a team to bypass lengthy discussions about whether data patterns (level, trend, peaks/valleys) indicate a need for action. Clear decision guidelines also allow the team to entrust the role of data analyst to an individual or subcommittee without sacrificing representative input in prioritizing the data shared with the team because the team has already come to agreement about the data patterns most likely to indicate a need for action, or at least further attention, by the team. Decision rules are sometimes organized around a set of evaluation questions, including: (a) Is the innovation being implemented with adequate fidelity? (b) Is the innovation sustainable or robust to common barriers (e.g., staff turn-over, new initiatives)? (c) Are an adequate number of students receiving access to the innovation? (d) Are an adequate number of students responding positively to the innovation? (e) Are there subgroups of students less likely to receive access or respond positively to the innovation? Decision rules may also be organized to support specific implementation actions that the team are likely to consider including: no action (e.g., continue using existing procedures), 125 scale up or expand usage of an innovation (e.g., train additional staff, recruit students), modification to intensify implementation efforts (e.g., increase dosage, increase data collection), modification to reduce implementation efforts (e.g., fade or simplify procedures), or discontinue use of a practice or intervention. Tiered decision systems will include decision guidelines for specific practices (e.g., CICO) or tiers (e.g., Tier 2 supports) as well as broader systems shared across practices. Comprehensive decision guidelines will also include indicators of implementation (e.g., fidelity, sustainability, capacity) as well as indicators of student outcomes. Training materials for the Team Initiated Problem Solving (TIPS) model includes a generic template of decision rules that teams can adapt with specific fidelity and outcome data sources (Horner et al., 2016). This template is provided in Appendix J and includes sample decision rules by implementation question for Tier 1 (p. 4), Tier 2 (p. 6), and Tier 3 (p. 8). To competently develop decision guidelines using this or other templates, school teams should also receive adequate training and coaching on data and decision systems so that the final guidelines directly inform contextually relevant decisions about PBIS practices, systems, data, and outcomes across all tiers of support. District Leadership Teams While schools and school PBIS teams were the unit of analysis for this study, there are also implications for the broader regional or district leaders. PBIS and improvement science literature both point to the need for supportive leadership to support development and sustainability of both organization and competency drivers within the organization. In education, the district leaders often have a greater influence on resource allocation and accountability standards than school leadership teams (Datnow, 2005). To better support 126 schools in use of data and sustained, high-fidelity implementation of PBIS, district leadership teams may benefit from engaging school team leaders in identifying a set of data systems that are likely to benefit school-level decisions about fidelity, sustainability, capacity, and outcomes. These data systems can be recommended to other schools and the district can coordinate access not only to the recommended data systems but also training and coaching on use of data for decisions that includes those data systems. A second implication for district leaders is the modeling of data use at the district- level. District leadership teams involved in PBIS implementation also need data and decision systems to guide their decisions and making those data and procedures available to school-level stakeholders may contribute toward creating a strong data culture across the district. Sharing of district-level data may also include aggregated or even disaggregated school-level data to support district decisions about allocation of resources or policy development to address implementation needs. District sharing of school-level data should be a collaborative process to alleviate potential misuse or misrepresentation of data that may reflect poorly on PBIS implementation or broader school improvement efforts. Initial efforts to share data across schools should focus on acknowledging and reinforcing school efforts and successes in use of data to improve school outcomes. Once a culture of trust and shared data use has been established it may be acceptable to begin sharing data that represent ongoing improvements and highlight implementation problems. It may be necessary to anonymize this type of data but eventually normalizing transparency about implementation challenges as well as successes may benefit school and district stakeholder decision making. A final implication for district leaders involved in PBIS implementation is public 127 commitment. The PBIS sustainability literature points to both school and district factors in predicting the sustained implementation of PBIS with adequate fidelity. While the current literature base of PBIS sustainability has focused on Tier 1, it is likely that district factors also predict sustainability of advanced tiers, especially at Tier 3 which often involves district personnel (e.g., behavior coach, district psychologist). One strategy for supporting school sustainability is to make district commitment to PBIS a consistent message for a predetermined number of years. For example, the district leadership may publicly commit to supporting school implementation of PBIS for a specific number of years and regularly include PBIS updates in messaging (e.g., newsletters, training, year-end reports). Committing for a specific number of years will also provide periodic opportunities to revisit PBIS commitment and to elicit school feedback about current implementation efforts and student outcomes before making an intentional decision to recommit or abandon the framework. Conclusion This study explored relations between three key aspects of school implementation of PBIS across tiers of support: sustainability, fidelity, and use of data. Many innovations (e.g., strategies, practices, interventions) have been identified over the last several decades to effectively meet student social, emotional, and behavioral needs. Recent attention has turned to questions about how best to implement one or more of those innovations at an adequate level of fidelity over a sustained period that will result in socially relevant scales of effects for students. The theoretical model adopted in this study identifies use of data as a key influencer of implementation including fidelity, scale or capacity, and sustainability. This study identified a small number of potential measures of data use to explore these 128 relations. Results indicate that simple and indirect measures of data use, based on access of student behavioral data, are insufficient to understand these complex relations at the school level. There may be many different uses of data that are influenced by stage of implementation, the purpose or priorities of the team, and the data or decision systems available to the team. Further research is needed to examine the role of data use in implementation of innovations within and across tiers of support in the PBIS framework. 129 APPENDIX A SWIS READINESS CHECKLIST 130 SWIS Readiness Checklist School-wide Information System School/Facility: _____________________________ Certified SWIS Facilitator: _________________________ Date: _____________ Status SWIS Requirements Data Source Next Not in place Partial In Place Check 1. Building administrator supports the implementation and use of SWIS. Administrator Interview 2. A school/facility-wide behavior support team exists and reviews SWIS Team Roster & Meeting referral data at least monthly. Schedule 3. The school/facility has an incident referral form and definitions for Incident Referral Form(s) behaviors resulting in administrative-managed (major) vs. staff-managed Problem Behavior (minor) incidents in place that is compatible with SWIS referral data entry. Definitions 4. Within three months of SWIS licensing, the school/facility is committed to having in place a clearly documented, predictable system for managing Written Guidelines disruptive behavior (e.g., School-wide PBIS). 5. Data entry time and staffing are scheduled to ensure that incident referral data will be current to within a week at all times. Data entry staff have access Data Entry & Report to all necessary information (e.g., student records). Generation Schedule 6. A small number of people within the school/facility are identified to gain Specific Date, Time, SWIS access and are scheduled to attend a 3-hour Swift at SWIS Training Location, Computers, conducted by a certified SWIS Facilitator. Internet Access 7. The school/facility agrees to maintain technology (i.e., internet browsers, district permissions) compatible with SWIS. 8. The school/facility agrees to both initial and ongoing coaching on the use of Administrator/ SWIS for school/facility-wide decision making. Coordinator Interview 9. The school/facility agrees to maintain SWIS compatibility and maintain communication with a certified SWIS Facilitator who agrees to provide Administrator/ ongoing support to the school/facility on the use of SWIS. Coordinator Interview Items that are Not in Place or Partially in Place can be organized into an action plan. 131 APPENDIX B SWIS REFERRAL COMPATIBILITY CHECKLIST SWIS Referral Form Compatibility Checklist School: Date: Compatibility Item Date Date Does a form exist that is SWIS compatible for SWIS data entry that includes the required categories (listed Yes No Yes No below)? Student name Yes No Yes No Student’s grade level Yes No Yes No Referring staff member Yes No Yes No Date of incident Yes No Yes No Time of incident Yes No Yes No Location of incident Yes No Yes No Problem Behavior Yes No Yes No Perceived Motivation Yes No Yes No Others involved Yes No Yes No (Optional) Restraint/Seclusion Yes No Yes No Actions Taken Yes No Yes No (Optional) Notes Yes No Yes No (Optional) Custom Fields Yes No Yes No Does a set of definitions exist that clearly defines all categories on the office discipline referral form? Yes No Yes No Does a clear distinction between problem behaviors that are staff managed versus office managed exist? Yes No Yes No Is the referral process documented and available for staff reference? Yes No Yes No Next review date: Redesign your process, form, and definitions until answers to all questions are “Yes.” When answers to all questions are “Yes”, readiness requirements 4 & 5 are complete. 132 APPENDIX C CICO-SWIS READINESS CHECKLIST 133 CICO-SWIS Readiness Checklist Check In Check Out SWIS School/Facility: ___________________________ Certified CICO-SWIS Facilitator: ________________________ Date: _______________ CICO-SWIS Requirements Data Source Status Next Not in place Partial In Place Check 1. Building administrator supports the implementation and use of the Check In Check Administrator Out Intervention and CICO-SWIS. Interview 2. A school/facility-wide behavior support team exists with access to training and Team Roster & support for the CICO Intervention and reviews CICO-SWIS data at least twice monthly. Meeting Schedule 3. The school/facility has a CICO point card with the following information: Standard for all students Defined number of check-in periods (up to ten) CICO Point Card Defined number of expectations/goals (3-5) A three-point rating scale 4. Within three months of CICO-SWIS licensing, the school/facility is committed to having a clearly documented CICO system. Procedures include: Description of program CICO Coordinator Written Guidelines Process for identifying students for CICO Process/materials for training adults, students, and families 5. Data entry time and staffing are scheduled to ensure that point card data will be current to within three days at all times. Data entry staff have access to all necessary Data Entry & Report information (e.g., student records). Generation Schedule 6. A small number of people within the school/facility are identified to gain CICO-SWIS Specific Date, Time, access and are scheduled to attend a 90-minute Swift at CICO-SWIS Training conducted Location, by a certified CICO-SWIS Facilitator. Computers, Internet 7. The school/facility agrees to maintain technology (i.e., internet browsers, district Administrator/ permissions) compatible with CICO-SWIS. Coordinator Interview 8. The school/facility agrees to both initial and ongoing coaching and support on the use Administrator/ of CICO-SWIS with a certified CICO-SWIS Facilitator. Coordinator Interview Administrator/ 9. The school/facility agrees to maintain CICO-SWIS readiness/compatibility. Coordinator Interview Items that are Not in Place or Partially in Place can be organized into an action plan. 134 APPENDIX D SET: SCHOOL-WIDE EVALAUTION TOOL SCORING GUIDE (Sugai, Lewis-Palmer, Todd, & Horner, 2005) 135 School-wide Evaluation Tool (SET) Scoring Guide School_____________________________________ Date______________ District _____________________________________ State______________ Pre_________ Post_________ SET data collector______________ Data Source Feature Evaluation Question (circle sources used) Score: P= product; I=interview; 0-2 O= observation 1. Is there documentation that staff has agreed to 5 or fewer positively stated school rules/ behavioral Discipline handbook, expectations? (0=no; 1= too many/negatively focused; 2 = Instructional materials P A. yes) Other_____________ Expectations Defined 2. Are the agreed upon rules & expectations publicly posted in 8 of 10 locations? (See interview & observation Wall posters form for selection of locations). (0= 0-4; 1= 5-7; 2= 8-10) Other_____________ O 1. Is there a documented system for teaching behavioral expectations to students on an annual basis? Lesson plan books, Instructional P (0= no; 1 = states that teaching will occur; 2= yes) materials Other_____________ 2. Do 90% of the staff asked state that teaching of behavioral expectations to students has occurred this year? Interviews B. (0= 0-50%; 1= 51-89%; 2=90%-100%) Other _____________ I Behavioral 3. Do 90% of team members asked state that the school- Expectations wide program has been taught/reviewed with staff on an Interviews Taught annual basis? Other_____________ I (0= 0-50%; 1= 51-89%; 2=90%-100%) 4. Can at least 70% of 15 or more students state 67% of the Interviews school rules? (0= 0-50%; 1= 51-69%; 2= 70-100%) Other_____________ I 5. Can 90% or more of the staff asked list 67% of the school Interviews rules? (0= 0-50%; 1= 51-89%; 2=90%-100%) Other_____________ I 1. Is there a documented system for rewarding student Instructional materials, Lesson behavior? Plans, Interviews P (0= no; 1= states to acknowledge, but not how; 2= yes) Other_____________ C. 2. Do 50% or more students asked indicate they have On-going Systemf or received a reward (other than verbal praise) for Interviews Rewarding expected behaviors over the past two months? Other_____________ I Behavioral (0= 0-25%; 1= 26-49%; 2= 50-100%) Expectations 3. Do 90% of staff asked indicate they have delivered a reward (other than verbal praise) to students for Interviews expected behavior over the past two months? Other_____________ I (0= 0-50%; 1= 51-89%; 2= 90-100%) 1. Is there a documented system for dealing with and reporting specific behavioral violations? Discipline handbook, P (0= no; 1= states to document; but not how; 2 = yes) Instructional materials D. Other_____________ System for 2. Do 90% of staff asked agree with administration on what problems are office-managed and what problems are Interviews Responding to classroom–managed? (0= 0-50%; 1= 51-89%; 2= 90-100%) Other_____________ I Behavioral Violations 3. Is the documented crisis plan for responding to extreme dangerous situations readily available in 6 of 7 locations? Walls O (0= 0-3; 1= 4-5; 2= 6-7) Other School-wide Evaluation Tool version 2.1, June 2005 © 2001 Sugai, Lewis-Palmer, Todd & Horner Educational and Community Supports University of Oregon Revised 06-29-05 NKS 136 Data Source Feature Evaluation Question (circle sources used) Score: 0-P= product; I=interview; 2 O= observation 4. Do 90% of staff asked agree with administration on the procedure for handling extreme emergencies (stranger in Interviews building with a weapon)? Other I (0= 0-50%; 1= 51-89%; 2= 90-100%) 1. Does the discipline referral form list (a) student/grade, (b) date, (c) time, (d) referring staff, (e) problem behavior, Referral form (f) location, (g) persons involved, (h) probable motivation, (circle items present on the & (i) administrative decision? referral form) P (0=0-3 items; 1= 4-6 items; 2= 7-9 items) 2. Can the administrator clearly define a system for collecting & summarizing discipline referrals (computer Interview software, data entry time)? Other I (0=no; 1= referrals are collected; 2= yes) 3. Does the administrator report that the team provides Interview E. discipline data summary reports to the staff at least three Other Monitoring & times/year? (0= no; 1= 1-2 times/yr.; 2= 3 or more times/yr) I Decision-Making 4. Do 90% of team members asked report that discipline data is used for making decisions in designing, Interviews implementing, and revising school-wide effective behavior Other support efforts? I (0= 0-50%; 1= 51-89%; 2= 90-100%) 1. Does the school improvement plan list improving School Improvement Plan, behavior support systems as one of the top 3 school Interview P improvement plan goals? (0= no; 1= 4th or lower priority; 2 Other = 1st- 3rd priority) I 2. Can 90% of staff asked report that there is a school-wide Interviews team established to address behavior support systems in the Other school? (0= 0-50%; 1= 51-89%; 2= 90-100%) I 3. Does the administrator report that team membership Interview includes representation of all staff? (0= no; 2= yes) Other I 4. Can 90% of team members asked identify the team Interviews leader? (0= 0-50%; 1= 51-89%; 2= 90-100%) Other I 5. Is the administrator an active member of the school-wide Interview behavior support team? Other (0= no; 1= yes, but not consistently; 2 = yes) I 6. Does the administrator report that team meetings occur at F. least monthly? Interview Management (0=no team meeting; 1=less often than monthly; 2= at least Other I monthly) 7. Does the administrator report that the team reports Interview progress to the staff at least four times per year? Other (0=no; 1= less than 4 times per year; 2= yes) I 8. Does the team have an action plan with specific goals Annual Plan, calendar that is less than one year old? (0=no; 2=yes) Other P G. 1. Does the school budget contain an allocated amount of Interview District-Level money for building and maintaining school-wide behavioral Other I Support support? (0= no; 2= yes) 2. Can the administrator identify an out-of-school liaison in Interview Other the district or state? (0= no; 2=yes) I Summary A = /4 B = /10 C = /6 D = /8 E = /8 Scores: F = /16 G = /4 Mean = /7 School-wide Evaluation Tool version 2.1, June 2005 © 2001 Sugai, Lewis-Palmer, Todd & Horner Educational and Community Supports University of Oregon Revised 06-29-05 NKS 137 APPENDIX E TFI: SWPBIS TIERED FIDELITY INVENTORY VERSION 2.1 (Algozzine, Barrett, Eber, George, Horner, Lewis, Putnam, Swain-Bradway, McIntosh, & Sugai, 2019) 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 APPENDIX F BOQ: BENCHMARKS OF QUALITY FULL INSTRUMENT (Algozzine, Barrett, Eber, George, Horner, Lewis, Putnam, Swain-Bradway, McIntosh, & Sugai, 2019) 158 159 160 161 162 163 164 165 166 167 168 169 170 APPENDIX G SUBSIST: SCHOOL-WIDE UNIVERSAL BEHAVIOR SUSTAINABILITY INDEX: SCHOOL TEAMS (McIntosh, Doolittle, Vincent, Horner, Ervin, 2009) School Factors Factor 1. School Priority S1.1 SWPBS (aka School-Wide PBS, PBIS, EBS) serves a critical need for the school S1.2 SWPBS addresses outcomes that are highly valued by school personnel S1.3 A vast majority of school personnel (80% or more) support SWPBS S1. 4 SWPBS has been integrated into new school or district initiatives (e.g., renamed to meet new needs, shown how it can meet the goals of the new initiatives well) S1.5 Parents are actively involved in the SWPBS effort (e.g., as part of PBS team or district committee) S1.6 The school administrators describe SWPBS as a top priority for the school S1.7 The school administrators actively support school personnel when implementing and aligning initiatives (e.g., shield staff from competing demands, change language to align SWPBS with new initiatives) to allow SWPBS to occur S1.8 A school administrator regularly attends and participates in SWPBS team meetings S1.9 The practices and strategies of SWPBS are evidence based (i.e., there is published research documenting their effectiveness) 171 S1.10 School personnel perceive SWPBS as an effective in helping them achieve desired outcomes S1.11 School personnel celebrate the positive effects of SWPBS at least yearly S1.12 SWPBS has a "crossover effect" in other areas (e.g., improved academic achievement scores, attendance) S1.13 SWPBS is effective for a large proportion of students S1.14 SWPBS has been expanded to other areas (e.g., classrooms, buses, students with intensive needs, parenting workshops) S1.15 SWPBS is implemented with fidelity (i.e., it is used as intended) S1.16 SWPBS becomes easier to use with continued experience S1.17 SWPBS is considered to be a typical operating procedure of the school (i.e., it has become "what we do here/what we've always done") S1.18 SWPBS is cost effective (in terms of money and effort) S1.19 Dara collected for SWPBS are easy to collect and do not interfere with teaching S1.20 Materials related to SWPBS (e.g., handbook, posters) can be used or adapted with ease across years Factor 2. Team Use of Data S2.1 The school team implementing SWPBS is knowledgeable and skilled in SWPBS S2.2 The school team implementing SWPBS is well organized and operates efficiently S2.3 The school team implementing SWPBS meets at least monthly S2.4 Needs assessments (e.g., EBS/PBIS Self Assessment Survey) are conducted 172 S2.5 There is regular measurement of fidelity of implementation (e.g., team checklist, set, benchmarks of quality) S2.6 There is regular measurement of student outcomes (e.g., ODRs, achievement data, school safety survey, student/parent satisfaction survey) S2.7 Data are reviewed regularly at team meetings S2.8 Data are presented to all school personnel at least four times per year S2.9 Data are presented at least once per year to key stakeholders outside of the school (e.g., district officials, school boards, community agencies/groups) S2.10 Data are used for problem solving, decision making, and action planning (to make SWPBS more effective and/or efficient) S2.11 All school personnel have a basic understanding of SWPBS (i.e., know the critical features and practices) District Factors Factor 3. District Priority D1.1 There are adequate district resources (funding and time) allocated for SWPBS D1.2 The district administration actively supports SWPBS (e.g., describes SWPBS as top priority, provides clear direction) D1.3 State/provincial officials actively support SWPBS (e.g., promotion, publicity, providing infrastructure) D1.4 SWPBS is promoted and visible to important organizations (e.g., school board, community agencies, businesses, parent groups) D1.5 SWPBS is embedded into school and/or district policy (e.g., school improvement 173 plans, mission/vision statements) Factor 4. Capacity Building D2.1 The school team has regular access to district SWPBS expertise (e.g., external/ district coaches or consultants) D2.2 School teams and new personnel are provided with professional development in SWPBS at least yearly D2.3 The school team is connected to a "community of practice" (e.g., network of ocher SWPBS schools in district, local/ regional conferences) 174 APPENDIX H ALTITUDE: ADVANCED LEVEL TIER INTERVENTIONS TREATMENT UTILIZATION AND DURABILITY EVALUATION (MCINTOSH, KITTELMAN, MERCER, NESE, 2021) Advanced Tier General Factor 1. There is adequate communication across all teams providing Tier 1, 2, and 3 behavior support. 2. The team(s) responsible for Tier 2 and 3 behavior systems have procedures in place to select and/or train team members on supporting these behavior systems (e.g., monitoring fidelity and student performance). 3. The team(s) and school personnel responsible for implementing Tier 2 and 3 behavior interventions receive acknowledgement for implementation efforts and accomplishments. 4. School personnel understand the importance of monitoring intervention fidelity for Tier 2 and 3 behavior interventions. 5. The school principal consistently expresses the importance of implementing Tier 2 and 3 behavior interventions. 6. District administrators express a commitment that students with intensive needs should be supported within schools instead of being removed to more restrictive settings. 7. The team(s) responsible for implementing Tier 2 and 3 behavior systems are connected to a “community of practice” (i.e., a network of other schools implementing Tier 2 and 3 behavior systems in the district, state, or region). 175 8. Materials (e.g., documents, curricula, tools) for implementing Tier 2 and 3 behavior interventions can be used or adapted with ease over time. 9. School personnel are committed to supporting students in the classroom instead of excluding them from instruction. 10. School personnel are committed to implementing Tier 2 and 3 behavior interventions. 11. The team(s) responsible for implementing Tier 2 and 3 behavior systems effectively problem-solve barriers to implementation. 12. School personnel build strong partnerships with families to support students with Tier 2 and 3 behavior needs. 13. Classroom teachers regularly receive fidelity and student progress data about their assigned students receiving Tier 2 and 3 behavior interventions. 14. Parents/caregivers regularly receive progress data about their children and youth participating in Tier 2 and 3 behavior interventions. Tier 2 Specific Factor 15. All Tier 2 behavior interventions (e.g., Check-In Check-Out, social/emotional skills small groups) are coordinated by one team. 16. The team responsible for Tier 2 behavior systems has adequate resources (e.g., time, personnel, materials) to implement Tier 2 behavior interventions with fidelity. 17. The team responsible for Tier 2 behavior systems has adequate access to training in Tier 2 behavior systems and interventions. 176 18. The team responsible for Tier 2 behavior systems has adequate access to coaching for Tier 2 behavior systems and interventions. 19. School personnel are knowledgeable in the logic and practices of Tier 2 behavior interventions (e.g., why the interventions should work, for whom specific interventions are most likely to be effective, specific skills needed to implement the interventions). 20. School personnel implementing Tier 2 behavior interventions have the necessary skills for collecting fidelity and student progress data accurately. 21. The team responsible for Tier 2 behavior systems uses fidelity and student progress data to improve behavior systems and outcomes. 22. Individuals implementing Tier 2 behavior interventions can collect fidelity and student progress data efficiently. 23. The school’s Tier 2 behavior data systems are easy for the team to use for decision making. Tier 3 Specific Factor 24. The team responsible for Tier 3 behavior systems has adequate resources (e.g., time, personnel, materials) to implement Tier 3 behavior interventions with fidelity. 25. The team responsible for Tier 3 behavior systems has adequate access to training in Tier 3 behavior systems and interventions. 26. The team responsible for Tier 3 behavior systems has adequate access to coaching for Tier 3 behavior systems and interventions. 177 27. School personnel are knowledgeable in the logic and practices of Tier 3 behavior interventions (e.g., why the interventions should work, for whom specific interventions are most likely to be effective, specific skills needed to implement the interventions). 28. School personnel implementing Tier 3 behavior interventions have the necessary skills for collecting fidelity and student progress data accurately. 29. The team responsible for Tier 3 behavior systems uses fidelity and student progress data to improve behavior systems and outcomes. 30. Individuals implementing Tier 3 behavior interventions can collect fidelity and student progress data without too much effort. 31. The school’s Tier 3 behavior data systems are easy for the team to use for decision making. 32. District administrators promote a standard set of evidence-based Tier 3 behavior systems, data, and practices (e.g., screening tools, interventions, decision rules) to be used district-wide. 178 APPENDIX I RQ3 PARTIAL CORRELATIONS SPSS SYNTAX AND RESULTS *Part 1. SUB Overall Score. DATASET ACTIVATE DataSet1. PARTIAL CORR /VARIABLES= SUB_201819_OverallRatio SUB_201819_SchPriority_TotalRatio BY ALT1_201819_OverallRatio /SIGNIFICANCE=TWOTAIL /STATISTICS=DESCRIPTIVES CORR /MISSING=LISTWISE. Descriptive Statistics Mean Std. Deviation N Subsist Overall Score .7772 .18627 614 SUB_201819_SchPriority_TotalRatio .7919 .18662 614 Altitude Overall Score .6101 .21302 614 Correlations SUB_201819_ Subsist Overall SchPriority_To Altitude Control Variables Score talRatio Overall Score -none-a Subsist Overall Score Correlation 1.000 .881 .688 Significance (2-tailed) . .000 .000 df 0 612 612 SUB_201819_SchPrio Correlation .881 1.000 .658 rity_TotalRatio Significance (2-tailed) .000 . .000 df 612 0 612 Altitude Overall Score Correlation .688 .658 1.000 Significance (2-tailed) .000 .000 . df 612 612 0 Altitude Overall Subsist Overall Score Correlation 1.000 .783 Score Significance (2-tailed) . .000 df 0 611 SUB_201819_SchPrio Correlation .783 1.000 rity_TotalRatio Significance (2-tailed) .000 . df 611 0 a. Cells contain zero-order (Pearson) correlations. 179 DATASET ACTIVATE DataSet1. PARTIAL CORR /VARIABLES= SUB_201819_OverallRatio SUB_201819_DataUse_TotalRatio BY ALT1_201819_OverallRatio /SIGNIFICANCE=TWOTAIL /STATISTICS=DESCRIPTIVES CORR /MISSING=LISTWISE. Descriptive Statistics Mean Std. Deviation N Subsist Overall Score .7772 .18627 614 SUB_201819_DataUse_TotalRatio .8131 .19497 614 Altitude Overall Score .6101 .21302 614 Correlations SUB_201819_ Subsist Overall DataUse_Total Altitude Overall Control Variables Score Ratio Score -none-a Subsist Overall Score Correlation 1.000 .888 .688 Significance (2-tailed) . .000 .000 df 0 612 612 SUB_201819_DataU Correlation .888 1.000 .588 se_TotalRatio Significance (2-tailed) .000 . .000 df 612 0 612 Altitude Overall Correlation .688 .588 1.000 Score Significance (2-tailed) .000 .000 . df 612 612 0 Altitude Overall Subsist Overall Score Correlation 1.000 .824 Score Significance (2-tailed) . .000 df 0 611 SUB_201819_DataU Correlation .824 1.000 se_TotalRatio Significance (2-tailed) .000 . df 611 0 a. Cells contain zero-order (Pearson) correlations. 180 DATASET ACTIVATE DataSet1. PARTIAL CORR /VARIABLES= SUB_201819_OverallRatio SUB_201819_DistPriority_TotalRatio BY ALT1_201819_OverallRatio /SIGNIFICANCE=TWOTAIL /STATISTICS=DESCRIPTIVES CORR /MISSING=LISTWISE. Descriptive Statistics Mean Std. Deviation N Subsist Overall Score .7778 .18590 612 SUB_201819_DistPriority_TotalRatio .7381 .22413 612 Altitude Overall Score .6106 .21319 612 Correlations Subsist SUB_201819 Altitude Overall _DistPriority Overall Control Variables Score _TotalRatio Score -none-a Subsist Overall Score Correlation 1.000 .877 .689 Significance (2-tailed) . .000 .000 df 0 610 610 SUB_201819_DistPriority_T Correlation .877 1.000 .613 otalRatio Significance (2-tailed) .000 . .000 df 610 0 610 Altitude Overall Score Correlation .689 .613 1.000 Significance (2-tailed) .000 .000 . df 610 610 0 Altitude Overall Subsist Overall Score Correlation 1.000 .793 Score Significance (2-tailed) . .000 df 0 609 SUB_201819_DistPriority_T Correlation .793 1.000 otalRatio Significance (2-tailed) .000 . df 609 0 a. Cells contain zero-order (Pearson) correlations. 181 DATASET ACTIVATE DataSet1. PARTIAL CORR /VARIABLES= SUB_201819_OverallRatio SUB_201819_DistCapacity_TotalRatio BY ALT1_201819_OverallRatio /SIGNIFICANCE=TWOTAIL /STATISTICS=DESCRIPTIVES CORR /MISSING=LISTWISE. Descriptive Statistics Mean Std. Deviation N Subsist Overall Score .7782 .18583 612 SUB_201819_DistCapacity_TotalRatio .7672 .23882 612 Altitude Overall Score .6108 .21277 612 Correlations SUB_201819_ Altitude Subsist DistCapacity_T Overall Control Variables Overall Score otalRatio Score -none-a Subsist Overall Score Correlation 1.000 .882 .687 Significance (2-tailed) . .000 .000 df 0 610 610 SUB_201819_DistCapac Correlation .882 1.000 .576 ity_TotalRatio Significance (2-tailed) .000 . .000 df 610 0 610 Altitude Overall Score Correlation .687 .576 1.000 Significance (2-tailed) .000 .000 . df 610 610 0 Altitude Overall Subsist Overall Score Correlation 1.000 .819 Score Significance (2-tailed) . .000 df 0 609 SUB_201819_DistCapac Correlation .819 1.000 ity_TotalRatio Significance (2-tailed) .000 . df 609 0 a. Cells contain zero-order (Pearson) correlations. 182 *Exclude Overall SUBSIST with Overall ALTITUDE (not appropriate to control for individual factors). DATASET ACTIVATE DataSet1. PARTIAL CORR /VARIABLES= SUB_201819_OverallRatio ALT1_201819_AdvTier_TotalRatio BY ALT1_201819_Tier2_TotalRatio ALT1_201819_Tier3_TotalRatio /SIGNIFICANCE=TWOTAIL /STATISTICS=DESCRIPTIVES CORR /MISSING=LISTWISE. Descriptive Statistics Mean Std. Deviation N Subsist Overall Score .7770 .18794 537 ALT1_201819_AdvTier_TotalRatio .6301 .19852 537 ALT1_201819_Tier2_TotalRatio .6537 .22429 537 ALT1_201819_Tier3_TotalRatio .5457 .27130 537 Correlations Subsist ALT1_2018 ALT1_2018 ALT1_2018 Overall 19_AdvTier 19_Tier2_To 19_Tier3_To Control Variables Score _TotalRatio talRatio talRatio -none-a Subsist Overall Correlation 1.000 .722 .715 .518 Score Significance (2-tailed) . .000 .000 .000 df 0 535 535 535 ALT1_201819_Ad Correlation .722 1.000 .843 .723 vTier_TotalRatio Significance (2-tailed) .000 . .000 .000 df 535 0 535 535 ALT1_201819_Tie Correlation .715 .843 1.000 .759 r2_TotalRatio Significance (2-tailed) .000 .000 . .000 df 535 535 0 535 ALT1_201819_Tie Correlation .518 .723 .759 1.000 r3_TotalRatio Significance (2-tailed) .000 .000 .000 . df 535 535 535 0 ALT1_20181 Subsist Overall Correlation 1.000 .342 9_Tier2_Tota Score Significance (2-tailed) . .000 lRatio & df 0 533 ALT1_20181 ALT1_201819_Ad Correlation .342 1.000 9_Tier3_Tota vTier_TotalRatio Significance (2-tailed) .000 . lRatio df 533 0 a. Cells contain zero-order (Pearson) correlations. 183 DATASET ACTIVATE DataSet1. PARTIAL CORR /VARIABLES= SUB_201819_OverallRatio ALT1_201819_Tier2_TotalRatio BY ALT1_201819_AdvTier_TotalRatio ALT1_201819_Tier3_TotalRatio /SIGNIFICANCE=TWOTAIL /STATISTICS=DESCRIPTIVES CORR /MISSING=LISTWISE. Descriptive Statistics Mean Std. Deviation N Subsist Overall Score .7770 .18794 537 ALT1_201819_Tier2_TotalRatio .6537 .22429 537 ALT1_201819_AdvTier_TotalRatio .6301 .19852 537 ALT1_201819_Tier3_TotalRatio .5457 .27130 537 Correlations Subsist ALT1_20181 ALT1_20181 ALT1_2018 Overall 9_Tier2_Tot 9_AdvTier_T 19_Tier3_T Control Variables Score alRatio otalRatio otalRatio -none-a Subsist Overall Correlation 1.000 .715 .722 .518 Score Significance (2-tailed) . .000 .000 .000 df 0 535 535 535 ALT1_201819_Tier Correlation .715 1.000 .843 .759 2_TotalRatio Significance (2-tailed) .000 . .000 .000 df 535 0 535 535 ALT1_201819_Adv Correlation .722 .843 1.000 .723 Tier_TotalRatio Significance (2-tailed) .000 .000 . .000 df 535 535 0 535 ALT1_201819_Tier Correlation .518 .759 .723 1.000 3_TotalRatio Significance (2-tailed) .000 .000 .000 . df 535 535 535 0 ALT1_201819 Subsist Overall Correlation 1.000 .316 _AdvTier_Tota Score Significance (2-tailed) . .000 lRatio & df 0 533 ALT1_201819 ALT1_201819_Tier Correlation .316 1.000 _Tier3_TotalRa 2_TotalRatio tio Significance (2-tailed) .000 . df 533 0 a. Cells contain zero-order (Pearson) correlations. 184 DATASET ACTIVATE DataSet1. PARTIAL CORR /VARIABLES= SUB_201819_OverallRatio ALT1_201819_Tier3_TotalRatio BY ALT1_201819_AdvTier_TotalRatio ALT1_201819_Tier2_TotalRatio /SIGNIFICANCE=TWOTAIL /STATISTICS=DESCRIPTIVES CORR /MISSING=LISTWISE. Descriptive Statistics Mean Std. Deviation N Subsist Overall Score .7770 .18794 537 ALT1_201819_Tier3_TotalRatio .5457 .27130 537 ALT1_201819_AdvTier_TotalRatio .6301 .19852 537 ALT1_201819_Tier2_TotalRatio .6537 .22429 537 Correlations Subsist ALT1_2018 ALT1_2018 ALT1_2018 Overall 19_Tier3_To 19_AdvTier 19_Tier2_To Control Variables Score talRatio _TotalRatio talRatio -none-a Subsist Overall Correlation 1.000 .518 .722 .715 Score Significance (2-tailed) . .000 .000 .000 df 0 535 535 535 ALT1_201819_Ti Correlation .518 1.000 .723 .759 er3_TotalRatio Significance (2-tailed) .000 . .000 .000 df 535 0 535 535 ALT1_201819_A Correlation .722 .723 1.000 .843 dvTier_TotalRati Significance (2-tailed) .000 .000 . .000 o df 535 535 0 535 ALT1_201819_Ti Correlation .715 .759 .843 1.000 er2_TotalRatio Significance (2-tailed) .000 .000 .000 . df 535 535 535 0 ALT1_201819_A Subsist Overall Correlation 1.000 -.141 dvTier_TotalRati Score Significance (2-tailed) . .001 o & df 0 533 ALT1_201819_Ti ALT1_201819_Ti Correlation -.141 1.000 er2_TotalRatio er3_TotalRatio Significance (2-tailed) .001 . df 533 0 a. Cells contain zero-order (Pearson) correlations. 185 *Part 2. School Priority partial correlations controlling for all others (including overall and factor-level scores). PARTIAL CORR /VARIABLES=SUB_201819_SchPriority_TotalRatio SUB_201819_DataUse_TotalRatio BY SUB_201819_DistPriority_TotalRatio SUB_201819_DistCapacity_TotalRatio ALT1_201819_OverallRatio /SIGNIFICANCE=TWOTAIL /STATISTICS=DESCRIPTIVES CORR /MISSING=LISTWISE. Descriptive Statistics Mean Std. Deviation N SUB_201819_SchPriority_TotalRatio .7931 .18618 610 SUB_201819_DataUse_TotalRatio .8147 .19390 610 SUB_201819_DistPriority_TotalRatio .7392 .22316 610 SUB_201819_DistCapacity_TotalRatio .7682 .23838 610 Altitude Overall Score .6113 .21295 610 Correlations SUB_2018 SUB_201 SUB_201 SUB_201 19_SchPri 819_Data 819_DistP 819_DistC Altitude ority_Tota Use_Total riority_Tot apacity_T Overall Control Variables lRatio Ratio alRatio otalRatio Score -none-a SUB_201819_ Correlation 1.000 .823 .695 .641 .660 SchPriority_To Significance (2-tailed) . .000 .000 .000 .000 talRatio df 0 608 608 608 608 SUB_201819_ Correlation .823 1.000 .649 .695 .588 DataUse_Total Significance (2-tailed) .000 . .000 .000 .000 Ratio df 608 0 608 608 608 SUB_201819_ Correlation .695 .649 1.000 .723 .610 DistPriority_To Significance (2-tailed) .000 .000 . .000 .000 talRatio df 608 608 0 608 608 SUB_201819_ Correlation .641 .695 .723 1.000 .576 DistCapacity_T Significance (2-tailed) .000 .000 .000 . .000 otalRatio df 608 608 608 0 608 Altitude Correlation .660 .588 .610 .576 1.000 Overall Score Significance (2-tailed) .000 .000 .000 .000 . df 608 608 608 608 0 SUB_201819_D SUB_201819_ Correlation 1.000 .617 istPriority_Total SchPriority_To Significance (2-tailed) . .000 Ratio & talRatio df 0 605 SUB_201819_D SUB_201819_ Correlation .617 1.000 istCapacity_Tot DataUse_Total alRatio & Significance (2-tailed) .000 . Ratio Altitude Overall df 605 0 Score a. Cells contain zero-order (Pearson) correlations. 186 PARTIAL CORR /VARIABLES=SUB_201819_SchPriority_TotalRatio SUB_201819_DistPriority_TotalRatio BY SUB_201819_DataUse_TotalRatio SUB_201819_DistCapacity_TotalRatio ALT1_201819_OverallRatio /SIGNIFICANCE=TWOTAIL /STATISTICS=DESCRIPTIVES CORR /MISSING=LISTWISE. Descriptive Statistics Mean Std. Deviation N SUB_201819_SchPriority_TotalRatio .7931 .18618 610 SUB_201819_DistPriority_TotalRatio .7392 .22316 610 SUB_201819_DataUse_TotalRatio .8147 .19390 610 SUB_201819_DistCapacity_TotalRatio .7682 .23838 610 Altitude Overall Score .6113 .21295 610 Correlations SUB_201 SUB_201 819_Dist SUB_20 SUB_2018 819_SchP Priority_ 1819_Da 19_DistCa riority_To TotalRati taUse_T pacity_Tot Altitude Control Variables talRatio o otalRatio alRatio Overall Score -none-a SUB_201819_Sch Correlation 1.000 .695 .823 .641 .660 Priority_TotalRati Significance (2- . .000 .000 .000 .000 o tailed) df 0 608 608 608 608 SUB_201819_Dist Correlation .695 1.000 .649 .723 .610 Priority_TotalRati Significance (2- .000 . .000 .000 .000 o tailed) df 608 0 608 608 608 SUB_201819_Dat Correlation .823 .649 1.000 .695 .588 aUse_TotalRatio Significance (2- .000 .000 . .000 .000 tailed) df 608 608 0 608 608 SUB_201819_Dist Correlation .641 .723 .695 1.000 .576 Capacity_TotalRat Significance (2- .000 .000 .000 . .000 io tailed) df 608 608 608 0 608 Altitude Overall Correlation .660 .610 .588 .576 1.000 Score Significance (2- .000 .000 .000 .000 . tailed) df 608 608 608 608 0 SUB_201819_ SUB_201819_Sch Correlation 1.000 .267 DataUse_Total Priority_TotalRati Significance (2- . .000 Ratio & o tailed) SUB_201819_ df 0 605 DistCapacity_ TotalRatio & SUB_201819_Dist Correlation .267 1.000 Altitude Priority_TotalRati Significance (2- .000 . Overall Score o tailed) df 605 0 a. Cells contain zero-order (Pearson) correlations. PARTIAL CORR 187 /VARIABLES=SUB_201819_SchPriority_TotalRatio SUB_201819_DistCapacity_TotalRatio BY SUB_201819_DataUse_TotalRatio SUB_201819_DistPriority_TotalRatio ALT1_201819_OverallRatio /SIGNIFICANCE=TWOTAIL /STATISTICS=DESCRIPTIVES CORR /MISSING=LISTWISE. Descriptive Statistics Mean Std. Deviation N SUB_201819_SchPriority_TotalRatio .7931 .18618 610 SUB_201819_DistCapacity_TotalRatio .7682 .23838 610 SUB_201819_DataUse_TotalRatio .8147 .19390 610 SUB_201819_DistPriority_TotalRatio .7392 .22316 610 Altitude Overall Score .6113 .21295 610 Correlations SUB_2018 SUB_20181 SUB_2018 SUB_20181 19_SchPri 9_DistCapa 19_DataUs 9_DistPriori Altitude ority_Total city_TotalR e_TotalRat ty_TotalRat Overall Control Variables Ratio atio io io Score -none-a SUB_201819_SchPrio Correlation 1.000 .641 .823 .695 .660 rity_TotalRatio Significance . .000 .000 .000 .000 (2-tailed) df 0 608 608 608 608 SUB_201819_DistCa Correlation .641 1.000 .695 .723 .576 pacity_TotalRatio Significance .000 . .000 .000 .000 (2-tailed) df 608 0 608 608 608 SUB_201819_DataUs Correlation .823 .695 1.000 .649 .588 e_TotalRatio Significance .000 .000 . .000 .000 (2-tailed) df 608 608 0 608 608 SUB_201819_DistPri Correlation .695 .723 .649 1.000 .610 ority_TotalRatio Significance .000 .000 .000 . .000 (2-tailed) df 608 608 608 0 608 Altitude Overall Score Correlation .660 .576 .588 .610 1.000 Significance .000 .000 .000 .000 . (2-tailed) df 608 608 608 608 0 SUB_201819_ SUB_201819_SchPrio Correlation 1.000 -.062 DataUse_Total rity_TotalRatio Significance . .130 Ratio & (2-tailed) SUB_201819_ df 0 605 DistPriority_To SUB_201819_DistCa Correlation -.062 1.000 talRatio & pacity_TotalRatio Significance .130 . Altitude (2-tailed) Overall Score df 605 0 a. Cells contain zero-order (Pearson) correlations. PARTIAL CORR /VARIABLES= SUB_201819_SchPriority_TotalRatio ALT1_201819_OverallRatio BY SUB_201819_DataUse_TotalRatio SUB_201819_DistPriority_TotalRatio SUB_201819_DistCapacity_TotalRatio /SIGNIFICANCE=TWOTAIL /STATISTICS=DESCRIPTIVES CORR /MISSING=LISTWISE. 188 Descriptive Statistics Mean Std. Deviation N SUB_201819_SchPriority_TotalRatio .7931 .18618 610 Altitude Overall Score .6113 .21295 610 SUB_201819_DataUse_TotalRatio .8147 .19390 610 SUB_201819_DistPriority_TotalRatio .7392 .22316 610 SUB_201819_DistCapacity_TotalRatio .7682 .23838 610 Correlations SUB_2018 SUB_2018 SUB_2018 SUB_2018 19_SchPrio Altitude 19_DataUs 19_DistPri 19_DistCap rity_TotalR Overall e_TotalRati ority_Total acity_Total Control Variables atio Score o Ratio Ratio -none-a SUB_201819_S Correlation 1.000 .660 .823 .695 .641 chPriority_Total Significance . .000 .000 .000 .000 Ratio (2-tailed) df 0 608 608 608 608 Altitude Overall Correlation .660 1.000 .588 .610 .576 Score Significance .000 . .000 .000 .000 (2-tailed) df 608 0 608 608 608 SUB_201819_D Correlation .823 .588 1.000 .649 .695 ataUse_TotalRat Significance .000 .000 . .000 .000 io (2-tailed) df 608 608 0 608 608 SUB_201819_D Correlation .695 .610 .649 1.000 .723 istPriority_Total Significance .000 .000 .000 . .000 Ratio (2-tailed) df 608 608 608 0 608 SUB_201819_D Correlation .641 .576 .695 .723 1.000 istCapacity_Tot Significance .000 .000 .000 .000 . alRatio (2-tailed) df 608 608 608 608 0 SUB_201819_D SUB_201819_S Correlation 1.000 .290 ataUse_TotalRat chPriority_Total Significance . .000 io & Ratio (2-tailed) SUB_201819_D df 0 605 istPriority_Total Altitude Overall Correlation .290 1.000 Ratio & Score SUB_201819_D Significance .000 . istCapacity_Tot (2-tailed) alRatio df 605 0 a. Cells contain zero-order (Pearson) correlations. PARTIAL CORR /VARIABLES=SUB_201819_SchPriority_TotalRatio ALT1_201819_AdvTier_TotalRatio BY SUB_201819_DataUse_TotalRatio SUB_201819_DistPriority_TotalRatio SUB_201819_DistCapacity_TotalRatio ALT1_201819_Tier2_TotalRatio ALT1_201819_Tier3_TotalRatio /SIGNIFICANCE=TWOTAIL /STATISTICS=DESCRIPTIVES CORR /MISSING=LISTWISE. Descriptive Statistics 189 Mean Std. Deviation N SUB_201819_SchPriority_TotalRatio .7925 .18830 534 ALT1_201819_AdvTier_TotalRatio .6308 .19873 534 SUB_201819_DataUse_TotalRatio .8133 .19671 534 SUB_201819_DistPriority_TotalRatio .7383 .22568 534 SUB_201819_DistCapacity_TotalRatio .7682 .23864 534 ALT1_201819_Tier2_TotalRatio .6543 .22470 534 ALT1_201819_Tier3_TotalRatio .5464 .27184 534 Correlations SUB_2 SUB_2 ALT1_ SUB_2 SUB_2 01819_ ALT1_ ALT1_ 01819_ 201819 01819_ 01819_ DistCap 201819 201819 SchPrio _AdvTi DataUs DistPrio acity_T _Tier2_ _Tier3_ rity_Tot er_Total e_Total rity_Tot otalRati TotalRa TotalRa Control Variables alRatio Ratio Ratio alRatio o tio tio -none-a SUB_201819_ Correlation 1.000 .722 .836 .691 .640 .672 .478 SchPriority_T Significance . .000 .000 .000 .000 .000 .000 otalRatio (2-tailed) df 0 532 532 532 532 532 532 ALT1_201819 Correlation .722 1.000 .642 .619 .587 .842 .724 _AdvTier_Tot Significance .000 . .000 .000 .000 .000 .000 alRatio (2-tailed) df 532 0 532 532 532 532 532 SUB_201819_ Correlation .836 .642 1.000 .647 .706 .621 .425 DataUse_Tota Significance .000 .000 . .000 .000 .000 .000 lRatio (2-tailed) df 532 532 0 532 532 532 532 SUB_201819_ Correlation .691 .619 .647 1.000 .723 .623 .492 DistPriority_T Significance .000 .000 .000 . .000 .000 .000 otalRatio (2-tailed) df 532 532 532 0 532 532 532 SUB_201819_ Correlation .640 .587 .706 .723 1.000 .622 .442 DistCapacity_ Significance .000 .000 .000 .000 . .000 .000 TotalRatio (2-tailed) df 532 532 532 532 0 532 532 ALT1_201819 Correlation .672 .842 .621 .623 .622 1.000 .759 _Tier2_TotalR Significance .000 .000 .000 .000 .000 . .000 atio (2-tailed) df 532 532 532 532 532 0 532 ALT1_201819 Correlation .478 .724 .425 .492 .442 .759 1.000 _Tier3_TotalR Significance .000 .000 .000 .000 .000 .000 . atio (2-tailed) df 532 532 532 532 532 532 0 SUB_201819_Da SUB_201819_ Correlation 1.000 .279 taUse_TotalRatio SchPriority_T Significance . .000 & otalRatio (2-tailed) SUB_201819_Dis df 0 527 tPriority_TotalRat ALT1_201819 Correlation .279 1.000 io & _AdvTier_Tot Significance .000 . SUB_201819_Dis alRatio (2-tailed) 190 SUB_2 SUB_2 ALT1_ SUB_2 SUB_2 01819_ ALT1_ ALT1_ 01819_ 201819 01819_ 01819_ DistCap 201819 201819 SchPrio _AdvTi DataUs DistPrio acity_T _Tier2_ _Tier3_ rity_Tot er_Total e_Total rity_Tot otalRati TotalRa TotalRa Control Variables alRatio Ratio Ratio alRatio o tio tio tCapacity_TotalR df 527 0 atio & ALT1_201819_Ti er2_TotalRatio & ALT1_201819_Ti er3_TotalRatio a. Cells contain zero-order (Pearson) correlations. 191 PARTIAL CORR /VARIABLES=SUB_201819_SchPriority_TotalRatio ALT1_201819_Tier2_TotalRatio BY SUB_201819_DataUse_TotalRatio SUB_201819_DistPriority_TotalRatio SUB_201819_DistCapacity_TotalRatio ALT1_201819_AdvTier_TotalRatio ALT1_201819_Tier3_TotalRatio /SIGNIFICANCE=TWOTAIL /STATISTICS=DESCRIPTIVES CORR /MISSING=LISTWISE. Descriptive Statistics Mean Std. Deviation N SUB_201819_SchPriority_TotalRatio .7925 .18830 534 ALT1_201819_Tier2_TotalRatio .6543 .22470 534 SUB_201819_DataUse_TotalRatio .8133 .19671 534 SUB_201819_DistPriority_TotalRatio .7383 .22568 534 SUB_201819_DistCapacity_TotalRatio .7682 .23864 534 ALT1_201819_AdvTier_TotalRatio .6308 .19873 534 ALT1_201819_Tier3_TotalRatio .5464 .27184 534 Correlations SUB_20 SUB_20 SUB_20 SUB_20 ALT1_2 1819_Sc ALT1_2 1819_D 1819_Di 1819_Di 01819_ ALT1_2 hPriority 01819_T ataUse_ stPriorit stCapaci AdvTier 01819_T _TotalR ier2_Tot TotalRat y_Total ty_Total _TotalR ier3_Tot Control Variables atio alRatio io Ratio Ratio atio alRatio -none-a SUB_20181 Correlation 1.000 .672 .836 .691 .640 .722 .478 9_SchPriorit Significance . .000 .000 .000 .000 .000 .000 y_TotalRati (2-tailed) o df 0 532 532 532 532 532 532 ALT1_2018 Correlation .672 1.000 .621 .623 .622 .842 .759 19_Tier2_T Significance .000 . .000 .000 .000 .000 .000 otalRatio (2-tailed) df 532 0 532 532 532 532 532 SUB_20181 Correlation .836 .621 1.000 .647 .706 .642 .425 9_DataUse_ Significance .000 .000 . .000 .000 .000 .000 TotalRatio (2-tailed) df 532 532 0 532 532 532 532 SUB_20181 Correlation .691 .623 .647 1.000 .723 .619 .492 9_DistPriori Significance .000 .000 .000 . .000 .000 .000 ty_TotalRati (2-tailed) o df 532 532 532 0 532 532 532 SUB_20181 Correlation .640 .622 .706 .723 1.000 .587 .442 9_DistCapa Significance .000 .000 .000 .000 . .000 .000 city_TotalR (2-tailed) atio df 532 532 532 532 0 532 532 ALT1_2018 Correlation .722 .842 .642 .619 .587 1.000 .724 19_AdvTier Significance .000 .000 .000 .000 .000 . .000 _TotalRatio (2-tailed) df 532 532 532 532 532 0 532 ALT1_2018 Correlation .478 .759 .425 .492 .442 .724 1.000 19_Tier3_T Significance .000 .000 .000 .000 .000 .000 . otalRatio (2-tailed) 192 SUB_20 SUB_20 SUB_20 SUB_20 ALT1_2 1819_Sc ALT1_2 1819_D 1819_Di 1819_Di 01819_ ALT1_2 hPriority 01819_T ataUse_ stPriorit stCapaci AdvTier 01819_T _TotalR ier2_Tot TotalRat y_Total ty_Total _TotalR ier3_Tot Control Variables atio alRatio io Ratio Ratio atio alRatio df 532 532 532 532 532 532 0 SUB_201819 SUB_20181 Correlation 1.000 .064 _DataUse_T 9_SchPriorit Significance . .139 otalRatio & y_TotalRati (2-tailed) SUB_201819 o df 0 527 _DistPriority _TotalRatio ALT1_2018 Correlation .064 1.000 & 19_Tier2_T Significance .139 . SUB_201819 otalRatio (2-tailed) _DistCapacit df 527 0 y_TotalRatio & ALT1_20181 9_AdvTier_T otalRatio & ALT1_20181 9_Tier3_Tota lRatio a. Cells contain zero-order (Pearson) correlations. 193 PARTIAL CORR /VARIABLES= SUB_201819_SchPriority_TotalRatio ALT1_201819_Tier3_TotalRatio BY SUB_201819_DataUse_TotalRatio SUB_201819_DistPriority_TotalRatio SUB_201819_DistCapacity_TotalRatio ALT1_201819_AdvTier_TotalRatio ALT1_201819_Tier2_TotalRatio /SIGNIFICANCE=TWOTAIL /STATISTICS=DESCRIPTIVES CORR /MISSING=LISTWISE. Descriptive Statistics Mean Std. Deviation N SUB_201819_SchPriority_TotalRatio .7925 .18830 534 ALT1_201819_Tier3_TotalRatio .5464 .27184 534 SUB_201819_DataUse_TotalRatio .8133 .19671 534 SUB_201819_DistPriority_TotalRatio .7383 .22568 534 SUB_201819_DistCapacity_TotalRatio .7682 .23864 534 ALT1_201819_AdvTier_TotalRatio .6308 .19873 534 ALT1_201819_Tier2_TotalRatio .6543 .22470 534 Correlations SUB_2 SUB_2 SUB_2 01819_ ALT1_ SUB_2 01819_ 01819_ ALT1_ ALT1_ SchPri 201819 01819_ DistPri DistCa 201819 201819 ority_T _Tier3 DataUs ority_T pacity_ _AdvT _Tier2 otalRat _Total e_Tota otalRat TotalR ier_Tot _Total Control Variables io Ratio lRatio io atio alRatio Ratio -none-a SUB_201819_ Correlation 1.000 .478 .836 .691 .640 .722 .672 SchPriority_T otalRatio Significance . .000 .000 .000 .000 .000 .000 (2-tailed) df 0 532 532 532 532 532 532 ALT1_201819 Correlation .478 1.000 .425 .492 .442 .724 .759 _Tier3_TotalR atio Significance .000 . .000 .000 .000 .000 .000 (2-tailed) df 532 0 532 532 532 532 532 SUB_201819_ Correlation .836 .425 1.000 .647 .706 .642 .621 DataUse_Tota lRatio Significance .000 .000 . .000 .000 .000 .000 (2-tailed) df 532 532 0 532 532 532 532 SUB_201819_ Correlation .691 .492 .647 1.000 .723 .619 .623 DistPriority_T otalRatio Significance .000 .000 .000 . .000 .000 .000 (2-tailed) df 532 532 532 0 532 532 532 SUB_201819_ Correlation .640 .442 .706 .723 1.000 .587 .622 DistCapacity_ TotalRatio Significance .000 .000 .000 .000 . .000 .000 (2-tailed) df 532 532 532 532 0 532 532 ALT1_201819 Correlation .722 .724 .642 .619 .587 1.000 .842 _AdvTier_Tot alRatio Significance .000 .000 .000 .000 .000 . .000 (2-tailed) 194 SUB_2 SUB_2 SUB_2 01819_ ALT1_ SUB_2 01819_ 01819_ ALT1_ ALT1_ SchPri 201819 01819_ DistPri DistCa 201819 201819 ority_T _Tier3 DataUs ority_T pacity_ _AdvT _Tier2 otalRat _Total e_Tota otalRat TotalR ier_Tot _Total Control Variables io Ratio lRatio io atio alRatio Ratio df 532 532 532 532 532 0 532 ALT1_201819 Correlation .672 .759 .621 .623 .622 .842 1.000 _Tier2_TotalR atio Significance .000 .000 .000 .000 .000 .000 . (2-tailed) df 532 532 532 532 532 532 0 SUB_201819_Da SUB_201819_ Correlation 1.000 -.109 taUse_TotalRatio SchPriority_T & otalRatio Significance . .013 SUB_201819_Dis (2-tailed) tPriority_TotalRat df 0 527 io & ALT1_201819 Correlation -.109 1.000 SUB_201819_Dis _Tier3_TotalR Significance .013 . tCapacity_TotalR atio (2-tailed) atio & df 527 0 ALT1_201819_A dvTier_TotalRati o & ALT1_201819_Ti er2_TotalRatio a. Cells contain zero-order (Pearson) correlations. 195 *Part 3. SUB Team data use. PARTIAL CORR /VARIABLES=SUB_201819_DataUse_TotalRatio SUB_201819_DistPriority_TotalRatio BY SUB_201819_SchPriority_TotalRatio SUB_201819_DistCapacity_TotalRatio ALT1_201819_OverallRatio /SIGNIFICANCE=TWOTAIL /STATISTICS=DESCRIPTIVES CORR /MISSING=LISTWISE. Descriptive Statistics Mean Std. Deviation N SUB_201819_DataUse_TotalRatio .8147 .19390 610 SUB_201819_DistPriority_TotalRatio .7392 .22316 610 SUB_201819_SchPriority_TotalRatio .7931 .18618 610 SUB_201819_DistCapacity_TotalRatio .7682 .23838 610 Altitude Overall Score .6113 .21295 610 Correlations SUB_201 SUB_201 SUB_201 SUB_201 819_Data 819_DistP 819_SchP 819_DistC Altitude Use_Total riority_To riority_To apacity_T Overall Control Variables Ratio talRatio talRatio otalRatio Score -none-a SUB_20181 Correlation 1.000 .649 .823 .695 .588 9_DataUse_ Significance . .000 .000 .000 .000 TotalRatio (2-tailed) df 0 608 608 608 608 SUB_20181 Correlation .649 1.000 .695 .723 .610 9_DistPriori Significance .000 . .000 .000 .000 ty_TotalRati (2-tailed) o df 608 0 608 608 608 SUB_20181 Correlation .823 .695 1.000 .641 .660 9_SchPriorit Significance .000 .000 . .000 .000 y_TotalRati (2-tailed) o df 608 608 0 608 608 SUB_20181 Correlation .695 .723 .641 1.000 .576 9_DistCapa Significance .000 .000 .000 . .000 city_TotalR (2-tailed) atio df 608 608 608 0 608 Altitude Correlation .588 .610 .660 .576 1.000 Overall Significance .000 .000 .000 .000 . Score (2-tailed) df 608 608 608 608 0 SUB_201819_Sch SUB_20181 Correlation 1.000 -.007 Priority_TotalRati 9_DataUse_ Significance . .861 o & TotalRatio (2-tailed) SUB_201819_Dist df 0 605 Capacity_TotalRat SUB_20181 Correlation -.007 1.000 io & Altitude 9_DistPriori Overall Score Significance .861 . ty_TotalRati (2-tailed) o df 605 0 a. Cells contain zero-order (Pearson) correlations. PARTIAL CORR /VARIABLES=SUB_201819_DataUse_TotalRatio SUB_201819_DistCapacity_TotalRatio BY 196 SUB_201819_SchPriority_TotalRatio SUB_201819_DistPriority_TotalRatio ALT1_201819_OverallRatio /SIGNIFICANCE=TWOTAIL /STATISTICS=DESCRIPTIVES CORR /MISSING=LISTWISE. Descriptive Statistics Mean Std. Deviation N SUB_201819_DataUse_TotalRatio .8147 .19390 610 SUB_201819_DistCapacity_TotalRatio .7682 .23838 610 SUB_201819_SchPriority_TotalRatio .7931 .18618 610 SUB_201819_DistPriority_TotalRatio .7392 .22316 610 Altitude Overall Score .6113 .21295 610 Correlations SUB_201 SUB_201 SUB_201 SUB_201 819_Data 819_Dist 819_SchP 819_DistP Altitude Use_Total Capacity_ riority_To riority_To Overall Control Variables Ratio TotalRatio talRatio talRatio Score -none-a SUB_201819_D Correlation 1.000 .695 .823 .649 .588 ataUse_TotalRat Significance . .000 .000 .000 .000 io (2-tailed) df 0 608 608 608 608 SUB_201819_D Correlation .695 1.000 .641 .723 .576 istCapacity_Tota Significance .000 . .000 .000 .000 lRatio (2-tailed) df 608 0 608 608 608 SUB_201819_S Correlation .823 .641 1.000 .695 .660 chPriority_Total Significance .000 .000 . .000 .000 Ratio (2-tailed) df 608 608 0 608 608 SUB_201819_D Correlation .649 .723 .695 1.000 .610 istPriority_Total Significance .000 .000 .000 . .000 Ratio (2-tailed) df 608 608 608 0 608 Altitude Overall Correlation .588 .576 .660 .610 1.000 Score Significance .000 .000 .000 .000 . (2-tailed) df 608 608 608 608 0 SUB_201819_S SUB_201819_D Correlation 1.000 .339 chPriority_Total ataUse_TotalRat Significance . .000 Ratio & io (2-tailed) SUB_201819_D df 0 605 istPriority_Total SUB_201819_D Correlation .339 1.000 Ratio & Altitude istCapacity_Tota Significance .000 . Overall Score lRatio (2-tailed) df 605 0 a. Cells contain zero-order (Pearson) correlations. 197 PARTIAL CORR /VARIABLES=SUB_201819_DataUse_TotalRatio ALT1_201819_OverallRatio BY SUB_201819_SchPriority_TotalRatio SUB_201819_DistPriority_TotalRatio SUB_201819_DistCapacity_TotalRatio /SIGNIFICANCE=TWOTAIL /STATISTICS=DESCRIPTIVES CORR /MISSING=LISTWISE. Descriptive Statistics Mean Std. Deviation N SUB_201819_DataUse_TotalRatio .8147 .19390 610 Altitude Overall Score .6113 .21295 610 SUB_201819_SchPriority_TotalRatio .7931 .18618 610 SUB_201819_DistPriority_TotalRatio .7392 .22316 610 SUB_201819_DistCapacity_TotalRatio .7682 .23838 610 Correlations SUB_201 SUB_201 SUB_201 SUB_201 819_Data Altitude 819_SchP 819_DistP 819_DistC Use_Total Overall riority_To riority_To apacity_T Control Variables Ratio Score talRatio talRatio otalRatio -none-a SUB_201819_ Correlation 1.000 .588 .823 .649 .695 DataUse_Tota lRatio Significance . .000 .000 .000 .000 (2-tailed) df 0 608 608 608 608 Altitude Correlation .588 1.000 .660 .610 .576 Overall Score Significance .000 . .000 .000 .000 (2-tailed) df 608 0 608 608 608 SUB_201819_ Correlation .823 .660 1.000 .695 .641 SchPriority_T otalRatio Significance .000 .000 . .000 .000 (2-tailed) df 608 608 0 608 608 SUB_201819_ Correlation .649 .610 .695 1.000 .723 DistPriority_T Significance .000 .000 .000 . .000 otalRatio (2-tailed) df 608 608 608 0 608 SUB_201819_ Correlation .695 .576 .641 .723 1.000 DistCapacity_ Significance .000 .000 .000 .000 . TotalRatio (2-tailed) df 608 608 608 608 0 SUB_201819_S SUB_201819_ Correlation 1.000 .004 chPriority_Total DataUse_Tota Significance . .930 Ratio & lRatio (2-tailed) SUB_201819_D df 0 605 istPriority_Total Altitude Correlation .004 1.000 Ratio & Overall Score SUB_201819_D Significance .930 . istCapacity_Tota (2-tailed) lRatio df 605 0 a. Cells contain zero-order (Pearson) correlations. PARTIAL CORR /VARIABLES=SUB_201819_DataUse_TotalRatio ALT1_201819_AdvTier_TotalRatio BY 198 SUB_201819_SchPriority_TotalRatio SUB_201819_DistPriority_TotalRatio SUB_201819_DistCapacity_TotalRatio ALT1_201819_Tier2_TotalRatio ALT1_201819_Tier3_TotalRatio /SIGNIFICANCE=TWOTAIL /STATISTICS=DESCRIPTIVES CORR /MISSING=LISTWISE. Descriptive Statistics Mean Std. Deviation N SUB_201819_DataUse_TotalRatio .8133 .19671 534 ALT1_201819_AdvTier_TotalRatio .6308 .19873 534 SUB_201819_SchPriority_TotalRatio .7925 .18830 534 SUB_201819_DistPriority_TotalRatio .7383 .22568 534 SUB_201819_DistCapacity_TotalRatio .7682 .23864 534 ALT1_201819_Tier2_TotalRatio .6543 .22470 534 ALT1_201819_Tier3_TotalRatio .5464 .27184 534 Correlations SUB_20 ALT1_2 SUB_20 SUB_20 SUB_20 ALT1_2 ALT1_2 1819_D 01819_ 1819_S 1819_D 1819_D 01819_ 01819_ ataUse_ AdvTier chPriori istPriori istCapac Tier2_T Tier3_T TotalRat _TotalR ty_Total ty_Total ity_Tota otalRati otalRati Control Variables io atio Ratio Ratio lRatio o o -none-a SUB_20181 Correlation 1.000 .642 .836 .647 .706 .621 .425 9_DataUse_ Significance . .000 .000 .000 .000 .000 .000 TotalRatio (2-tailed) df 0 532 532 532 532 532 532 ALT1_2018 Correlation .642 1.000 .722 .619 .587 .842 .724 19_AdvTier Significance .000 . .000 .000 .000 .000 .000 _TotalRatio (2-tailed) df 532 0 532 532 532 532 532 SUB_20181 Correlation .836 .722 1.000 .691 .640 .672 .478 9_SchPriorit Significance .000 .000 . .000 .000 .000 .000 y_TotalRati (2-tailed) o df 532 532 0 532 532 532 532 SUB_20181 Correlation .647 .619 .691 1.000 .723 .623 .492 9_DistPriori Significance .000 .000 .000 . .000 .000 .000 ty_TotalRati (2-tailed) o df 532 532 532 0 532 532 532 SUB_20181 Correlation .706 .587 .640 .723 1.000 .622 .442 9_DistCapa Significance .000 .000 .000 .000 . .000 .000 city_TotalR (2-tailed) atio df 532 532 532 532 0 532 532 ALT1_2018 Correlation .621 .842 .672 .623 .622 1.000 .759 19_Tier2_T Significance .000 .000 .000 .000 .000 . .000 otalRatio (2-tailed) df 532 532 532 532 532 0 532 ALT1_2018 Correlation .425 .724 .478 .492 .442 .759 1.000 19_Tier3_T Significance .000 .000 .000 .000 .000 .000 . otalRatio (2-tailed) df 532 532 532 532 532 532 0 SUB_20181 SUB_20181 Correlation 1.000 .013 9_SchPriorit 9_DataUse_ Significance . .760 y_TotalRati TotalRatio (2-tailed) 199 SUB_20 ALT1_2 SUB_20 SUB_20 SUB_20 ALT1_2 ALT1_2 1819_D 01819_ 1819_S 1819_D 1819_D 01819_ 01819_ ataUse_ AdvTier chPriori istPriori istCapac Tier2_T Tier3_T TotalRat _TotalR ty_Total ty_Total ity_Tota otalRati otalRati Control Variables io atio Ratio Ratio lRatio o o o & df 0 527 SUB_20181 ALT1_2018 Correlation .013 1.000 9_DistPriori 19_AdvTier Significance .760 . ty_TotalRati _TotalRatio (2-tailed) o & df 527 0 SUB_20181 9_DistCapa city_TotalR atio & ALT1_2018 19_Tier2_T otalRatio & ALT1_2018 19_Tier3_T otalRatio a. Cells contain zero-order (Pearson) correlations. 200 PARTIAL CORR /VARIABLES=SUB_201819_DataUse_TotalRatio ALT1_201819_Tier2_TotalRatio BY SUB_201819_SchPriority_TotalRatio SUB_201819_DistPriority_TotalRatio SUB_201819_DistCapacity_TotalRatio ALT1_201819_AdvTier_TotalRatio ALT1_201819_Tier3_TotalRatio /SIGNIFICANCE=TWOTAIL /STATISTICS=DESCRIPTIVES CORR /MISSING=LISTWISE. Descriptive Statistics Mean Std. Deviation N SUB_201819_DataUse_TotalRatio .8133 .19671 534 ALT1_201819_Tier2_TotalRatio .6543 .22470 534 SUB_201819_SchPriority_TotalRatio .7925 .18830 534 SUB_201819_DistPriority_TotalRatio .7383 .22568 534 SUB_201819_DistCapacity_TotalRatio .7682 .23864 534 ALT1_201819_AdvTier_TotalRatio .6308 .19873 534 ALT1_201819_Tier3_TotalRatio .5464 .27184 534 Correlations SUB_2 SUB_2 ALT1_ SUB_2 SUB_2 01819_ ALT1_ ALT1_2 01819_ 201819 01819_ 01819_ DistCap 201819 01819_ DataUs _Tier2_ SchPrio DistPrio acity_T _AdvTi Tier3_T e_Total TotalRa rity_Tot rity_Tot otalRati er_Total otalRati Control Variables Ratio tio alRatio alRatio o Ratio o -none-a SUB_20181 Correlation 1.000 .621 .836 .647 .706 .642 .425 9_DataUse_ Significance . .000 .000 .000 .000 .000 .000 TotalRatio (2-tailed) df 0 532 532 532 532 532 532 ALT1_2018 Correlation .621 1.000 .672 .623 .622 .842 .759 19_Tier2_T Significance .000 . .000 .000 .000 .000 .000 otalRatio (2-tailed) df 532 0 532 532 532 532 532 SUB_20181 Correlation .836 .672 1.000 .691 .640 .722 .478 9_SchPriorit Significance .000 .000 . .000 .000 .000 .000 y_TotalRati (2-tailed) o df 532 532 0 532 532 532 532 SUB_20181 Correlation .647 .623 .691 1.000 .723 .619 .492 9_DistPriori Significance .000 .000 .000 . .000 .000 .000 ty_TotalRati (2-tailed) o df 532 532 532 0 532 532 532 SUB_20181 Correlation .706 .622 .640 .723 1.000 .587 .442 9_DistCapa Significance .000 .000 .000 .000 . .000 .000 city_TotalR (2-tailed) atio df 532 532 532 532 0 532 532 ALT1_2018 Correlation .642 .842 .722 .619 .587 1.000 .724 19_AdvTier Significance .000 .000 .000 .000 .000 . .000 _TotalRatio (2-tailed) df 532 532 532 532 532 0 532 ALT1_2018 Correlation .425 .759 .478 .492 .442 .724 1.000 19_Tier3_T Significance .000 .000 .000 .000 .000 .000 . otalRatio (2-tailed) 201 SUB_2 SUB_2 ALT1_ SUB_2 SUB_2 01819_ ALT1_ ALT1_2 01819_ 201819 01819_ 01819_ DistCap 201819 01819_ DataUs _Tier2_ SchPrio DistPrio acity_T _AdvTi Tier3_T e_Total TotalRa rity_Tot rity_Tot otalRati er_Total otalRati Control Variables Ratio tio alRatio alRatio o Ratio o df 532 532 532 532 532 532 0 SUB_201819 SUB_20181 Correlation 1.000 .032 _SchPriority 9_DataUse_ Significance . .463 _TotalRatio TotalRatio (2-tailed) & df 0 527 SUB_201819 _DistPriority ALT1_2018 Correlation .032 1.000 _TotalRatio 19_Tier2_T Significance .463 . & otalRatio (2-tailed) SUB_201819 df 527 0 _DistCapacit y_TotalRatio & ALT1_20181 9_AdvTier_T otalRatio & ALT1_20181 9_Tier3_Tota lRatio a. Cells contain zero-order (Pearson) correlations. 202 PARTIAL CORR /VARIABLES=SUB_201819_DataUse_TotalRatio ALT1_201819_Tier3_TotalRatio BY SUB_201819_SchPriority_TotalRatio SUB_201819_DistPriority_TotalRatio SUB_201819_DistCapacity_TotalRatio ALT1_201819_AdvTier_TotalRatio ALT1_201819_Tier2_TotalRatio /SIGNIFICANCE=TWOTAIL /STATISTICS=DESCRIPTIVES CORR /MISSING=LISTWISE. Descriptive Statistics Mean Std. Deviation N SUB_201819_DataUse_TotalRatio .8133 .19671 534 ALT1_201819_Tier3_TotalRatio .5464 .27184 534 SUB_201819_SchPriority_TotalRatio .7925 .18830 534 SUB_201819_DistPriority_TotalRatio .7383 .22568 534 SUB_201819_DistCapacity_TotalRatio .7682 .23864 534 ALT1_201819_AdvTier_TotalRatio .6308 .19873 534 ALT1_201819_Tier2_TotalRatio .6543 .22470 534 Correlations SUB_2 SUB_2 SUB_2 SUB_2 ALT1_ 01819_ 01819_ 01819_ ALT1_ ALT1_ 01819_ 201819 SchPri DistPri DistCa 201819 201819 DataUs _Tier3 ority_T ority_T pacity_ _AdvTi _Tier2_ e_Total _Total otalRat otalRat TotalR er_Tota TotalR Control Variables Ratio Ratio io io atio lRatio atio -none-a SUB_201819 Correlation 1.000 .425 .836 .647 .706 .642 .621 _DataUse_T Significance . .000 .000 .000 .000 .000 .000 otalRatio (2-tailed) df 0 532 532 532 532 532 532 ALT1_20181 Correlation .425 1.000 .478 .492 .442 .724 .759 9_Tier3_Tota Significance .000 . .000 .000 .000 .000 .000 lRatio (2-tailed) df 532 0 532 532 532 532 532 SUB_201819 Correlation .836 .478 1.000 .691 .640 .722 .672 _SchPriority Significance .000 .000 . .000 .000 .000 .000 _TotalRatio (2-tailed) df 532 532 0 532 532 532 532 SUB_201819 Correlation .647 .492 .691 1.000 .723 .619 .623 _DistPriority Significance .000 .000 .000 . .000 .000 .000 _TotalRatio (2-tailed) df 532 532 532 0 532 532 532 SUB_201819 Correlation .706 .442 .640 .723 1.000 .587 .622 _DistCapacit Significance .000 .000 .000 .000 . .000 .000 y_TotalRatio (2-tailed) df 532 532 532 532 0 532 532 ALT1_20181 Correlation .642 .724 .722 .619 .587 1.000 .842 9_AdvTier_T Significance .000 .000 .000 .000 .000 . .000 otalRatio (2-tailed) df 532 532 532 532 532 0 532 ALT1_20181 Correlation .621 .759 .672 .623 .622 .842 1.000 9_Tier2_Tota Significance .000 .000 .000 .000 .000 .000 . lRatio (2-tailed) df 532 532 532 532 532 532 0 203 SUB_2 SUB_2 SUB_2 SUB_2 ALT1_ 01819_ 01819_ 01819_ ALT1_ ALT1_ 01819_ 201819 SchPri DistPri DistCa 201819 201819 DataUs _Tier3 ority_T ority_T pacity_ _AdvTi _Tier2_ e_Total _Total otalRat otalRat TotalR er_Tota TotalR Control Variables Ratio Ratio io io atio lRatio atio SUB_201819_S SUB_201819 Correlation 1.000 -.049 chPriority_Total _DataUse_T Significance . .265 Ratio & otalRatio (2-tailed) SUB_201819_D df 0 527 istPriority_Total ALT1_20181 Correlation -.049 1.000 Ratio & 9_Tier3_Tota Significance .265 . SUB_201819_D lRatio (2-tailed) istCapacity_Tota lRatio & df 527 0 ALT1_201819_ AdvTier_TotalR atio & ALT1_201819_ Tier2_TotalRati o a. Cells contain zero-order (Pearson) correlations. 204 *Part 4. SUB Dist Priority. PARTIAL CORR /VARIABLES=SUB_201819_DistPriority_TotalRatio SUB_201819_DistCapacity_TotalRatio BY SUB_201819_SchPriority_TotalRatio SUB_201819_DataUse_TotalRatio ALT1_201819_OverallRatio /SIGNIFICANCE=TWOTAIL /STATISTICS=DESCRIPTIVES CORR /MISSING=LISTWISE. Descriptive Statistics Mean Std. Deviation N SUB_201819_DistPriority_TotalRatio .7392 .22316 610 SUB_201819_DistCapacity_TotalRatio .7682 .23838 610 SUB_201819_SchPriority_TotalRatio .7931 .18618 610 SUB_201819_DataUse_TotalRatio .8147 .19390 610 Altitude Overall Score .6113 .21295 610 Correlations SUB_2018 SUB_2018 SUB_2018 SUB_2018 19_DistPri 19_DistCa 19_SchPri 19_DataUs Altitude ority_Total pacity_Tot ority_Total e_TotalRat Overall Control Variables Ratio alRatio Ratio io Score -none-a SUB_201819_ Correlation 1.000 .723 .695 .649 .610 DistPriority_T Significance . .000 .000 .000 .000 otalRatio (2-tailed) df 0 608 608 608 608 SUB_201819_ Correlation .723 1.000 .641 .695 .576 DistCapacity_ Significance .000 . .000 .000 .000 TotalRatio (2-tailed) df 608 0 608 608 608 SUB_201819_ Correlation .695 .641 1.000 .823 .660 SchPriority_T Significance .000 .000 . .000 .000 otalRatio (2-tailed) df 608 608 0 608 608 SUB_201819_ Correlation .649 .695 .823 1.000 .588 DataUse_Tota Significance .000 .000 .000 . .000 lRatio (2-tailed) df 608 608 608 0 608 Altitude Correlation .610 .576 .660 .588 1.000 Overall Score Significance .000 .000 .000 .000 . (2-tailed) df 608 608 608 608 0 SUB_201819_ SUB_201819_ Correlation 1.000 .438 SchPriority_T DistPriority_T Significance . .000 otalRatio & otalRatio (2-tailed) SUB_201819_ df 0 605 DataUse_Tota SUB_201819_ Correlation .438 1.000 lRatio & DistCapacity_ Significance .000 . Altitude TotalRatio (2-tailed) Overall Score df 605 0 a. Cells contain zero-order (Pearson) correlations. 205 PARTIAL CORR /VARIABLES=SUB_201819_DistPriority_TotalRatio ALT1_201819_OverallRatio BY SUB_201819_SchPriority_TotalRatio SUB_201819_DataUse_TotalRatio SUB_201819_DistCapacity_TotalRatio /SIGNIFICANCE=TWOTAIL /STATISTICS=DESCRIPTIVES CORR /MISSING=LISTWISE. Descriptive Statistics Mean Std. Deviation N SUB_201819_DistPriority_TotalRatio .7392 .22316 610 Altitude Overall Score .6113 .21295 610 SUB_201819_SchPriority_TotalRatio .7931 .18618 610 SUB_201819_DataUse_TotalRatio .8147 .19390 610 SUB_201819_DistCapacity_TotalRatio .7682 .23838 610 Correlations SUB_2018 SUB_2018 SUB_2018 SUB_2018 19_DistPri Altitude 19_SchPrio 19_DataUs 19_DistCap ority_Total Overall rity_TotalR e_TotalRati acity_Total Control Variables Ratio Score atio o Ratio -none-a SUB_201819 Correlation 1.000 .610 .695 .649 .723 _DistPriority Significance . .000 .000 .000 .000 _TotalRatio (2-tailed) df 0 608 608 608 608 Altitude Correlation .610 1.000 .660 .588 .576 Overall Score Significance .000 . .000 .000 .000 (2-tailed) df 608 0 608 608 608 SUB_201819 Correlation .695 .660 1.000 .823 .641 _SchPriority Significance .000 .000 . .000 .000 _TotalRatio (2-tailed) df 608 608 0 608 608 SUB_201819 Correlation .649 .588 .823 1.000 .695 _DataUse_T Significance .000 .000 .000 . .000 otalRatio (2-tailed) df 608 608 608 0 608 SUB_201819 Correlation .723 .576 .641 .695 1.000 _DistCapacit Significance .000 .000 .000 .000 . y_TotalRatio (2-tailed) df 608 608 608 608 0 SUB_201819_ SUB_201819 Correlation 1.000 .176 SchPriority_T _DistPriority Significance . .000 otalRatio & _TotalRatio (2-tailed) SUB_201819_ df 0 605 DataUse_Tota lRatio & Altitude Correlation .176 1.000 SUB_201819_ Overall Score Significance .000 . DistCapacity_ (2-tailed) TotalRatio df 605 0 a. Cells contain zero-order (Pearson) correlations. 206 PARTIAL CORR /VARIABLES=SUB_201819_DistPriority_TotalRatio ALT1_201819_AdvTier_TotalRatio BY SUB_201819_SchPriority_TotalRatio SUB_201819_DataUse_TotalRatio SUB_201819_DistCapacity_TotalRatio ALT1_201819_Tier2_TotalRatio ALT1_201819_Tier3_TotalRatio /SIGNIFICANCE=TWOTAIL /STATISTICS=DESCRIPTIVES CORR /MISSING=LISTWISE. Descriptive Statistics Mean Std. Deviation N SUB_201819_DistPriority_TotalRatio .7383 .22568 534 ALT1_201819_AdvTier_TotalRatio .6308 .19873 534 SUB_201819_SchPriority_TotalRatio .7925 .18830 534 SUB_201819_DataUse_TotalRatio .8133 .19671 534 SUB_201819_DistCapacity_TotalRatio .7682 .23864 534 ALT1_201819_Tier2_TotalRatio .6543 .22470 534 ALT1_201819_Tier3_TotalRatio .5464 .27184 534 Correlations SUB_2 SUB_2 SUB_2 01819_ ALT1_ 01819_ SUB_2 01819_ ALT1_ ALT1_ DistPri 201819 SchPri 01819_ DistCa 201819 201819 ority_T _AdvT ority_T DataUs pacity_ _Tier2 _Tier3 otalRat ier_Tot otalRat e_Total TotalR _Total _Total Control Variables io alRatio io Ratio atio Ratio Ratio -none-a SUB_201819_ Correlation 1.000 .619 .691 .647 .723 .623 .492 DistPriority_T Significance . .000 .000 .000 .000 .000 .000 otalRatio (2-tailed) df 0 532 532 532 532 532 532 ALT1_201819 Correlation .619 1.000 .722 .642 .587 .842 .724 _AdvTier_Tot Significance .000 . .000 .000 .000 .000 .000 alRatio (2-tailed) df 532 0 532 532 532 532 532 SUB_201819_ Correlation .691 .722 1.000 .836 .640 .672 .478 SchPriority_T Significance .000 .000 . .000 .000 .000 .000 otalRatio (2-tailed) df 532 532 0 532 532 532 532 SUB_201819_ Correlation .647 .642 .836 1.000 .706 .621 .425 DataUse_Tota Significance .000 .000 .000 . .000 .000 .000 lRatio (2-tailed) df 532 532 532 0 532 532 532 SUB_201819_ Correlation .723 .587 .640 .706 1.000 .622 .442 DistCapacity_ Significance .000 .000 .000 .000 . .000 .000 TotalRatio (2-tailed) df 532 532 532 532 0 532 532 ALT1_201819 Correlation .623 .842 .672 .621 .622 1.000 .759 _Tier2_TotalR Significance .000 .000 .000 .000 .000 . .000 atio (2-tailed) df 532 532 532 532 532 0 532 ALT1_201819 Correlation .492 .724 .478 .425 .442 .759 1.000 _Tier3_TotalR Significance .000 .000 .000 .000 .000 .000 . atio (2-tailed) df 532 532 532 532 532 532 0 207 SUB_2 SUB_2 SUB_2 01819_ ALT1_ 01819_ SUB_2 01819_ ALT1_ ALT1_ DistPri 201819 SchPri 01819_ DistCa 201819 201819 ority_T _AdvT ority_T DataUs pacity_ _Tier2 _Tier3 otalRat ier_Tot otalRat e_Total TotalR _Total _Total Control Variables io alRatio io Ratio atio Ratio Ratio SUB_201819_S SUB_201819_ Correlation 1.000 .025 chPriority_Total DistPriority_T Significance . .562 Ratio & otalRatio (2-tailed) SUB_201819_D df 0 527 ataUse_TotalRat ALT1_201819 Correlation .025 1.000 io & _AdvTier_Tot Significance .562 . SUB_201819_D alRatio (2-tailed) istCapacity_Tota lRatio & df 527 0 ALT1_201819_ Tier2_TotalRati o & ALT1_201819_ Tier3_TotalRati o a. Cells contain zero-order (Pearson) correlations. 208 PARTIAL CORR /VARIABLES=SUB_201819_DistPriority_TotalRatio ALT1_201819_Tier2_TotalRatio BY SUB_201819_SchPriority_TotalRatio SUB_201819_DataUse_TotalRatio SUB_201819_DistCapacity_TotalRatio ALT1_201819_AdvTier_TotalRatio ALT1_201819_Tier3_TotalRatio /SIGNIFICANCE=TWOTAIL /STATISTICS=DESCRIPTIVES CORR /MISSING=LISTWISE. Descriptive Statistics Mean Std. Deviation N SUB_201819_DistPriority_TotalRatio .7383 .22568 534 ALT1_201819_Tier2_TotalRatio .6543 .22470 534 SUB_201819_SchPriority_TotalRatio .7925 .18830 534 SUB_201819_DataUse_TotalRatio .8133 .19671 534 SUB_201819_DistCapacity_TotalRatio .7682 .23864 534 ALT1_201819_AdvTier_TotalRatio .6308 .19873 534 ALT1_201819_Tier3_TotalRatio .5464 .27184 534 Correlations SUB_2 SUB_2 SUB_2 01819_ ALT1_ 01819_ SUB_2 01819_ ALT1_ ALT1_ DistPri 201819 SchPri 01819_ DistCa 201819 201819 ority_T _Tier2 ority_T DataUs pacity_ _AdvTi _Tier3_ otalRati _Total otalRat e_Total TotalR er_Tota TotalR Control Variables o Ratio io Ratio atio lRatio atio -none-a SUB_201819 Correlation 1.000 .623 .691 .647 .723 .619 .492 _DistPriority Significance . .000 .000 .000 .000 .000 .000 _TotalRatio (2-tailed) df 0 532 532 532 532 532 532 ALT1_20181 Correlation .623 1.000 .672 .621 .622 .842 .759 9_Tier2_Tota Significance .000 . .000 .000 .000 .000 .000 lRatio (2-tailed) df 532 0 532 532 532 532 532 SUB_201819 Correlation .691 .672 1.000 .836 .640 .722 .478 _SchPriority Significance .000 .000 . .000 .000 .000 .000 _TotalRatio (2-tailed) df 532 532 0 532 532 532 532 SUB_201819 Correlation .647 .621 .836 1.000 .706 .642 .425 _DataUse_T Significance .000 .000 .000 . .000 .000 .000 otalRatio (2-tailed) df 532 532 532 0 532 532 532 SUB_201819 Correlation .723 .622 .640 .706 1.000 .587 .442 _DistCapacit Significance .000 .000 .000 .000 . .000 .000 y_TotalRatio (2-tailed) df 532 532 532 532 0 532 532 ALT1_20181 Correlation .619 .842 .722 .642 .587 1.000 .724 9_AdvTier_T Significance .000 .000 .000 .000 .000 . .000 otalRatio (2-tailed) df 532 532 532 532 532 0 532 ALT1_20181 Correlation .492 .759 .478 .425 .442 .724 1.000 9_Tier3_Tota Significance .000 .000 .000 .000 .000 .000 . lRatio (2-tailed) df 532 532 532 532 532 532 0 209 SUB_2 SUB_2 SUB_2 01819_ ALT1_ 01819_ SUB_2 01819_ ALT1_ ALT1_ DistPri 201819 SchPri 01819_ DistCa 201819 201819 ority_T _Tier2 ority_T DataUs pacity_ _AdvTi _Tier3_ otalRati _Total otalRat e_Total TotalR er_Tota TotalR Control Variables o Ratio io Ratio atio lRatio atio SUB_201819_S SUB_201819 Correlation 1.000 .033 chPriority_Total _DistPriority Significance . .448 Ratio & _TotalRatio (2-tailed) SUB_201819_D df 0 527 ataUse_TotalRat ALT1_20181 Correlation .033 1.000 io & 9_Tier2_Tota Significance .448 . SUB_201819_D lRatio (2-tailed) istCapacity_Tota lRatio & df 527 0 ALT1_201819_ AdvTier_TotalR atio & ALT1_201819_ Tier3_TotalRati o a. Cells contain zero-order (Pearson) correlations. 210 PARTIAL CORR /VARIABLES=SUB_201819_DistPriority_TotalRatio ALT1_201819_Tier3_TotalRatio BY SUB_201819_SchPriority_TotalRatio SUB_201819_DataUse_TotalRatio SUB_201819_DistCapacity_TotalRatio ALT1_201819_AdvTier_TotalRatio ALT1_201819_Tier2_TotalRatio /SIGNIFICANCE=TWOTAIL /STATISTICS=DESCRIPTIVES CORR /MISSING=LISTWISE. Descriptive Statistics Mean Std. Deviation N SUB_201819_DistPriority_TotalRatio .7383 .22568 534 ALT1_201819_Tier3_TotalRatio .5464 .27184 534 SUB_201819_SchPriority_TotalRatio .7925 .18830 534 SUB_201819_DataUse_TotalRatio .8133 .19671 534 SUB_201819_DistCapacity_TotalRatio .7682 .23864 534 ALT1_201819_AdvTier_TotalRatio .6308 .19873 534 ALT1_201819_Tier2_TotalRatio .6543 .22470 534 Correlations SUB_2 SUB_2 ALT1_ SUB_2 SUB_2 01819_ ALT1_ ALT1_ 01819_ 201819 01819_ 01819_ DistCap 201819 201819 DistPrio _Tier3_ SchPrio DataUs acity_T _AdvTi _Tier2_ rity_Tot TotalRa rity_Tot e_Total otalRati er_Tota TotalRa Control Variables alRatio tio alRatio Ratio o lRatio tio -none-a SUB_20181 Correlation 1.000 .492 .691 .647 .723 .619 .623 9_DistPriori Significance . .000 .000 .000 .000 .000 .000 ty_TotalRati (2-tailed) o df 0 532 532 532 532 532 532 ALT1_2018 Correlation .492 1.000 .478 .425 .442 .724 .759 19_Tier3_T Significance .000 . .000 .000 .000 .000 .000 otalRatio (2-tailed) df 532 0 532 532 532 532 532 SUB_20181 Correlation .691 .478 1.000 .836 .640 .722 .672 9_SchPriorit Significance .000 .000 . .000 .000 .000 .000 y_TotalRati (2-tailed) o df 532 532 0 532 532 532 532 SUB_20181 Correlation .647 .425 .836 1.000 .706 .642 .621 9_DataUse_ Significance .000 .000 .000 . .000 .000 .000 TotalRatio (2-tailed) df 532 532 532 0 532 532 532 SUB_20181 Correlation .723 .442 .640 .706 1.000 .587 .622 9_DistCapa Significance .000 .000 .000 .000 . .000 .000 city_TotalR (2-tailed) atio df 532 532 532 532 0 532 532 ALT1_2018 Correlation .619 .724 .722 .642 .587 1.000 .842 19_AdvTier Significance .000 .000 .000 .000 .000 . .000 _TotalRatio (2-tailed) df 532 532 532 532 532 0 532 ALT1_2018 Correlation .623 .759 .672 .621 .622 .842 1.000 19_Tier2_T Significance .000 .000 .000 .000 .000 .000 . otalRatio (2-tailed) df 532 532 532 532 532 532 0 211 SUB_2 SUB_2 ALT1_ SUB_2 SUB_2 01819_ ALT1_ ALT1_ 01819_ 201819 01819_ 01819_ DistCap 201819 201819 DistPrio _Tier3_ SchPrio DataUs acity_T _AdvTi _Tier2_ rity_Tot TotalRa rity_Tot e_Total otalRati er_Tota TotalRa Control Variables alRatio tio alRatio Ratio o lRatio tio SUB_201819_ SUB_20181 Correlation 1.000 .092 SchPriority_T 9_DistPriori Significance . .034 otalRatio & ty_TotalRati (2-tailed) SUB_201819_ o df 0 527 DataUse_Tota ALT1_2018 Correlation .092 1.000 lRatio & 19_Tier3_T Significance .034 . SUB_201819_ otalRatio (2-tailed) DistCapacity_ TotalRatio & df 527 0 ALT1_201819 _AdvTier_Tot alRatio & ALT1_201819 _Tier2_TotalR atio a. Cells contain zero-order (Pearson) correlations. 212 *Part 5. SUB District Capacity. PARTIAL CORR /VARIABLES=SUB_201819_DistCapacity_TotalRatio ALT1_201819_OverallRatio BY SUB_201819_SchPriority_TotalRatio SUB_201819_DataUse_TotalRatio SUB_201819_DistPriority_TotalRatio /SIGNIFICANCE=TWOTAIL /STATISTICS=DESCRIPTIVES CORR /MISSING=LISTWISE. Descriptive Statistics Mean Std. Deviation N SUB_201819_DistCapacity_TotalRatio .7682 .23838 610 Altitude Overall Score .6113 .21295 610 SUB_201819_SchPriority_TotalRatio .7931 .18618 610 SUB_201819_DataUse_TotalRatio .8147 .19390 610 SUB_201819_DistPriority_TotalRatio .7392 .22316 610 Correlations SUB_2018 SUB_2018 SUB_2018 SUB_2018 19_DistCa Altitude 19_SchPri 19_DataU 19_DistPri pacity_Tot Overall ority_Tota se_TotalR ority_Total Control Variables alRatio Score lRatio atio Ratio -none-a SUB_201819 Correlation 1.000 .576 .641 .695 .723 _DistCapacit Significance . .000 .000 .000 .000 y_TotalRatio (2-tailed) df 0 608 608 608 608 Altitude Correlation .576 1.000 .660 .588 .610 Overall Score Significance .000 . .000 .000 .000 (2-tailed) df 608 0 608 608 608 SUB_201819 Correlation .641 .660 1.000 .823 .695 _SchPriority Significance .000 .000 . .000 .000 _TotalRatio (2-tailed) df 608 608 0 608 608 SUB_201819 Correlation .695 .588 .823 1.000 .649 _DataUse_T Significance .000 .000 .000 . .000 otalRatio (2-tailed) df 608 608 608 0 608 SUB_201819 Correlation .723 .610 .695 .649 1.000 _DistPriority Significance .000 .000 .000 .000 . _TotalRatio (2-tailed) df 608 608 608 608 0 SUB_201819_Sc SUB_201819 Correlation 1.000 .140 hPriority_TotalRa _DistCapacit Significance . .001 tio & y_TotalRatio (2-tailed) SUB_201819_Da df 0 605 taUse_TotalRatio & Altitude Correlation .140 1.000 SUB_201819_Dis Overall Score Significance .001 . tPriority_TotalRat (2-tailed) io df 605 0 a. Cells contain zero-order (Pearson) correlations. 213 PARTIAL CORR /VARIABLES=SUB_201819_DistCapacity_TotalRatio ALT1_201819_AdvTier_TotalRatio BY SUB_201819_SchPriority_TotalRatio SUB_201819_DataUse_TotalRatio SUB_201819_DistPriority_TotalRatio ALT1_201819_Tier2_TotalRatio ALT1_201819_Tier3_TotalRatio /SIGNIFICANCE=TWOTAIL /STATISTICS=DESCRIPTIVES CORR /MISSING=LISTWISE. Descriptive Statistics Mean Std. Deviation N SUB_201819_DistCapacity_TotalRatio .7682 .23864 534 ALT1_201819_AdvTier_TotalRatio .6308 .19873 534 SUB_201819_SchPriority_TotalRatio .7925 .18830 534 SUB_201819_DataUse_TotalRatio .8133 .19671 534 SUB_201819_DistPriority_TotalRatio .7383 .22568 534 ALT1_201819_Tier2_TotalRatio .6543 .22470 534 ALT1_201819_Tier3_TotalRatio .5464 .27184 534 Correlations SUB_2 SUB_2 01819_ ALT1_ SUB_2 SUB_2 01819_ ALT1_ ALT1_ DistCa 201819 01819_ 01819_ DistPri 201819 201819 pacity_ _AdvTi SchPrio DataUs ority_T _Tier2_ _Tier3_ TotalRa er_Tota rity_To e_Total otalRati TotalR TotalRa Control Variables tio lRatio talRatio Ratio o atio tio -none-a SUB_201819 Correlation 1.000 .587 .640 .706 .723 .622 .442 _DistCapacit Significance . .000 .000 .000 .000 .000 .000 y_TotalRatio (2-tailed) df 0 532 532 532 532 532 532 ALT1_20181 Correlation .587 1.000 .722 .642 .619 .842 .724 9_AdvTier_T Significance .000 . .000 .000 .000 .000 .000 otalRatio (2-tailed) df 532 0 532 532 532 532 532 SUB_201819 Correlation .640 .722 1.000 .836 .691 .672 .478 _SchPriority Significance .000 .000 . .000 .000 .000 .000 _TotalRatio (2-tailed) df 532 532 0 532 532 532 532 SUB_201819 Correlation .706 .642 .836 1.000 .647 .621 .425 _DataUse_T Significance .000 .000 .000 . .000 .000 .000 otalRatio (2-tailed) df 532 532 532 0 532 532 532 SUB_201819 Correlation .723 .619 .691 .647 1.000 .623 .492 _DistPriority Significance .000 .000 .000 .000 . .000 .000 _TotalRatio (2-tailed) df 532 532 532 532 0 532 532 ALT1_20181 Correlation .622 .842 .672 .621 .623 1.000 .759 9_Tier2_Tota Significance .000 .000 .000 .000 .000 . .000 lRatio (2-tailed) df 532 532 532 532 532 0 532 ALT1_20181 Correlation .442 .724 .478 .425 .492 .759 1.000 9_Tier3_Tota Significance .000 .000 .000 .000 .000 .000 . lRatio (2-tailed) df 532 532 532 532 532 532 0 SUB_201819_ SUB_201819 Correlation 1.000 -.007 214 SUB_2 SUB_2 01819_ ALT1_ SUB_2 SUB_2 01819_ ALT1_ ALT1_ DistCa 201819 01819_ 01819_ DistPri 201819 201819 pacity_ _AdvTi SchPrio DataUs ority_T _Tier2_ _Tier3_ TotalRa er_Tota rity_To e_Total otalRati TotalR TotalRa Control Variables tio lRatio talRatio Ratio o atio tio SchPriority_Tot _DistCapacit Significance . .880 alRatio & y_TotalRatio (2-tailed) SUB_201819_ df 0 527 DataUse_Total ALT1_20181 Correlation -.007 1.000 Ratio & 9_AdvTier_T Significance .880 . SUB_201819_ otalRatio (2-tailed) DistPriority_To df 527 0 talRatio & ALT1_201819_ Tier2_TotalRati o & ALT1_201819_ Tier3_TotalRati o a. Cells contain zero-order (Pearson) correlations. 215 PARTIAL CORR /VARIABLES=SUB_201819_DistCapacity_TotalRatio ALT1_201819_Tier2_TotalRatio BY SUB_201819_SchPriority_TotalRatio SUB_201819_DataUse_TotalRatio SUB_201819_DistPriority_TotalRatio ALT1_201819_AdvTier_TotalRatio ALT1_201819_Tier3_TotalRatio /SIGNIFICANCE=TWOTAIL /STATISTICS=DESCRIPTIVES CORR /MISSING=LISTWISE. Descriptive Statistics Mean Std. Deviation N SUB_201819_DistCapacity_TotalRatio .7682 .23864 534 ALT1_201819_Tier2_TotalRatio .6543 .22470 534 SUB_201819_SchPriority_TotalRatio .7925 .18830 534 SUB_201819_DataUse_TotalRatio .8133 .19671 534 SUB_201819_DistPriority_TotalRatio .7383 .22568 534 ALT1_201819_AdvTier_TotalRatio .6308 .19873 534 ALT1_201819_Tier3_TotalRatio .5464 .27184 534 Correlations SUB_2 SUB_2 01819_ ALT1_ SUB_2 SUB_2 01819_ ALT1_ ALT1_ DistCap 201819 01819_ 01819_ DistPri 201819 201819 acity_T _Tier2_ SchPrio DataUs ority_T _AdvTi _Tier3_ otalRati TotalRa rity_Tot e_Total otalRati er_Tota TotalRa Control Variables o tio alRatio Ratio o lRatio tio -none-a SUB_201819 Correlation 1.000 .622 .640 .706 .723 .587 .442 _DistCapacit Significance . .000 .000 .000 .000 .000 .000 y_TotalRatio (2-tailed) df 0 532 532 532 532 532 532 ALT1_20181 Correlation .622 1.000 .672 .621 .623 .842 .759 9_Tier2_Tota Significance .000 . .000 .000 .000 .000 .000 lRatio (2-tailed) df 532 0 532 532 532 532 532 SUB_201819 Correlation .640 .672 1.000 .836 .691 .722 .478 _SchPriority _TotalRatio Significance .000 .000 . .000 .000 .000 .000 (2-tailed) df 532 532 0 532 532 532 532 SUB_201819 Correlation .706 .621 .836 1.000 .647 .642 .425 _DataUse_T otalRatio Significance .000 .000 .000 . .000 .000 .000 (2-tailed) df 532 532 532 0 532 532 532 SUB_201819 Correlation .723 .623 .691 .647 1.000 .619 .492 _DistPriority Significance .000 .000 .000 .000 . .000 .000 _TotalRatio (2-tailed) df 532 532 532 532 0 532 532 ALT1_20181 Correlation .587 .842 .722 .642 .619 1.000 .724 9_AdvTier_T Significance .000 .000 .000 .000 .000 . .000 otalRatio (2-tailed) df 532 532 532 532 532 0 532 ALT1_20181 Correlation .442 .759 .478 .425 .492 .724 1.000 9_Tier3_Tota Significance .000 .000 .000 .000 .000 .000 . lRatio (2-tailed) 216 SUB_2 SUB_2 01819_ ALT1_ SUB_2 SUB_2 01819_ ALT1_ ALT1_ DistCap 201819 01819_ 01819_ DistPri 201819 201819 acity_T _Tier2_ SchPrio DataUs ority_T _AdvTi _Tier3_ otalRati TotalRa rity_Tot e_Total otalRati er_Tota TotalRa Control Variables o tio alRatio Ratio o lRatio tio df 532 532 532 532 532 532 0 SUB_201819 SUB_201819 Correlation 1.000 .177 _SchPriority _DistCapacit _TotalRatio y_TotalRatio Significance . .000 (2-tailed) & SUB_201819 df 0 527 _DataUse_T ALT1_20181 Correlation .177 1.000 otalRatio & 9_Tier2_Tota SUB_201819 lRatio Significance .000 . _DistPriority (2-tailed) _TotalRatio df 527 0 & ALT1_20181 9_AdvTier_T otalRatio & ALT1_20181 9_Tier3_Tota lRatio a. Cells contain zero-order (Pearson) correlations. 217 PARTIAL CORR /VARIABLES=SUB_201819_DistCapacity_TotalRatio ALT1_201819_Tier3_TotalRatio BY SUB_201819_SchPriority_TotalRatio SUB_201819_DataUse_TotalRatio SUB_201819_DistPriority_TotalRatio ALT1_201819_AdvTier_TotalRatio ALT1_201819_Tier2_TotalRatio /SIGNIFICANCE=TWOTAIL /STATISTICS=DESCRIPTIVES CORR /MISSING=LISTWISE. Descriptive Statistics Mean Std. Deviation N SUB_201819_DistCapacity_TotalRatio .7682 .23864 534 ALT1_201819_Tier3_TotalRatio .5464 .27184 534 SUB_201819_SchPriority_TotalRatio .7925 .18830 534 SUB_201819_DataUse_TotalRatio .8133 .19671 534 SUB_201819_DistPriority_TotalRatio .7383 .22568 534 ALT1_201819_AdvTier_TotalRatio .6308 .19873 534 ALT1_201819_Tier2_TotalRatio .6543 .22470 534 Correlations SUB_2 01819_ ALT1_ SUB_2 SUB_2 SUB_2 ALT1_ ALT1_2 DistCap 201819 01819_ 01819_ 01819_ 201819 01819_ acity_T _Tier3_ SchPrio DataUs DistPrio _AdvTi Tier2_T otalRati TotalRa rity_Tot e_Total rity_Tot er_Total otalRati Control Variables o tio alRatio Ratio alRatio Ratio o -none-a SUB_20181 Correlation 1.000 .442 .640 .706 .723 .587 .622 9_DistCapa Significance . .000 .000 .000 .000 .000 .000 city_TotalR (2-tailed) atio df 0 532 532 532 532 532 532 ALT1_2018 Correlation .442 1.000 .478 .425 .492 .724 .759 19_Tier3_T Significance .000 . .000 .000 .000 .000 .000 otalRatio (2-tailed) df 532 0 532 532 532 532 532 SUB_20181 Correlation .640 .478 1.000 .836 .691 .722 .672 9_SchPriorit Significance .000 .000 . .000 .000 .000 .000 y_TotalRati (2-tailed) o df 532 532 0 532 532 532 532 SUB_20181 Correlation .706 .425 .836 1.000 .647 .642 .621 9_DataUse_ Significance .000 .000 .000 . .000 .000 .000 TotalRatio (2-tailed) df 532 532 532 0 532 532 532 SUB_20181 Correlation .723 .492 .691 .647 1.000 .619 .623 9_DistPriori Significance .000 .000 .000 .000 . .000 .000 ty_TotalRati (2-tailed) o df 532 532 532 532 0 532 532 ALT1_2018 Correlation .587 .724 .722 .642 .619 1.000 .842 19_AdvTier Significance .000 .000 .000 .000 .000 . .000 _TotalRatio (2-tailed) df 532 532 532 532 532 0 532 ALT1_2018 Correlation .622 .759 .672 .621 .623 .842 1.000 19_Tier2_T Significance .000 .000 .000 .000 .000 .000 . otalRatio (2-tailed) df 532 532 532 532 532 532 0 218 SUB_2 01819_ ALT1_ SUB_2 SUB_2 SUB_2 ALT1_ ALT1_2 DistCap 201819 01819_ 01819_ 01819_ 201819 01819_ acity_T _Tier3_ SchPrio DataUs DistPrio _AdvTi Tier2_T otalRati TotalRa rity_Tot e_Total rity_Tot er_Total otalRati Control Variables o tio alRatio Ratio alRatio Ratio o SUB_201819 SUB_20181 Correlation 1.000 -.055 _SchPriority 9_DistCapa Significance . .207 _TotalRatio city_TotalR (2-tailed) & atio df 0 527 SUB_201819 ALT1_2018 Correlation -.055 1.000 _DataUse_T 19_Tier3_T Significance .207 . otalRatio & otalRatio (2-tailed) SUB_201819 _DistPriority df 527 0 _TotalRatio & ALT1_20181 9_AdvTier_T otalRatio & ALT1_20181 9_Tier2_Tota lRatio a. Cells contain zero-order (Pearson) correlations. 219 APPENDIX J TIPS TIERED DECISION GUIDELINES Measuring System Implementation & Student Outcomes Tiered Problem Solving & Decision-Making Guidelines for Social and Academic Behavior This document provides a general framework for team organization, problem solving, and decision-making for both academic and social behavior/performance across the three Tiers of Support. To enhance the effectiveness and efficiency of decision-making, teams need a tool for getting organized for team problem solving and decision making. Team organization follows a similar format for teams across the tiers. Team organization begins by defining a set of meeting foundations that provide clarity of purpose, predictability for team meetings, and the decisions that the team needs to make. Even though teams across the tiers of support have different purposes and goals, each team needs to be organized and needs to be clear regarding what decisions they are able to make, the decision-making cycle, the data source(s) used, and goal/benchmark levels. Every plan put in place, no matter the tier, needs to have two parts to the evaluation plan: 1) a plan and schedule for monitoring and reporting implementation fidelity at both systems and student intervention levels, and 2) a plan for monitoring student progress toward goals/benchmarks. Measuring student outcomes comes after reviewing implementation fidelity data for both systems features and intervention implementation (for specific problems). The systems fidelity measure referenced in this document is the Tiered Fidelity Inventory (TFI). To monitor implementation fidelity of specific solutions to specific problems, teams use a variety of strategies ranging from a show of hands, verbal check-ins, and short survey’s (public and/or private). Student performance is measured through a variety of tools that collect and summarize student academic and social performance. Within each tier, teams determine the status of student outcome data in relation to national data and/or desired targets. Teams use those data for further queries as they drill down to determine if there are potential problems/referrals. Tier I Systems of Support include: • 3-5 positively stated expectations/ academic curriculum with implementation plan for • defining & teaching expectations/lesson plans, • instructional schedule with explicit goals & strategies for meeting goals, • acknowledgement system for student success, • corrective processes and procedures for academic and social errors, and • a schedule for using data to monitor implementation fidelity and effects on student outcomes/ progress toward goals. The purpose of the Tier I team is to: • coordinate implementation of Tier I systems and supports, • monitor fidelity of implementation & overall status of progress towards goals/grade level benchmarks. • identify & develop data-based plans for new problems. • communicate with other school teams 220 Tier II Systems of Supports include: • interventions that provide supplemental support (additional instruction, additional structure & predictability, and increased opportunity for feedback) that is systematic, consistent, and always available for student access. • A link to Tier I supports • Student plans, data collection and data entry are consistent across students accessing Tier II support(s). The purpose of a Tier II team is to: • coordinate, identify & select students in need of Tier II supports, • monitor progress for students receiving Tier II supports and, • monitor fidelity of Implementation of Tier II supports. Tier III Systems of Supports include: • interventions that provide & monitor intensive individualized support, • a Systems Coordination Team to monitor systems implementation fidelity and student referrals for support, • Individual Student Support Teams The purpose of the Tier III Systems Coordination Team is to: • Coordinate implementation of Tier III systems and supports • Establish and facilitate individual support teams as needed • Monitor systems & interventions for fidelity of implementation • Monitor overall status of student progress towards goals. The purpose of an Individual Student Support Team is: • Recruit team members • Complete functional behavior assessment • Develop the competing behavior pathway and support plan strategies • Implement plan and measure interventions for fidelity of implementation • Monitor overall status of student progress towards goals. • Report student progress to Tier III Coordination Team The decision-making guidelines outlined, are organized with both academic and social performance in mind, as well as system implementation. At set of decision guidelines, decision- making cycle and goals/benchmarks are provided for each of the four teams described above. 221 Tier I Coordination and Problem-Solving Team Meeting Foundations Tier I Team Purpose Team Agreements 1. Develop and implement Tier I Respect systems & interventions for • Before meeting, complete tasks, inform facilitator of academic and social success absence/tardy, avoid side talk 2. Monitor fidelity of • During meeting, avoid side talk, stay focused implementation of Tier I systems • Start and end meeting on time & supports Relevance 3. Monitor academic and social • Question fidelity of implementation progress for all students • Make data-based decisions based on precision statements 4. Screen, select, & refer students (what, where, when, who, why & how often) in need of Tier II & III supports Reality • Consider feasibility, social acceptability, & contextual fit Team Members Facilitator Minute Taker Data Analyst Administrator Others Primary Back Up Team Meeting Schedule When Where Start/End Time Meeting Minute Location Data Collection Report & Data Entry Generation Schedule What, Who & What, Who & When Question When Fidelity of Are systems of support in place and being Implementation implemented as planned? Student How many months are problem levels at or Outcomes below the national median or expected for each grade? Is there a gradual increase or decrease in problem levels across a 4-month period of time? Are there peaks in problem levels or dips in academic data that are 15-20% higher/lower? Are Tier I interventions working for 80- 85% of students? What percentage of students are receiving Tier II and Tier III supports? Do any students need Tier II or Tier III supports? 222 Tier I / Primary Level of Support Social and Academic Performance & System Evaluation and Student Outcome Guidelines Used for monitoring system implementation & effectiveness of school wide academic and social performance Target (Goal)/Review Cycle Measure Questions to Answer Behavior Academic Quarterly/ per plan/goal Monthly Review Cycle Benchmark Review Implement- Are systems of support Aim for 70% implementation Aim for 80% ation in place and being fidelity (e.g., TFI-I review implementation fidelity on Fidelity implemented as quarterly, staff reporting 80% R-TFI/quarterly, and staff planned? implementation fidelity/ review reporting 80% monthly, students/families/ implementation community members’ input/ fidelity/review monthly review annually) Current How many months are Aim for 8 of 10 months to be at Aim for 8 of 10 months to Problem problem levels at or or below the national median be at or above the Levels below the national across a school year/review expected level for each median or expected for monthly grade level/review each grade? monthly Trends Is there a gradual Aim for consistent and/or Aim for consistent increase or decrease in decrease in problem levels increase in growth toward problem levels across a across time and grade levels/ benchmark/ review 4-month period of review monthly monthly time? Are there peaks in Aim for consistent and/or Aim for all grade levels problem levels or dips decrease in problem levels being within the in academic data that across time and grade benchmark range across are 15-20% levels/review monthly time/ review monthly higher/lower? Student Are Tier I interventions Aim for 85% of students having Aim for 80% emerging/on Proportions working for 80-85% of no more than one major ODR grade level, 15% students? What across time and grade strategic, and 5% percentage of students levels/review monthly intensive/ review monthly are receiving Tier II and Tier III supports? Do any students need Aim for no more than 15% Aim for no more than Groups and Tier II or Tier III students requiring Tier II 15% students requiring Individual supports? supports and no more than 5% Tier II supports and no Students of student requiring Tier III more than 5% of student supports/review monthly requiring Tier III support/review monthly Use information to create A Big Picture-Overall Status Statement (Primary Statement) regarding Behavior and Academic performance in relation to national data and to trigger further queries of data. 223 Tier I New Problem Tier I Progress Monitoring Guidelines • Check levels of implementation Fidelity of Implementation fidelity • TFI-Tier I to measure the systems procedures & processes • Look for increase/spike in • Fidelity checklist for participating staff errors/problem behaviors Student Outcomes • Review of skills & expectations • If less than 85% of students are succeeding review after extended absences implementation fidelity before adjusting the plan • Use previous year’s data trends • Make sure the problem is defined with precision and for prevention planning solutions with contextual fit • Consider Tier II or III supports for students with 2+ referrals Tier II Coordination and Problem-Solving Team Meeting Foundations Tier II Team Purpose Team Agreements • Identify & select students in need of Tier II supports • Inform facilitator of absence/tardy • Monitor progress for students receiving Tier II supports before meeting • Monitor fidelity of Implementation of Tier II supports • Avoid side talk • Stay focused & active • Start and end on time Team Members Facilitator Minute Data Administrator Others Taker Analyst Primary Back Up Team Meeting Schedule When Where Start/End Time Meeting Minute Location Data Collection & Report Generation Question Data Entry Schedule What, Who & When What, Who & When Fidelity of Implementation Student Outcomes 224 Tier II (Secondary Level): Social and Academic Performance & System Evaluation and Student Outcome Guidelines Used for monitoring system implementation and effectiveness of Tier II academic and social supports Questions to Answer per plan/goal Target (Goal)/ Monthly Review Cycle Implement- Are systems of support in place and Aim for 70% systems implementation fidelity ation being implemented as planned? on TFI-II, quarterly & staff reporting 80% Fidelity implementation fidelity, monthly Current How many students are receiving Tier Aim for no more than 15% of student Level II supports? population (at one time) requiring Tier II of student supports, monthly proportion Trends What proportion of our students is Aim for no more than 15% of student in student receiving Tier II supports? population (at one time) requiring Tier II proportion supports, monthly Trends What are the trends of overall progress At least 70% of students receiving Tier III in overall across students with Tier II supports? supports are starting or progressing, monthly student p rogress What proportion of students receiving Aim for 80% of students receiving Tier II Tier II support for 6 weeks are support for at least 6 week to be progressing, progressing and have met goals? monthly Trends Do any students need to be referred Aim for no more than 5% of student population in individual for Tier III supports? (at one time) requiring Tier III supports, student data monthly Use information to create an Overall Status Statement regarding fidelity of implementation and student progress toward goals & to trigger further queries of the data. Tier II New Problem/ New Referral Triggers Tier II Progress Monitoring Guidelines • Student was receiving Tier II support in Fidelity of Implementation Measures prior placement • Tiered Fidelity Inventory for Tier II (TFI-Tier II) • Students who enroll in school after the first to measure the systems procedures and processes 3 weeks of the school year participate in • Fidelity Checklist for participating staff CICO for the first 2-5 days of attendance Student Outcomes as an orientation to school expectations, • As defined by student support plan/IEP procedures and locations • After documenting fidelity of implementation • Student receives 2 or more office Retain intervention for at least 6 weeks of success discipline referrals or upward trend toward goal • Student has more than 5 absences in a 30- Modify intervention with more intensive supports day period if after two weeks of implementation, there is no • There is significant concern regarding improvement mental health issues, anti-social behavior, • add to basic CICO, or or serious concerns about family support • move to Tier III supports & create a • Student has a 504 plan student support team • Student, teacher and/or family request Fade supports to a self management system when • Student is not in crisis student has been successful 4 days a week (80% • Instructional staff are trained to implement of time) for at least 6 weeks Tier II interventions (fidelity of Graduate off Tier II intervention with self implementation management success for 4-6 weeks Tier III Coordination Team Meeting Foundations 225 Tier III Team Purpose Team Agreements • Coordinate implementation of Tier III systems • Inform facilitator of absence/tardy before and supports meeting • Establish and facilitate individual support • Avoid side talk teams as needed • Stay focused & active • Monitor systems & interventions for fidelity of • Start and end on time implementation • Monitor overall status of student progress towards goals. Team Members Facilitator Minute Taker Data Analyst Administrator Others Primary Back Up Team Meeting Schedule When Where Start/End Time Meeting Minute Location Data Collection & Report Generation Question Data Entry Schedule What, Who & When What, Who & When Fidelity of Implementation Student Outcomes 226 Tier III System Evaluation Guidelines Used for monitoring system implementation and effectiveness of individual student support plans Questions to Answer Target (Goal)/ Bi-Weekly Review Cycle Implementation What percentage of system features Aim for 70% on TFI-Tier III or equivalent Fidelity is in place? measure, quarterly & staff reporting 80% implementation fidelity, weekly Current Level What proportion of our students is Aim for no more than 5% of student of student receiving Tier III supports? population (at one time) requiring Tier III proportions supports, monthly Trends in What are the trends of overall At least 70% of students receiving Tier III student progress across students with Tier supports are starting or progressing, monthly proportions III supports? What percentage of students Aim for 80% of students receiving Tier III Trends in receiving Tier III support for 6 support for at least 6 week to be progressing, overall progress weeks are progressing and have met bi-weekly g oals? Use information to create A Big Picture- Overall Status Statement (Primary Statement) regarding Tier III systems & to trigger further queries of the data. Tier III New Problem/ New Referral Tier III Progress Monitoring Guidelines Triggers • Progress is below the expected rate Fidelity of Implementation after 2-6 weeks of receiving Tier II • Tiered Fidelity Inventory for Tier III (TFI-Tier III) supports to measure the systems procedures and processes • Student receives 6 office discipline • Fidelity Checklist for staff participating in Tier III referrals intervention implementation as defined by student • Student has more than 5 absences in a support plan 30-day period Student Outcomes • There is significant concern regarding • Use goals defined in student support plan mental health issues, anti-social • After documenting fidelity of implementation behavior, or serious concerns about Retain intervention for at least 6 weeks of success or family support upward trend toward goal • Student’s behavior poses a potential Modify intervention if after two weeks of risk to self or others. implementation, there is no improvement • Student has an IEP Fade supports to an individualized self management • Teacher and/or family request system or Tier II Check In Check Out system when student has been successful 4 days a week (80% of time) for at least 6 weeks 227 Tier III Individual Student Support Team Meeting Foundations Tier III Individual Student Support Team Team Agreements Purpose • Recruit team members Respect • Complete functional behavior assessment • Before meeting, complete tasks, inform • Develop the competing behavior pathway facilitator of absence/tardy, avoid side talk and support plan strategies • During meeting, avoid side talk, stay focused • Implement plan and measure • Start and end meeting on time interventions for fidelity of Relevance implementation • Question fidelity of implementation • Monitor overall status of student progress • Make data based decisions based on precision towards goals. statements (what, where, when, who, why & • Report student progress to Tier III how often) Coordination Team Reality • Think about feasibility, social acceptability, & contextual fit Student Information Name_____________________ Grade________ IEP? 504? Team Members Facilitator Minute Data Administrator Others Taker Analyst Primary Back Up Team Meeting Schedule When Where Start/End Time Meeting Minute Location Data Collection & Data Entry Schedule Report Generation Question What, Who & When What, Who & When Fidelity of Implementation Student Outcomes 228 Tier III Individual Student Support Evaluation Guidelines Used for monitoring individual student progress toward goals Questions to Answer per plan/goal Target (Goal)/ Weekly Review Cycle Implementation Was the plan implemented as Aim for 80% implementation fidelity, weekly Fidelity planned? Current Level What is the current status of problem Aim for making progress toward goal, of student in relation to previous review and weekly problem goal? Trends in What are the trends of overall Aim for increasing trend toward goal, frequency progress toward goal(s)? weekly Peaks in Are there peaks in problem behavior Aim for consistent and/or decrease in frequency across time problem level across time (or increase in skill level), weekly Team Name : __________________________ Date: ____________________ Team Purpose Team Agreements Team Members Facilitator Minute Data Administrator Others Taker Analyst Primary Back Up Team Meeting Schedule When Where Start/End Time Meeting Minute Location Data Collection & Report Generation Question Data Entry Schedule What, Who & When What, Who & When Fidelity of Implementation Student Outcomes Team Meeting Foundations & Decision Guidelines (2016). Horner, Flannery, Nese, Chaparro, Conley, Todd, University of Oregon. Adapted from TIPS, PBIS & MTSS Training Materials. tips2info@blogspot.com 229 REFERENCES CITED Adelman, H. S., & Taylor, L. (2007). Systemic change for school improvement. Journal of Educational and Psychological Consultation, 17, 55-77. Alberto, P. A., & Troutman, A. C. (2008). Applied behavior analysis for teachers (8th ed.). Prentice Hall. Algozzine, R. F., Barrett, S., Eber, L., George, H., Horner, R. H., Lewis, T. J., Putnam, R. F., Swain-Bradway, J., McIntosh, K., & Sugai, G. (2014). SWPBIS Tiered Fidelity Inventory [Unpublished measure]. OSEP Technical Assistance Center on Positive Behavioral Interventions and Supports. https://www.pbisapps.org/Applications/Pages/PBIS-Assessment.aspx Algozzine, R. F., Horner, R. H., Todd, A. W., Newton, J. S., Algozzine, K. M., & Cusumano, D. (2016). Measuring the process and outcomes of team problem solving. Journal of Psychoeducational Assessment, 34(3), 211-229. https://doi.org/10.1177/0734282915592535 Antoniou, P., & Kyriakides, L. (2013). A Dynamic Integrated Approach to teacher professional development: Impact and sustainability of the effects on improving teacher behaviour and student outcomes. Teaching and Teacher Education, 29, 1- 12. https://doi.org/http://dx.doi.org/10.1016/j.tate.2012.08.001 Armstrong, R. A. (2014). When to use the Bonferroni correction. Ophthalmic & Physiological Optics, 34, 502-508. https://doi.org/10.1111/opo.12131 Atkins, L., Francis, J., Islam, R., O’Connor, D., Patey, A., Ivers, N., Foy, R., Duncan, E. M., Colquhoun, H., Grimshaw, J. M., Lawton, R., & Michie, S. (2017). A guide to using the theoretical domains framework of behaviour change to investigate implementation problems. Implementation Science, 12, 1-18. https://doi.org/10.1186/s13012-017-0605-9 Atkins, M. S., Frazier, S. L., Leathers, S. J., Graczyk, P. A., Talbott, E., Jakobsons, L., Adil, J. A., Marinez-Lora, A., Demirtas, H., & Gibbons, R. B. (2008). Teacher key opinion leaders and mental health consultation in low-income urban schools. Journal of Consulting and Clinical Psychology, 76, 905-908. Auld, E., & Morris, P. (2016). PISA, policy and persuasion: translating complex conditions into education ‘best practice’. Comparative Education, 52, 202-229. https://doi.org/10.1080/03050068.2016.1143278 Bambara, L., Nonnemacher, S., & Kern, L. (2009). Sustaining school-based individualized positive behavior support: Perceived barriers and enablers. Journal of Positive Behavior Interventions, 11(3), 161-178. https://doi.org/10.1177/1098300708330878 230 Bambara, L. M., Goh, A. E., Kern, L., & Caskie, G. (2012). Perceived barriers and enablers to implementing individualized positive behavior interventions and supports in school settings. Journal of Positive Behavior Interventions, 14(4), 228-240. https://doi.org/10.1177/1098300712437219 Barnes, S. A., Iovannone, R., Blair, K.-S., Crosland, K., & George, H. P. (2020). Evaluating the prevent-teach-reinforce (PTR) model in general education settings. Preventing School Failure: Alternative Education for Children and Youth, 64, 128-141. https://doi.org/10.1080/1045988X.2019.1688228 Barrett, S., Perales, K., & Eber, L. (2016). Interconnected School Framework Action Planning Companion Guide to SWPBIS-Tiered Fidelity Inventory 2.0. http://www.midwestpbis.org/interconnected-systems-framework/tools Barrett, S. B., Bradshaw, C., & Lewis-Palmer, T. L. (2008). Maryland statewide positive behavior interventions and supports initiative: Systems, evaluation, and next steps. Journal of Positive Behavior Interventions, 10, 105-114. Bear, G. G., Gaskins, C., Blank, J., & Chen, F. F. (2011). Delaware School Climate Survey—Student: Its factor structure, concurrent validity, and reliability. Journal of School Psychology, 49, 157-174. https://doi.org/http://dx.doi.org/10.1016/j.jsp.2011.01.001 Bethune, K. S., & Wood, C. L. (2013). Effects of coaching on teachers’ use of function- based interventions for students with severe disabilities. Teacher Education and Special Education, 36, 97-114. https://doi.org/10.1177/0888406413478637 Blase, K. A., Kiser, L., & Van Dyke, M. (2013). The Hexagon Tool: Exploring context. National Implementation Research Network, FPG Child Development Institute. Blase, K. A., Van Dyke, M. K., & Fixsen, D. L. (2013). Implementation drivers: Assessing best practices. The University of North Carolina, FPG, SISEP. http://implementation.fpg.unc.edu/resources/implementation-drivers-assessing- best-practices Borgmeier, C., Croissant, E., Loman, S., Collier, T., & Strickland-Cohen, M. K. (2017). Basic fba to bip e-learning modules Kansas TASN. Retrieved 4/15/2019 from www.basicfba.com Boudett, K. P., City, E. A., & Murnane, R. J. (Eds.). (2013). Data wise: A step-by-step guide to using assessment results to improve teaching and learning (Revised and Expanded edition. ed.). Harvard Education Press. 231 Bradshaw, C. P., Mitchell, M. M., & Leaf, P. J. (2010). Examining the effects of schoolwide positive behavioral interventions and supports on student outcomes: Results from a randomized controlled effectiveness trial in elementary schools. Journal of Positive Behavior Interventions, 12, 16. https://doi.org/10.1177/1098300709334798 Bradshaw, C. P., & Pas, E. T. (2011). A statewide scale up of positive behavioral interventions and supports: A description of the development of systems of support and analysis of adoption and implementation [Report]. School Psychology Review, 40, 530-548. Bradshaw, C. P., Waasdorp, T. E., & Leaf, P. J. (2012). Effects of school-wide positive behavioral interventions and supports on child behavior problems and adjustment. Pediatrics, e1136-e1145. https://doi.org/10.1542/peds.2012-0243 Bragg, D., Carvalho, M., Charles, J., Hoang, A., Horner, R. H., Lachgar, H., & Sullivan, D. (2015). PBIS Assessment In Educational and Community Supports, University of Oregon. https://www.pbisapps.org/ Breiter, A., & Light, D. (2006). Data for school improvement: Factors for designing effective information systems to support decision-making in schools. Journal of Educational Technology & Society, 9, 206-217. Bruhn, A. L., Lane, K. L., & Hirsch, S. E. (2013). A review of tier 2 interventions conducted within multitiered models of behavioral prevention. Journal of Emotional and Behavioral Disorders, 22, 171-189. https://doi.org/10.1177/1063426613476092 Bruhn, A. L., Wehby, J. H., & Hasselbring, T. S. (2020). Data-based decision making for social behavior: Setting a research agenda. Journal of Positive Behavior Interventions, 22, 116-126. https://doi.org/10.1177/1098300719876098 Bryk, A. S. (Ed.). (2020). Improvement in action: Advancing quality in America's schools. Harvard Education Press. Castro-Olivo, S., Preciado, J., Le, L., Marciante, M., & Garcia, M. (2018). The effects of culturally adapted version of First Steps to Success for Latino English language learners: Preliminary pilot study. Psychology in the Schools, 55(1), 36-49. https://doi.org/10.1002/pits.22092 Center on Positive Behavioral Interventions and Supports. (2018). "Schools Using PBIS." Center on PBIS. Retrieved 5/2/2021 from www.pbis.org Center on Positive Behavioral Interventions and Supports. (2020). Positive behavioral interventions and supports (PBIS) evaluation blueprint [Blueprint]. 232 Childs, K. E., Kincaid, D., George, H. P., & Gage, N. A. (2016). The relationship between school-wide implementation of positive behavior intervention and supports and student discipline outcomes. Journal of Positive Behavior Interventions, 18(2), 89-99. https://doi.org/10.1177/1098300715590398 Cohen, R., Kincaid, D. K., & Childs, K. E. (2007). Measuring school-wide positive behavior support implementation: Development and validation of the Benchmarks of Quality. Journal of Positive Behavior Interventions, 9, 203-213. Conley, K. M., Kittelman, A., Massar, M., & McIntosh, K. (2018). What are patterns and predictors of CICO participation in U.S. Schools? [Evaluation Brief]. OSEP Technical Assistance Center on Positive Behavioral Interventions and Supports. www.pbis.org. Coyne, M. D., Oldham, A., Dougherty, S. M., Leonard, K., Koriakin, T., Gage, N. A., Burns, D., & Gillis, M. (2018, 2018/07/01). Evaluating the effects of supplemental reading intervention within an MTSS or RTI reading reform initiative using a regression discontinuity design. Exceptional Children, 84, 350- 367. https://doi.org/10.1177/0014402918772791 Curtin, F., & Schulz, P. (1998). Multiple correlations and Bonferroni’s correction. Biological Psychiatry, 44, 775-777. https://doi.org/https://doi.org/10.1016/S0006- 3223(98)00043-2 Daniels, B., Volpe, R. J., Briesch, A. M., & Gadow, K. D. (2017). Dependability and treatment sensitivity of multi-item direct behavior rating scales for interpersonal peer conflict. Assessment for Effective Intervention, 43, 48-59. https://doi.org/10.1177/1534508417698456 Datnow, A. (2005). The sustainability of comprehensive school reform models in changing district and state contexts. Educational Administration Quarterly, 41, 121-153. https://doi.org/10.1177/0013161X04269578 Debnam, K. J., Pas, E. T., & Bradshaw, C. P. (2012). Secondary and tertiary support systems in schools implementing school-wide positive behavioral interventions and supports: A preliminary descriptive analysis. Journal of Positive Behavior Interventions, 14, 142-152. Drummond, T. (1994). The student risk screening scale (SRSS). Josephine County Mental Health Program. Dunlap, G., Iovannone, R., English, C. L., Kincaid, D. K., Wilson, K., Christiansen, K., & Strain, P. S. (2009). Prevent-Teach-Reinforce: The school-based model of individualized positive behavior support. Brookes. 233 Dunlap, G., Sailor, W., Horner, R. H., & Sugai, G. (2009). Overview and history of positive behavior support. In Handbook of positive behavior support (pp. 3-16). Springer. Eber, L., Hyde, K., & Suter, J. C. (2011). Integrating wraparound into a schoolwide system of positive behavior supports. Journal of Child and Family Studies, 20, 782-790. Educational and Community Supports. (2021). PBISApps [Online App]. Ennis, R. P., Royer, D. J., Lane, K. L., & Dunlap, K. D. (2020). Behavior-specific praise in pre-K–12 settings: Mapping the 50-year knowledge base. Behavioral Disorders, 45, 131-147. https://doi.org/10.1177/0198742919843075 Evanovich, L. L., & Kern, L. (2018). Precorrection: Preventing Predictable Problem Behaviors in School Settings. Beyond Behavior, 1074295618769892. https://doi.org/10.1177/1074295618769892 Fabiano, G. A., Pyle, K., Kelty, M. B., & Parham, B. R. (2017). Progress monitoring using direct behavior rating single item scales in a multiple-baseline design study of the daily report card intervention. Assessment for Effective Intervention, 43, 21-33. https://doi.org/10.1177/1534508417703024 Fixsen, D. L., Blase, K. A., Naoom, S. F., & Wallace, F. (2009). Core implementation components. Research on Social Work Practice, 19(5), 531-540. https://doi.org/10.1177/1049731509335549 Fixsen, D. L., Blase, K. A., & Van Dyke, M. K. (2019). Implementation practice & science. Active Implementation Research Network. Fixsen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M., & Wallace, F. (2005). Implementation research: Synthesis of the literature. University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network (FMHI Publication #231). Flay, B. R., Biglan, A., Boruch, R. F., Castro, F. G., Gottfredson, D., Kellam, S., Moscicki, E. K., Schinke, S., Valentine, J. C., & Ji, P. (2005). Standards of evidence: Criteria for efficacy, effectiveness and dissemination. Prevention science, 6, 151-175. Forman, S. G., Shapiro, E. S., Codding, R. S., Gonzales, J. E., Reddy, L. A., Rosenfield, S. A., Sanetti, L. M. H., & Stoiber, K. C. (2013). Implementation science and school psychology. School Psychology Quarterly, 28, 77-100. Fullan, M. (2011). Choosing the wrong drivers for whole system reform [Seminar Series Paper No. 204]. Centre for Strategic Education. www.cse.edu.au 234 George, H. P., Cox, K. E., Minch, D., & Sandomierski, T. (2018). District practices associated with successful SWPBIS implementation. Behavioral Disorders, 43, 393-406. https://doi.org/10.1177/0198742917753612 Gilbert, T. F. (1978). Human competence: Engineering worthy performance. McGraw- Hill (Tribute Edition published by HRD Press and ISPI Publications, Washington, D.C., 1996). Hagger, M. S., & Luszczynska, A. (2014). Implementation intention and action planning interventions in health contexts: State of the research and proposals for the way forward. Applied Psychology: Health and Well‐Being, 6, 1-47. https://doi.org/10.1111/aphw.12017 Hall, L. J. (2015). Sustaining evidence-based practices by graduated special educators of students with asd: Creating a community of practice. Teacher Education and Special Education, 38, 28-43. https://doi.org/10.1177/0888406414558883 Halverson, R., Grigg, J., Prichett, R., & Thomas, C. (2015). The new instructional leadersion: Creating data-drive instructional systems in schools (2007). Journal of School Leadership, 25, 447-481. Harn, B. A., Parisi, D., & Stoolmiller, M. (2013). Balancing fidelity with flexibility and fit: What do we really know about fidelity of implementation in schools? Exceptional Children, 79, 181-193. Hawken, L. S., Adolphson, S. L., Macleod, K. S., & Schumann, J. (2009). Secondary-tier interventions and supports In W. Sailor, G. Dunlap, G. Sugai, & R. H. Horner (Eds.), Handbook of Positive Behavior Support (pp. 395-420). Springer. Hawken, L. S., Bundock, K., Kladis, K., O'Keeffe, B., & Barrett, C. A. (2014). Systematic review of the check-in, check-out intervention for students at risk for emotional and behavioral disorders. Education & Treatment of Children, 37, 635- 658. https://doi.org/10.1080/17489530802505412. Hawken, L. S., Crone, D. A., Bundock, K., & Horner, R. H. (2021). Responding to problem behavior in schools: the check-in, check-out intervention (Third ed.). New York : The Guilford Press. Hawken, L. S., O'Neill, R. E., & MacLeod, K. S. (2011). An investigation of the impact of function of problem behavior on effectiveness of the behavior education program (BEP). Education & Treatment of Children, 34, 551-574. http://search.ebscohost.com/login.aspx?direct=true&db=pbh&AN=65471282&sit e=ehost-live Hawkins, M., & James, C. (2017). Developing a perspective on schools as complex, evolving, loosely linking systems. Educational Management Administration & Leadership, 46, 729-748. https://doi.org/10.1177/1741143217711192 235 Hieneman, M., & Dunlap, G. (2000). Factors affecting the outcomes of community-based behavioral support: I. Identification and description of factor categories. Journal of Positive Behavior Interventions, 2, 161-169, 178. https://doi.org/10.1177/109830070000200304 Hieneman, M., & Dunlap, G. (2001). Factors affecting the outcomes of community-based behavioral support: II. Factor category importance. Journal of Positive Behavior Interventions, 3, 67-74. https://doi.org/10.1177/109830070100300202 Horner, R. H. (2009). Expanding the science, values and vision of positive behavior support. Horner, R. H., Flannery, K. B., Nese, R., Chaparro, E., Conley, K., & Todd, A. W. (2016). Measuring system implementation and student outcomes: Tiered problem solving and decision making guidelines for social and academic behavior. University of Oregon. Horner, R. H., Newton, J. S., Todd, A. W., Algozzine, R. F., Algozzine, K. M., Cusumano, D., & Preston, A. (2018). A randomized waitlist controlled analysis of team-initiated problem solving professional development and use. Behavioral Disorders, 43, 444-456. https://doi.org/10.1177/0198742917745638 Horner, R. H., Sugai, G., & Fixsen, D. L. (2017). Implementing effective educational practices at scales of social importance. Clinical Child and Family Psychology Review, 20, 25-35. https://doi.org/10.1007/s10567-017-0224-7 Horner, R. H., Sugai, G., Smolkowski, K., Eber, L., Nakasato, J., Todd, A. W., & Esparanza, J. (2009). A randomized, wait-list controlled effectiveness trial assessing school-wide positive behavior support in elementary schools. Journal of Positive Behavior Interventions, 11, 133-144. Horner, R. H., Sugai, G., & Todd, A. W. (2001). "Data" need not be a four-letter word: Using data to improve schoolwide discipline. Beyond Behavior, 11(1), 20-26. Horner, R. H., Todd, A. W., Lewis-Palmer, T. L., Irvin, L. K., Sugai, G., & Boland, J. B. (2004). The school-wide evaluation tool (SET): A research instrument for assessing school-wide positive behavior support. Journal of Positive Behavior Interventions, 6, 3-12. Hunter, K. K., Chenier, J. S., & Gresham, F. M. (2013). Evaluation of check in/check out for students with internalizing behavior problems. Journal of Emotional and Behavioral Disorders, 22, 135-148. https://doi.org/10.1177/1063426613476091 IBM. (2019). SPSS Statistics for Windows. In IBM Corp. 236 Irvin, L. K., Horner, R. H., Ingram, K., Todd, A. W., Sugai, G., Sampson, N. K., & Boland, J. B. (2006). Using office discipline referral data for decision making about student behavior in elementary and middle schools: An empirical evaluation of validity. Journal of Positive Behavior Interventions, 8, 10-23. https://doi.org/10.1177/10983007060080010301 Kaufman, J. S., Jaser, S. S., Vaughan, E. L., Reynolds, J. S., DiDonato, J., Bernard, S. N., & More, m. (2010). Patterns in office referral data by grade, race/ethnicity, and gender. Journal of Positive Behavior Interventions, 12, 44-54. Kincaid, D. K., Childs, K., Blase, K. A., & Wallace, F. (2007). Identifying barriers and facilitators in implementing schoolwide positive behavior support. Journal of Positive Behavior Interventions, 9, 174-184. Kincaid, D. K., Childs, K. E., & George, H. (2005). School-wide Benchmarks of Quality (BoQ) [Unpublished measure]. https://www.pbisapps.org/Applications/Pages/PBIS-Assessment.aspx Kittelman, A., Bromley, K. W., Mercer, S. H., & McIntosh, K. (2019). Validation of a measure of sustainability of school-wide behavior interventions and supports. Remedial and Special Education, 40, 67-73. https://doi.org/10.1177/0741932517753821 Kittelman, A., Eliason, B. M., Dickey, C. R., & McIntosh, K. (2018). How are schools using the SWPBIS Tiered Fidelity Inventory (TFI)? [Evaluation Brief]. OSEP Technical Assistance Center on Positive Behavioral Interventions and Supports www.pbis.org. Kittelman, A., Mercer, S. H., McIntosh, K., & Hoselton, R. (2021). Optimal timing for launching installation of tier 2 and 3 systems of school-wide positive behavioral interventions and supports. Journal of Positive Behavior Interventions. https://doi.org/10.1177/1098300721996084 Kittelman, A., Mercer, S. H., McIntosh, K., & Nese, R. (2021). Development and validation of a measure assessing sustainability of tier 2 and 3 behavior support systems. Journal of School Psychology, 85, 140-154. https://doi.org/https://doi.org/10.1016/j.jsp.2021.02.001 Kittelman, A., Strickland-Cohen, M. K., Pinkelman, S. E., & McIntosh, K. (2020). Variables contributing to abandonment and readoption of SWPBIS. Journal of Positive Behavior Interventions, 22, 67-77. https://doi.org/10.1177/1098300719888748 237 Koczwara, B., Stover, A. M., Davies, L., Davis, M. M., Fleisher, L., Ramanadhan, S., Schroeck, F. R., Zullig, L. L., Chambers, D. A., & Proctor, E. (2018). Harnessing the synergy between improvement science and implementation science in cancer: A call to action. Journal of Oncology Practice, 14, 335-340. https://doi.org/10.1200/jop.17.00083 La Salle, T. P., McIntosh, K., & Eliason, B. M. (2018). School climate survey suite administration manual. University of Oregon. Lane, K. L., Menzies, H. M., Oakes, W. P., & Kalberg, J. R. (2012). Systematic screenings of behavior to support instruction: From preschool to high school. Guilford. Langley, G. J., Moen, R. D., Nolan, K. M., Norman, C. L., & Provost, L. P. (2009). The improvement guide: a practical approach to enhancing organizational performance (Second ed.). Jossey-Bass. Ledford, J. R., & Gast, D. L. (2014). Measuring procedural fidelity in behavioural research. Neuropsychological Rehabilitation, 24, 332-348. https://doi.org/10.1080/09602011.2013.861352 Leverson, M., Smith, K., McIntosh, K., Rose, J., & Pinkelman, S. E. (2021). PBIS cultural responsiveness field guide: Resources for trainers and coaches Center on PBIS. Lewis, C. (2015). What is improvement science? Do we need it in education? Educational Researcher, 44, 54-61. https://doi.org/10.3102/0013189x15570388 Lewis, T. J., Sugai, G., & Colvin, G. (1998). Reducing problem behavior through a school-wide system of effective behavioral support: Investigation of a school- wide social skills training program and contextual interventions. School Psychology Review, 27, 446-459. Lindland, E., Fond, M., Haydon, A., Volmert, A., & Kendall-Taylor, N. (2015). "Just do it" communicating implementation science and practice: A frameworks strategic report. FrameWorks Institute. Loman, S. L., Rodriguez, B. J., & Horner, R. H. (2010). Sustainability of a targeted intervention package: First step to success in oregon. Journal of Emotional and Behavioral Disorders, 18, 178-191. https://doi.org/10.1177/1063426610362899 Maggin, D. M., Zurheide, J., Pickett, K. C., & Baillie, S. J. (2015). A systematic evidence review of the check-in/check-out program for reducing student challenging behaviors. Journal of Positive Behavior Interventions, 17, 197-208. https://doi.org/10.1177/1098300715573630 238 Markelz, A. M., Taylor, J. C., Kitchen, T., Riccomini, P. J., Scheeler, M. C., & McNaughton, D. B. (2019). Effects of tactile prompting and self-monitoring on teachers’ use of behavior-specific praise. Exceptional Children, 85, 471-489. https://doi.org/10.1177/0014402919846500 Marshall, M., Pronovost, P., & Dixon-Woods, M. (2013, 2013/02/02/). Promotion of improvement as a science. The Lancet, 381, 419-421. https://doi.org/https://doi.org/10.1016/S0140-6736(12)61850-9 Mason, M. (2008). What is complexity theory and what are its implications for educational change? Educational Philosophy and Theory, 40, 35-49. https://doi.org/10.1111/j.1469-5812.2007.00413.x May, S., Ard, W. I., Todd, A., Horner, R. H., Glasgow, A., Sugai, G., & Sprague, J. (1998). School-wide information system suite (SWIS). In University of Oregon. www.pbisapps.org May, S., Ard, W. I., Todd, A. W., Horner, R. H., Glasgow, A., Sugai, G., & Sprague, J. R. (2013). School-wide information system (SWIS). In http://www.pbisapps.org May, S., Talmadge, N., Todd, A. W., Horner, R. H., & Rossetto Dickey, C. (2008). Check-in check-out school-wide information system (CICO-SWIS). In University of Oregon. McIntosh, K., Doolittle, J. D., Vincent, C. G., Horner, R. H., & Ervin, R. A. (2009). School-wide universal behavior sustainability index: School teams (SUBSIST) [Unpublished research measure]. University of British Columbia. McIntosh, K., Frank, J. L., & Spaulding, S. A. (2010). Establishing research-based trajectories of office discipline referrals for individual students. School Psychology Review, 39, 380-394. https://doi.org/10.1080/02796015.2010.12087759 McIntosh, K., Horner, R. H., & Sugai, G. (2009). Sustainability of systems-level evidence-based practices in schools: Current knowledge and future directions. In W. Sailor, G. Dunlap, G. Sugai, & R. H. Horner (Eds.), Handbook of positive behavior support (pp. 327-352). Springer. McIntosh, K., Kim, J., Mercer, S. H., Strickland-Cohen, M. K., & Horner, R. H. (2015). Variables associated with enhanced sustainability of school-wide positive behavioral interventions and supports. Assessment for Effective Intervention, 40, 8. https://doi.org/1534508414556503 McIntosh, K., Kittelman, A., Mercer, S. H., & Nese, R. N. T. (2018). Advanced level tier interventions treatment utilization and durability evaluation (ALTITUDE) [Unpublished research measure]. University of Oregon. 239 McIntosh, K., MacKay, L. D., Hume, A. E., Doolittle, J. D., Vincent, C. G., Horner, R. H., & Ervin, R. A. (2011). Development and initial validation of a measure to assess factors related to sustainability of school-wide positive behavior support. Journal of Positive Behavior Interventions, 13, 208-218. https://doi.org/10.1177/1098300710385348 McIntosh, K., Mercer, S. H., Horner, R. H., & Nese, R. (2018). Identifying factors predicting implementation and sustainability of tier 2 and 3 behavior support systems [Grant]. University of Oregon. McIntosh, K., Mercer, S. H., Hume, A. E., Frank, J. L., Turri, M. G., & Mathews, S. (2013). Factors related to sustained implementation of schoolwide positive behavior support. Exceptional Children, 79, 293-311. McIntosh, K., Mercer, S. H., Nese, R., Strickland-Cohen, M. K., Kittelman, A., Hoselton, R., & Horner, R. H. (2018). Factors predicting sustained implementation of a universal behavior support framework. Educational Researcher, 47, 307-316. https://doi.org/10.3102/0013189X18776975 McIntosh, K., Mercer, S. H., Nese, R. N. T., & Ghemraoui, A. (2016). Identifying and predicting distinct patterns of implementation in a school-wide behavior support framework. Prevention science, 17, 992-1001. https://doi.org/10.1007/s11121- 016-0700-1 Mercer, S. H., McIntosh, K., Strickland-Cohen, M. K., & Horner, R. H. (2014). Measurement invariance of an instrument assessing sustainability of school-based universal behavior practices. School Psychology Quarterly, 29, 125-137. https://doi.org/10.1037/spq0000054 Merrell, K. W., & Buchanan, R. S. (2006). Intervention selection in school-based practice: Using public health models to enhance systems capacity of schools. School Psychology Review, 35, 167-180. Mukaka, M. M. (2012). Statistics corner: A guide to appropriate use of correlation coefficient in medical research. Malawi medical journal, 24, 69-71. Mundfrom, D. J., Perrett, J. J., Schaffer, J., Piccone, A., & Roozeboom, M. (2006). Bonferroni adjustments in tests for regression coefficients. Multiple Linear Regression Viewpoints, 32, 1-6. Neely, A. (1999). The performance measurement revolution: why now and what next? International journal of operations & production management, 19, 205-228. https://doi.org/10.1108/01443579910247437 240 Nese, R., McIntosh, K., Nese, J. F. T., Hoselton, R., Bloom, J., Johnson, N., Richter, M., Phillips, D., & Ghemraoui, A. (2016). Predicting abandonment of school-wide positive behavioral interventions and supports. Behavioral Disorders, 42, 261- 270. https://doi.org/10.17988/BD-15-95.1 Newton, J. S., Horner, R. H., Algozzine, B., Todd, A. W., & Algozzine, K. (2012). A randomized wait-list controlled analysis of the implementation integrity of team- initiated problem solving processes. Journal of School Psychology, 50(4), 421- 441. https://doi.org/10.1016/j.jsp.2012.04.002 Newton, J. S., Horner, R. H., Algozzine, R. F., Todd, A. W., & Algozzine, K. M. (2009). Using a problem-solving model to enhance data-based decision making in schools. In W. Sailor, G. Dunlap, G. Sugai, & R. H. Horner (Eds.), Handbook of positive behavior support (pp. 551-580). Springer. O'Gorman, T. W., & Woolson, R. F. (1995). Using Kendall's τ-b correlations to improve variable selection methods in case-control studies. Biometrics, 51, 1451-1460. https://doi.org/10.2307/2533275 Onwuegbuzie, A. J., & Daniel, L. G. (1999, November 1999). Uses and misuses of the correlation coefficient. Mid-South Educational Research Association (MSERA), Point Clear, AL. Park, V., Elise St, J., Datnow, A., & Bailey, C. (2017). The balancing act: Student classroom placement routines and the uses of data in elementary schools. Journal of Educational Administration, 55, 390-406. https://doi.org/http://dx.doi.org/10.1108/JEA-09-2016-0098 Peterson, L., Homer, A., & Wonderlich, S. (1982). The integrity of independent variables in behavior analysis. Journal of Applied Behavior Analysis, 15(4), 477-492. https://doi.org/10.1901/jaba.1982.15-477 Predy, L. K., McIntosh, K., Frank, J. L., & Flitchcock, J. (2014). Utility of number and type of office discipline referrals in predicting chronic problem behavior in middle schools. School Psychology Review, 43, 18. https://doi.org/10.17105/SPR-13-0043.1 Robertson, R. E., Kokina, A. A., & Moore, D. W. (2020). Barriers to implementing behavior intervention plans: Results of a statewide survey. Journal of Positive Behavior Interventions. https://doi.org/10.1177/1098300720908013 Scott, T. M., Alter, P. J., Rosenberg, M., & Borgmeier, C. (2010). Decision-making in secondary and tertiary interventions of school-wide systems of positive behavior support. Education & Treatment of Children, 33, 513-535. 241 Scott, T. M., & Barrett, S. B. (2004). Using staff and student time engaged in disciplinary procedures to evaluate the impact of school-wide PBS. Journal of Positive Behavior Interventions, 6, 21-27. Simon, G. (1977). A nonparametric test of total independence based on Kendall's tau. Biometrika, 64, 277-282. https://doi.org/10.2307/2335694 Simonsen, B., Fairbanks, S., Briesch, A. M., & Sugai, G. (2006). Positive behaviour support classroom management self-assessment. Center on Positive Behavioral Interventions and Supports. Skiba, R. J., Peterson, R. L., & Williams, T. (1997). Office referrals and suspensions: Disciplinary intervention in middle schools. Education and Treatment of Children, 20, 295-315. Slocum, T. A., Detrich, R., Wilczynski, S. M., Spencer, T. D., Lewis, T., & Wolfe, K. (2014). The evidence-based practice of applied behavior analysis. The Behavior Analyst, 37, 41-56. https://doi.org/10.1007/s40614-014-0005-2 Stevenson, N. A., & Mussalow, P. R. (2019). The effects of planning, goal setting, and performance feedback on avoidance behaviors for struggling readers. Journal of Positive Behavior Interventions, 21, 171-180. https://doi.org/10.1177/1098300718804566 Strickland-Cohen, M. K., & Horner, R. H. (2015). Typical school personnel developing and implementing basic behavior support plans. Journal of Positive Behavior Interventions, 17, 83-94. https://doi.org/10.1177/1098300714554714 Strickland-Cohen, M. K., Pinkelman, S. E., Jimerson, J. B., Berg, T. A., Pinkney, C. J., & McIntosh, K. (2018). Sustaining effective individualized behavior support: Barriers and enablers. Preventing School Failure: Alternative Education for Children and Youth, 1-11. https://doi.org/10.1080/1045988X.2018.1456399 Sugai, G. (2008). School-wide expectations – Teaching matrix. Center on Positive Behavioral Interventions and Supports Sugai, G., & Horner, R. H. (2006). A promising approach for expanding and sustaining the implementation of school-wide positive behavior support. School Psychology Review, 35, 245-259. Sugai, G., & Horner, R. H. (2020). Sustaining and scaling positive behavioral interventions and supports: Implementation drivers, outcomes, and considerations. Exceptional Children, 86, 120-136. https://doi.org/10.1177/0014402919855331 242 Sugai, G., Lewis-Palmer, T. L., Todd, A. W., & Horner, R. H. (2005). School-wide evaluation tool [Unpublished research measure]. Educational and Community Supports. Available at http://www.pbis.org. https://www.pbisapps.org/Applications/Pages/PBIS-Assessment.aspx Sugai, G., Sprague, J. R., Horner, R. H., & Walker, H. M. (2000). Preventing school violence: The use of office discipline referrals to assess and monitor school-wide discipline interventions. Journal of Emotional and Behavioral Disorders, 8, 94- 101. https://doi.org/10.1177/106342660000800205 Sullivan, E. (2015). Psychometric investigation of the check-in, check-out fidelity of implementation measure [Dissertation, ProQuest Dissertations Publishing]. Taylor-Greene, S., Brown, D., Nelson, L., Longton, J., Gassman, T., Cohen, J., Swartz, J., Horner, R. H., Sugai, G., & Hall, S. (1997). School-wide behavioral support: Starting the year off right. Journal of Behavioral Education, 7, 99-112. Todd, A. W., Campbell, A. L., Meyer, G. G., & Horner, R. H. (2008). The effects of a targeted intervention to reduce problem behaviors: Elementary school implementation of check-in check-out. Journal of Positive Behavior Interventions, 10, 46-55. Todd, A. W., Horner, R. H., Berry, D., Sanders, C., Bugni, M., Currier, A., Potts, N., Newton, J. S., Algozzine, B., & Algozzine, K. (2012). A case study of team- initiated problem solving addressing student behavior in one elementary school. Journal of Special Education Leadership, 25, 81-89. Turri, M. G., Mercer, S. H., McIntosh, K., Nese, R. N. T., Strickland-Cohen, M. K., & Hoselton, R. (2016). Examining barriers to sustained implementation of school- wide prevention practices. Assessment for Effective Interventions, 42(1), 6-17. https://doi.org/10.1177/1534508416634624 Umbreit, J., & Ferro, J. B. (2015). Function-based intervention: Accomplishments and future directions. Remedial and Special Education, 36, 89-93. https://doi.org/10.1177/0741932514555024 Uyanık, G. K., & Güler, N. (2013). A study on multiple linear regression analysis. Procedia, social and behavioral sciences, 106, 234-240. https://doi.org/10.1016/j.sbspro.2013.12.027 Vanlommel, K., & Schildkamp, K. (2019). How do teachers make sense of data in the context of high-stakes decision making? American educational research journal, 56, 792-821. https://doi.org/10.3102/0002831218803891 Vesey, W. B., Vesey, J. T., Stroter, A. D., & Middleton, K. V. (2011). Multiple linear regression: A return to basics in educational research. Multiple Linear Regression Viewpoints, 37, 14-22. 243 Xu, W., Hou, Y., Hung, Y. S., & Zou, Y. (2013). A comparative analysis of Spearman's rho and Kendall's tau in normal and contaminated normal models. Signal Processing, 93, 261-276. https://doi.org/10.1016/j.sigpro.2012.08.005 Yeung, A. S., Craven, R., Chen, Z., Mooney, M., Tracey, D., Barker, K., Power, A., Dobia, B., Schofield, J., Whitefield, P., & Lewis, T. (2016). Positive behavior interventions: The issue of sustainability of positive effects. Educational Psychology Review, 28, 145-170. https://doi.org/10.1007/s10648-015-9305-7 Zettel, J. J., & Ballard, J. (1979). The education for all handicapped children act of 1975 PL 94-142: Its history, origins, and concepts. Journal of Education, 161, 5-22. https://doi.org/10.1177/002205747916100303 244