Finding Faults: Comparing Resilience Scorecards Researched by: Alexandra Corvello, MCRP Graduate Student at University of Oregon Source: UN Disaster Resilience Scorecard for Cities, 2017 ABSTRACT As the risks facing communities rapidly change, the need for assessing community-level resilience increases. Resilience is the ability to mitigate, prepare for, adapt to, recover, or avoid disruptive events (Cutter, 2016). To help communities accomplish the complicated and challenging task of anticipating future disruptions, communities have created resilience assessments to measure current resilience to future threats. There are three main types of resilience assessments: indices, toolkits, and scorecards. This report focuses on resilience scorecards because the distinction allows for better comparison among a single assessment type that does not have much research data. Overall validating the scorecards was difficult due to numerous differences in the scorecards, which were born from the unique processes the communities follow to create and use the scorecards. I examined five different scorecards: the UN Disaster Resilience Scorecard for Cities, Plan Integration for Resilience Scorecard, GEM Resilience Performance Scorecard, Resilient Communities Scorecard, and the Torrens Community Resilience Scorecard. The report compares the different scorecards against each other, and other resilience plans currently used to measure vulnerability within Oregon, which include Natural Hazard Mitigation Plans and the Oregon Resilience Plan. This content analysis was complemented by expert interviews and a focus group that addressed the feasibility and barriers to implementing a scorecard. The work helped advise a specific local non-profit, Cascadia Prepared, on their scorecard process to measure the state of Oregon’s vulnerability to the Cascadia Subduction Zone Earthquake. The report also is an informational tool for other communities to find easily accessible information on various resilience scorecards and information on the feasibility and issues surrounding the implementation process. Corvello, Alexandra 1 Table of Contents Abstract ......................................................................................................................................................................... 1 Executive Summary ....................................................................................................................................................... 3 Overview of Findings ................................................................................................................................................ 5 Implementing and Completing a Scorecard ......................................................................................................... 6 Considerations for Cascadia Prepared ................................................................................................................. 7 Introduction ................................................................................................................................................................... 9 Research Questions of Project ................................................................................................................................ 10 Literature Review ........................................................................................................................................................ 10 Measuring Resilience Scorecard Effectiveness ....................................................................................................... 11 Issues with Validity ................................................................................................................................................. 11 Research Gaps ......................................................................................................................................................... 12 Background Information .............................................................................................................................................. 13 UN Disaster Resilience Scorecard for Cities ............................................................................................................ 13 Plan Integration for Resilience Scorecard ............................................................................................................... 13 Resilience Performance Scorecard ......................................................................................................................... 13 Resilient Communities Scorecard ........................................................................................................................... 14 Torrens Community Resilience Toolkit and Scorecard ........................................................................................... 14 Other Resilience Plans ............................................................................................................................................ 14 Natural Hazard Mitigation Plan (NHMP) ............................................................................................................ 15 Oregon Resilience Plan....................................................................................................................................... 15 Methods and Data ....................................................................................................................................................... 16 Interviews ............................................................................................................................................................... 17 Focus Group ............................................................................................................................................................ 17 Limitations .............................................................................................................................................................. 18 Findings ........................................................................................................................................................................ 18 Resilience Scorecard Processes .............................................................................................................................. 18 Resilience Scorecard Content ................................................................................................................................ 19 UN Disaster Resilience Scorecard ...................................................................................................................... 19 Torrens Community Resilience Scorecard ......................................................................................................... 20 Plan Integration Scorecard ................................................................................................................................. 21 Important Categories to address in Resilience Scorecards ..................................................................................... 22 Implementing and Completing a Scorecard............................................................................................................ 24 Barriers ............................................................................................................................................................... 24 Feasibility ........................................................................................................................................................... 27 Considerations for Cascadia Prepared .................................................................................................................... 28 Option 1: Not Produce a Resilience Scorecard .................................................................................................. 28 Finding Faults: Comparing Resilience Scorecards June 2019 Page | 2 Option 2: Produce a Resilience Scorecard ......................................................................................................... 29 Option 3: Address Communities’ Social Capacity .............................................................................................. 30 Conclusion ................................................................................................................................................................... 31 Resources: ................................................................................................................................................................... 32 Appendix A - Interview Guide ...................................................................................................................................... 34  Do you know what a resilience scorecard is? ............................................................................................ 34 Appendix B - Focus Group Agenda .............................................................................................................................. 35 Appendix C - Resilience Scorecard Categories ............................................................................................................. 36 Acknowledgements: I would like to thank the University of Oregon and Cascadia Prepared for this interesting project topic. Thank you to all of the expert participants, I really appreciate all of your time and insights. Thank you to my 2019 MCRP cohort, without whom this whole experience would not have been nearly as fun. Thank you to Rich Margerum and Josh Bruce for all of the project advice, draft readings, and support. Thank you to my CPW team for helping me gain all of these amazing skills to complete this project. Finding Faults: Comparing Resilience Scorecards June 2019 Page | 3 EXECUTIVE SUMMARY My research project examined five resilience scorecards: the UN Disaster Resilience Scorecard for Cities, Plan Integration for Resilience Scorecard, GEM Resilience Performance Scorecard, Resilient Communities Scorecard, and the Torrens Community Resilience Scorecard. The report compared the different scorecards against each other, and other resilience plans currently used to measure vulnerability within Oregon, which include Natural Hazard Mitigation Plans and the Oregon Resilience Plan. This content analysis was complemented by expert interviews and a focus group that assessed the feasibility and barriers to implementing a scorecard. There are some limitations to the collected data due to researcher inexperience and time. This work helped inform a specific local non-profit, Cascadia Prepared, on their scorecard process to measure the state of Oregon's vulnerability to the Cascadia Subduction Zone Earthquake. The report will help other communities to find easily accessible information on various resilience scorecards and information on the feasibility and issues surrounding the implementation process. The scorecards fell into two types of formats: - prescriptive with well defined sections and subsections that required certain data to obtain the scores - content analysis format that required expert knowledge to implement The first was used by the majority of scorecards with the exception of the Plan Integration Scorecard, which used the content analysis approach (National Resource Institute, 2017). The scorecards all involved the community in the process with most of them having the community take charge of the process. An exception was the Resilience Performance Scorecard, which had the Global Model Earthquake organization become a consultant that led the communities through the process and generated scores with proprietary software (GEM, 2017). The rankings were typically numerical with the higher number indicating more resilience, with the outlier being the Plan Integration Scorecard that conducted more of an ordinal content analysis on the community's current plans and policies (National Resource Institute, 2017). Finding Faults: Comparing Resilience Scorecards June 2019 Page | 4 Overview of Findings The resilience scorecards I examined measured the following categories: Table 1: Resilience Scorecard Comparison, From Comparing Resilience Scorecards Report, 2019 Torrens Community Resilient GEM Resilience UN City Resilience Resilience Communities Plan Integration for Performance Categories that are measured Scorecard Scorecard Scorecard Resilience Scorecard Scorecard Risk/Vulnerability X X X Financial Capacity X X X X Urban Development X X X X Natural Environment X X Institutional Capacity for X X X X Resilience X X X X Social Capacity for Resilience Infrastructure Resilience X X X X X X X Protection of Cultural Heritage Disaster Response X X X X X X X Disaster Recovery Plans Definitions: Risk/Vulnerability social and physical vulnerability Protection of Cultural historic buildings; cultural vulnerability Financial Capacity incentives; penalties Heritage Urban Development land use; building; zoning Infrastructure water; energy; transportation; communication; conservation; use of environment to Natural Environment Resilience healthcare; education; food systems; etc. enhance resilience Institutional Capacity for emergency plans; emergency supplies; support of public education; awareness Disaster Response Resilience first responders private sector; social networks; interaction Disaster Recovery continuity; pre-event recovery plans; capital Social Capacity for Resilience between formal and informal institutions Plans improvement plans Corvello, Alexandra 5 These categories address the various topics needed to measure a community’s resilience to hazards, with the most critical categories for post-disaster preparedness determined by the content analysis, interviews, and focus group being: risk and vulnerability; financial capacity; social capacity for resilience; infrastructure resilience; disaster response; and disaster recovery plans. Some topics that were also mentioned aligned better with pre-disaster planning, which are urban development and the natural environment. Implementing and Completing a Scorecard The focus groups and interviews addressed the main barriers to implementing a resilience scorecard, which were: resources, finances, participation, and time. 1) Resources is an all-encompassing term; for this project, the resources will address the methodology and data needed to complete a scorecard. - terminology: requires consistent and understandable terminology to be effectively used by multiple communities and get consistent results - gaps in data sources: need to be identified because it can cause variation in the measuring capacity among topics 2) Finances were the next significant barrier, primarily due to the broad geographic scope of the proposed project. - necessary for everything within the process: compensating for staff time to analyzing the data, public outreach of the information gathered, and for incentives to participate (this requires a lot of money for the scope of the statewide project) - limited public funds and the challenge of people perceiving resilience as something that will be needed in the future causes more barriers to getting the required funds 3) Participation Coordination is necessary because the whole scorecard process depends on multiple stakeholders from the community, state and federal government, as well as experts such as economists. - primary issue is trust, which takes time and commitment to build - have to answer the questions of “Why should a community participate?” and “What does a community gain from participating?” 4) Time is affected by any issues that happen within the barriers above. - main problem is project’s timeframe will be heavily impacted by how quickly participants engage in process, which could lead to the project going over time or budget These barriers affect the feasibility for completing a scorecard, though the reputation and limitations of scorecards also help. Throughout the research, resilience scorecards had a reputation of being a point in time snapshot of all categories necessary to accurately assess a community's vulnerability, but results in no long term improvement. Though resilience scorecards measure topics significant to determining resilience, they are all subjective in how and what data sources and measurements they use to establish a community's score. This subjectivity can create a false sense of security among government officials and community members because it can deem a community resilient to a hazard based on faulty information or Corvello, Alexandra 6 misguided preconceived ideas around how a hazard might manifest (In Person Interview 5.8.19). Being seen as a point in time snapshot also hinders the support for follow up and implementation of action items. Thus there needs to be a lead organization or team that oversees the initial process, help communities execute their resilience action items, and monitor the progress as a whole. Considerations for Cascadia Prepared Cascadia Prepared has many different avenues available to support the communities of Oregon in its mission to improving resilience against the Cascadia Earthquake. Here are three options. 1) Not Produce a Resilience Scorecard There are already adequate resilience assessments for the state of Oregon around the Cascadia Earthquake that have monitoring and updates built into their systems – the Oregon Resilience Plan and Natural Hazard Mitigation Plan. Cascadia Prepared could put their resources towards implementing some of the action items identified within these plans and work at a more individual or community level of preparedness. Most Natural Hazard Mitigation Plans within Oregon have action items around public education and awareness, or around identifying and upkeep on designated shelter locations and buildings, which is something Cascadia Prepared could help communities implement. 2) Produce a Resilience Scorecard based on the Oregon Resilience Plan This scorecard would address the recommendation offered by the Oregon Resilience Plan. The Oregon Resilience Plan acknowledged that regional resilience information is not as applicable for communities to implement due to the extreme variations among Oregon’s regional and county areas, therefore it recommended for the use of a similar framework to the Oregon Resilience Plan that produced an in-depth analysis on the community level (OSSPAC, 2013). This would address the physical and organizational capacities of resilience. If Cascadia Prepared goes in this direction, they will have to contend with some unique challenges. These challenges include: - having a tougher barrier to overcome due to them being an outside organization - the limitation of focusing on a specific hazard, earthquakes - getting the funding necessary to provide valid financial incentives - gaining communities’ trust as partners - not being perceived as an outside “watch dog” that will publicize a community’s scores and potentially negatively affect them 3) Address Communities’ Social Capacity This option is an exciting opportunity for Cascadia Prepared because social capacity is not typically addressed very well in resilience assessments (Cutter 2016). Therefore, Cascadia Prepared could use a scorecard mentioned above or outside organizational resilience diagnostic tools (e.g., New Zealand Resilient Organisations) to close the data gap on this crucial category. The organization could focus on the individual's Finding Faults: Comparing Resilience Scorecards June 2019 Page | 7 preparedness and scale up to the neighborhood and other social network levels, which would allow for the process to grow with the organization and give immediate impact to a community’s resilience (In Person Interview 4.30.19). Social capacity within the resilience field is a tough topic to measure due to the subjectivity and barriers around collecting this highly variable and informal information (Cutter 2016). Cascadia Prepared could be part of the process that would work on community action items around individual preparedness and education, while also creating individual diagnostic toolkits to measure and improve a community’s resilience. This could be accomplished through pre and post surveys, implementing community toolkits, and creating databases that could bridge the gap between the general information typically stated in resilience plans and the actual individual data for each community (Phone Interview 5.8.19). This "next step" for resilience could be significantly helped by outside organizations like Cascadia Prepared. The data could be used to inform neighborhood or community population resilience, especially around traditionally vulnerable communities which require more time and resources to measure accurately. As one might image, scorecards that are addressing the physical and organizational side of resilience as well, typically, cannot put the necessary resources towards social capacity because their scope has to address two other major components of resilience. Cascadia Prepared is in a prime position to implement this social capacity diagnostic and improvement plan due to their ability to specialize and their position as a non-profit organization that doesn’t have to contend with government limitations. Finding Faults: Comparing Resilience Scorecards June 2019 Page | 8 INTRODUCTION The ever-increasing intensity of natural hazards makes resilience a crucial topic for communities to discuss and improve upon. The term resilience broadly means the ability to prepare for, absorb, recover from, adapt to, or avoid disruptive events (Cutter, 2016). Resilience addresses the interplay between a cluster of systems that are in a continual feedback loop (Winderl, 2014), with the most prominent networked resources including economic development, social capital, information and communication, and community competence (Sherrieb, 2010). These systems are examined through the concepts of stress, adaptation, wellness, and resources dynamics (Norris, 2008). A commonly used method of analysis is the resilience assessment. Communities use different types of resilience assessment models to gauge vulnerability to disruptive events, with the majority of assessment models falling into three types: indices, scorecards, and toolkits. These different assessments have similarities and differences that are subtle but distinguishable. For example, indices mainly rely on quantitative data for generating an aggregate index value using weighted sums, while toolkits have a broader scope that can incorporate indices and scorecards as well as suggest recommendations for improving resilience (Cutter, 2016). Scorecards, on the other hand, are used to obtain values for performance against resilience categories through the use of dichotomous or multiple-choice questions, calculated statistical values, or judgements and/or perceptions that are then scored with a numerical ranking (Rowcliffe, 2000). All resilience assessments use a systems thinking approach to measure a community’s vulnerabilities by examining the physical, organizational, and social capacities individually, while also emphasizing the cross-scalar interconnections between the three systems (Vaughan, 2018; Arbon, 2012). These assessments try to uncover and understand the complex topic of resilience. Resilience is multi-faceted because when a shock hits a system, the effects do not happen in isolation. Instead, it affects all the systems working within a community, and those effects are compounded because the systems are all dependent on each other (Vaughan, 2018). An example of a shock or disruptive event is the Cascadia Subduction Earthquake. The Cascadia Earthquake is predicted to cause a 9.0 magnitude earthquake along the Cascadia subduction zone which spans the states of Washington, Oregon, and Northern California. The earthquake will affect the regional infrastructure systems such as electric lines and roads. These systems are needed for everything we do in our daily lives- including transportation, communication, and business. There is a push in these states by governments and various organizations to start mitigating and preparing for the Cascadia Earthquake. One such organization is an Oregon state non-profit, Cascadia Prepared, based out of Eugene. Cascadia Prepared’s mission is to ensure the “social and economic survival of [the state of Oregon] by strengthening the resilience of [Oregon’s] critical infrastructure lifelines to mitigate losses” during and after a Cascadia Earthquake (Cascadia Prepared, 2019). This organization is looking towards creating or implementing a resilience scorecard that will analyze the vulnerability of Oregon cities or counties to the earthquake. Finding Faults: Comparing Resilience Scorecards June 2019 Page | 9 The report examined five different scorecards as well as current resilience assessment plans being implemented within the state of Oregon with the goal of informing Cascadia Prepared’s decision to move forward with their proposed project. I also focused on information around the feasibility and barriers to implementing and completing the scorecard process at a city or county level, in general, and in relation to having Cascadia Prepared lead such a process. Research Questions of Project I went into the project, wanting to answer four questions around resilience scorecards and helping to inform Cascadia Prepared’s process. These were: - What are the different ways scorecards measure resilience? - What are the critical elements for measuring resilience? - What is the feasibility and barriers to implementing a scorecard? - What are recommendations Cascadia Prepared can use for creating and implementing their scorecard? To answer these questions, I researched current available data and gaps around resilience assessments through a literature review. LITERATURE REVIEW Disaster resilience assessments are a powerful tool that provides support for the majority of disciplined fields (Cai, 2018), including political, economic, and business. Resilience assessments help communities measure different systems, both internally and externally. With the increase in data-driven and cost-efficient solutions to problems that arise, the underlying logic of the common phrase “what gets measured, gets done” justifies the use of resilience assessments (Winderl, 2014). Assessments can help characterize a community’s baseline resiliency, as well as prioritize their needs and goals for increased resilience (Cutter, 2016). Communities, increasingly, want data that can monitor progress and show the cost-benefit analysis to help inform their actions and policies (Burton, 2015). Assessments are effective, rigorous tools that make the conversation of how to collectively reduce the socio-economic impact of disruptive events accessible for non-academic community stakeholders (Arbon, 2012). While measurement tools can't create a resilient community, they can provide insight, help communities take ownership of their agendas, and ensure the integration of diversity and inclusion (Fitzpatrick, 2016). These resilience assessments measure and compare various categories within a community against each other, which makes it ironic how difficult it is to find studies that compare various types of assessments-especially scorecards- against each other. In the following sections, I review scorecard effectiveness, validity of scorecards, and research gaps. Finding Faults: Comparing Resilience Scorecards June 2019 Page | 10 Measuring Resilience Scorecard Effectiveness Measuring the effectiveness of resilience scorecards has been infrequently attempted. There have been more studies that compare indices (Cutter, 2016), due to the structure of using large data sets that help analyze the differences among the various indices. There are also a lot more indices than scorecards, which give more empirical data points of reference. Of studies that have been done, evaluating scorecards has typically been of one or two scorecard that are evaluated against each other or against resilience research standards that compare the elements of disaster measured, levels of resilience (input, output, outcome, and impact), and the smallest unit of measurement (Winderl, 2014). This approach is not as common an evaluation method as the use of case studies that look at a critical component of resilience. The other method of measuring scorecard effectiveness uses different benchmarks seen throughout the post-disaster recovery process as a form of validation. Some benchmarks used are the post-disaster population change, multi-period fieldwork observation of recovery, and economic damage from the disaster itself (Cai, 2018). These benchmark criteria methods lend themselves well to the rampant use of case studies as a framework that measures the effectiveness of an assessment (Khazai, 2018). Throughout the research, some articles showed how scorecards were valid based on criteria considered relevant for different communities or regions such as the Mississippi Gulf region post-Hurricane Katrina (Burton, 2015); Lalitpur, Nepal before and after the Gorkha earthquake (Khazai, 2018); and others. All of these approaches at defining the validity of various scorecards show the issues that are currently being examined within the field of disaster resilience. Issues with Validity The Oxford Dictionary defines validity as the "state of being legally or officially accepted," which is enforced through a certain standard among academic research that typically includes large data sets and repeatable findings (Burton, 2015). The push for better metrics that include the community in the process and access to technology has led to the creation of some exciting assessment tools. Unfortunately, there are relatively few third-party studies that implement and/or test these measurements for resilience (Cutter, 2016). The absence of studies is due to the challenging nature of developing comprehensive standards, with researchers asking (Burton, 2015): - What set of indicators provide the best comparative assessment among communities? - And to what extent do these indicators predict a known and measurable outcome? The majority of these assessments have not been empirically validated because issues are surrounding common definitions and unavailable data. The fact that resilience is such a personalized journey for each community makes validating different measurements difficult. Out of 174 articles, only 10.3% had used empirical methods to verify their derived resilience indices (Burton, 2015). This scarcity of validated techniques is due to significant issues around the definition of resilience, the clarity of whether resilience measurements should be an overall Finding Faults: Comparing Resilience Scorecards June 2019 Page | 11 concept or specific to a type of hazard, and how resilience is related to other terms like vulnerability, recovery, adaptability, and sustainability (Cai, 2018). Until recently, there were not many easily accessible tools that communities could use to assess their resilience in preparing for an emergency event at the community or regional level rather than individual (Fitzpatrick, 2016). There is no published validated tool that communities can use to measure their resilience in preparing for an event because of the very complex and tangled set of systems that are affected by a disruptive event (Torrens, 2012). Until the scientific and practitioner communities agree on fundamental foci of community disaster resilience—like the set of assets, processes, or a combination of the two—the question of a valid measurement form will be unanswered (Cutter, 2016). This will be especially true for resilience scorecards, which are currently overlooked throughout the research. Research Gaps The complexity around resilience assessments is caused by problems on basic definitions within the field of resilience, the confusion around empirically based analysis, and vast community differences that influence the effectiveness of different assessments. These issues, along with a large number of assessment types, create misunderstanding among communities looking for resilience assessments. The few studies on validity around resilience assessments have been done within academic papers that analyze the methodology of the assessments, while few have attempted to address a practitioner or community's concerns around the differences between what is being measured and how the processes differentiate on the implementation side. Therefore, this paper will examine these different scorecards with Cascadia Prepared in mind. Cascadia Prepared is focusing on a single disaster, the Cascadia earthquake, and is looking at a regional scope. The hope is to give baseline information on the different available scorecards, so communities take advantage of existing resources. While the resilience field continues to help individual communities and regions work with different types of assessments, the pursuit of more objective metrics will be informed by that journey. The research questions for this project will address some of these research gaps and lead to a document that can be used by Cascadia Prepared as well as other communities looking into using resilience scorecards. Finding Faults: Comparing Resilience Scorecards June 2019 Page | 12 BACKGROUND INFORMATION The many resilience scorecards out there were narrowed down to the examined five scorecards below. The research also looked into current resilience plans and assessments that have been already implemented within the state of Oregon. Here is some background on the plans themselves. UN Disaster Resilience Scorecard for Cities The UN Disaster Resilience Scorecard for Cities, created by the UN, AECOM, and IBM, is structured around the "Ten Essentials for Making Cities Resilient" (2017). The scorecard can be used as a standalone tool. It provides a set of assessments that will allow all levels of governments to assess and monitor their disaster resilience, as well as review the progress and challenges in their resilience action plan implementation of the Sendai Framework for Disaster Risk Reduction. This is another UN document that gives recommendations in ways for communities to become more resilient. It requires a city to categorize their hazards into ‘most probable' and ‘most severe,' then address the vulnerability of their financial, social, and physical infrastructure to these hazards (UNISDR, 2017). There are helpful tips mentioned throughout. This scorecard has similar components to a Natural Hazard Mitigation Plan from FEMA, with more of an emphasis on post-disaster documentation. The ranking is numerical from 1-3 for a more preliminary view or 1-5 for a detailed assessment, with the higher numbers indicating more resilience (UNISDR, 2017). Plan Integration for Resilience Scorecard The Plan Integration for Resilience scorecard, created by the National Research Council, is based on the United States’ FEMA 2013 Local Mitigation Planning Handbook and 2015 Plan Integration Handbook: Linking Local Planning Efforts (2017). These documents mention the issues around cities "plethora of plans problem," which leads to inconsistencies across a community's network of policies (National Resource Council 2017). This can have various positive and negative effects on hazard resilience among different agencies and within communities. The scorecard focuses on the current plans and policies within a community and evaluates how those plans integrate to encourage or hinder community resilience. A community's planning department or local government officials can implement the scorecard in an internal audit of the plans and come up with better policies to help support more resilience within the community. The ranking is numerical and based around an ordinal content analysis system, with -1 being a policy that is less resilient, +1 being more resilient, 0 being neutral affect, and NA meaning there were no resilience policies mentioned within the document (National Resource Council 2017). Resilience Performance Scorecard The Resilience Performance Scorecard, created by the Global Earthquake Model Foundation (GEM), is based on the UN’s Ten Essentials of a Resilient City and the Sendai Framework for Finding Faults: Comparing Resilience Scorecards June 2019 Page | 13 Disaster Risk Reduction (2017). It is the only scorecard that focuses on a specific hazard – earthquakes – and assesses the community’s vulnerability by identifying gaps that can lead to the construction of resilience management strategies or benchmarks. Local government officials are the targeted audience, with this assessment looking at the city level of government. This scorecard is unique to focus on one hazard and also to specifically call out social inequality as a topic to address around hazards and resilience. The process is led by the Global Earthquake Model organization, with the community being a part of stakeholder meetings and data collection. The scores are determined by GEM’s software that computes the numerical tallies, with a higher number indicating more resilience (Global Earthquake Model Foundation, 2017). This is the only scorecard that did not have the community lead the process itself, instead opting for a consulting based format. Resilient Communities Scorecard The Resilient Communities Scorecard, created by the Vermont National Resources Council, is based on smart growth principles and land use planning (2013). It is for local and regional governments to identify ways to promote smart growth principles, build community resilience, and adapt to climate change (Vermont National Resources Council, 2013). It has sections for communities to fill out and recommendations that come from the framework of smart growth. There are interesting connections that the scorecard makes about smart growth and resilience. The approach of this scorecard is for communities to assess community members’ preferences and perceptions to resilience and smart growth instead of as an assessment tool to compare communities. The format is pre-ranked multiple choice, with the higher scores indicating more resilience (Vermont National Resources Council, 2013). Torrens Community Resilience Toolkit and Scorecard The Torrens Community Resilience Toolkit and Scorecard, created by Torrens Resilient Institute through the Australian Commonwealth Government National Emergency Management Program, is based on the Australian National Strategy for Disaster Resilience (2012). It provides a point-in-time snapshot of some key measurements critical to resilience for local or regional communities. There is information on how to answer the questions presented in the different sections and the data needed to implement. This scorecard can be used for a quick general look at a community's resilience or do an in-depth analysis of the topics covered. It is similar to the UN Disaster Resilience Scorecard but does not have as many categories or subcategories. The ranking is numerical, with the higher number being more resilient (Torrens Resilience Institute, 2012). Other Resilience Plans By focusing on scorecards, I addressed an area that has not been as well researched from the information gathered in my literature review and was able to compare similar resilience assessments. The other documents I reviewed were plans that currently measure resilience within the state of Oregon. These were: Finding Faults: Comparing Resilience Scorecards June 2019 Page | 14 - Natural Hazard Mitigation Plans- which measures similar topics to the scorecards I viewed and is already being used by counties and cities throughout Oregon for their hazards (FEMA, 2019) - Oregon Resilience Plan - which was a plan done by the Oregon Seismic Safety Policy Advisory Commission to measure the physical and organizational resilience of the state of Oregon to the Cascadia Subduction Earthquake (OSSPAC, 2013). This data created a base knowledge of information as well as helped inform my interviews and focus group discussions. Natural Hazard Mitigation Plan (NHMP) In 2000 the Robert T. Stafford Disaster Relief and Emergency Assistance Act was amended by the Disaster Mitigation Act of 2000 to “reduce the loss of life and property, human suffering, economic disruption, and disaster assistance costs resulting from natural disasters (FEMA, 2018). This legislation mandated that all levels of government were required to develop a hazard mitigation plan to receive certain types of non-emergency assistance through the Hazard Mitigation Assistance Programs. The Plan was a response to this regulatory requirement and based around the Title 44 Code of Federal Regulations 201.6 (FEMA, 2018). The NHMP is similar to a resilience scorecard because it requires the community to complete a risk assessment, mitigation strategy, and include public involvement in the process. During the risk assessment, a community identifies all of the natural hazards it is vulnerable to and comes up with action items to mitigate that vulnerability. The state and counties have NHMPs, with the special districts and cities within each county creating separate addendums to their county’s plan (FEMA, 2018). The plan has to be updated every five years to maintain compliance with federal regulations. While this plan assesses communities for multiple hazards, the Oregon Resilience Plan addresses Oregon's vulnerability to the specific hazard of the Cascadia Subduction Earthquake. Oregon Resilience Plan In 2013, the Oregon Seismic Safety Policy Advisory Commission (OSSPAC) led a planning effort to look into the effects of the Cascadia Subduction Zone Earthquake on the state of Oregon. This was a response to the 2011 Japan Tohoku earthquake and subsequent tsunami and backed by House Resolution 3 (OSSPAC, 2013). The Oregon Resilience Plan was broken down into four main regions of impact - the tsunami zone, coastal zone, valley zone, and eastern zone. From there the plan implementers had different stakeholder groups within those regions identify the time it would take to get important services back up and running, otherwise known as lifeline infrastructure capacity (OSSPAC, 2013). These stakeholder groups involved various governmental, private sector, and resilience expert participants. The private sector participants were included to gauge the capacity of businesses to survive interruptions of infrastructure services such as power, transportation, etc. Its findings were not optimistic for most of Oregon’s capacity to weather the Cascadia Earthquake. The plan did start a conversation around how to improve those odds and created a base capacity measurement that helped to validate its recommendations to state and local on how to improve local, regional, and Finding Faults: Comparing Resilience Scorecards June 2019 Page | 15 statewide resilience (OSSPAC, 2013). This was a comprehensive starting point for figuring out some base measurements, top-down policy, and infrastructure recommendations to help the people of Oregon comprehend a disaster event that they have no previous experience with. In 2018, there was a five-year progress report published by OSSPAC on what had been accomplished since the 2013 plan. This is a positive step in the right direction of making scorecards more than just a point in time snapshot and instead, a document that evolves with the state's changes in capacity and resilience. This process was similar to a scorecard, in the fact it looked at the major factors that would be affected by the Cascadia Subduction Earthquake and then created "a path of policy and investment priorities for the next fifty years" to help improve the capacity of Oregon to endure through the Earthquake (OSSPAC, 2013). Its regional level of impact measurement did not accurately address the variation among different counties or cities, which have widely different capacities to handle the earthquake and intensity of vulnerability to the earthquake. The original plan made a recommendation for local Oregon communities to use the framework and methodology of the Oregon Resilience Plan to “conduct more refined assessments that consider local seismic and tsunami hazards, and develop community-specific recommendations to meet their response and recovery needs” (OSSPAC, 2013). This recommendation could be the impetus for Cascadia Prepared to implement the Oregon Resilience Plan framework. This focus on the physical and, to an extent, the organizational capacities of Oregon was part of the project's scope. This limitation focused the information and findings to topics that are relatively easy to find data and metrics for, while also overlooking the third critical type of resilience: the social or individual capacity of a community. The report was a top-down look at resilience for the whole state, which is important and a good first step in measuring resilience. Let's go through the methods of this report. METHODS AND DATA The methods used to answer the research questions were content analysis, interviews, and a focus group. Content Analysis My research focused on resilience scorecards due to Cascadia Prepared’s emphasis on using scorecards, the research that has already been conducted by Cutter (2016) on other resilience indices or toolkits, and to compare a different type of assessment. Resilience assessments include indices, toolkits, and scorecards. These different types of assessments have similarities in the way they measure categories, but the intent behind the assessments differ. Indices mainly rely on quantitative data for generating an aggregate index value using weighted sums, while toolkits have a broader scope that can incorporate indices and scorecards as well as suggest recommendations for improving resilience (Cutter, 2016). Scorecards, on the other hand, are used to obtain values for performance against resilience categories and often use dichotomous or multiple-choice questions, calculated statistical values, or judgements and/or Finding Faults: Comparing Resilience Scorecards June 2019 Page | 16 perceptions that are then scored with a numerical ranking (Rowcliffe, 2000). The content analysis examined second-hand information. I found six different scorecards during my research, which were then narrowed down to five because one of the scorecards, the CoastSmart Communities Scorecard, focused on coastal communities only and didn’t address the whole scope of inland Oregon communities. The five scorecards I examined were: UN Disaster Resilience Scorecard for Cities, Plan Integration for Resilience Scorecard, Resilience Performance Scorecard, Resilient Communities Scorecard, and the Torrens Community Resilience Toolkit and Scorecard. These scorecards all focus on the city community level and focus on multiple hazards, with the exception of Resilience Performance Scorecard – which looks at earthquakes. Interviews For the interviews, I contacted resilience experts around the state of Oregon - those who have worked with scorecards before, and professionals that work within the different levels of Oregon government. Through recommendations by Cascadia Prepared and previous knowledge of Oregon resilience professionals, I contacted 12 people to be interviewed. I was only able to interview 6 people due to professionals’ limited time capacity or non-responsiveness to multiple inquiries. The interviewees were contacted through email, and interviews were conducted either in-person or over the phone. The conversations with professionals who did participate were focused on the feasibility and barriers to implementing a scorecard within the state of Oregon.1 Out of the interviews, certain themes were determined around scorecard categories to measure and barriers to implementing the scorecard – such as issues around measuring social capacity. These viewpoints complemented and expanded on the focus group’s findings. The number of interviews is a limitation to the project because there is only a small sample of opinions; this has been slightly offset by the focus group that was also conducted. Focus Group The focus group was conducted in late April and included six local resilience professionals that included emergency managers and public health officials at various government and organizational levels. An online poll of interested participants was included in the inquiry emails and informed the time and location of the focus group. The hour session examined the feasibility and barriers to implementing a scorecard, with a specific focus into critical categories to measure, current hazard and disaster data sources within Oregon, and process barriers to completing the scorecard.2 These insights were gathered through individually written notes, the use of a whiteboard, and group discussion. The group discussion was recorded and transcribed. The transcription was where key themes and concerns were drawn from. The feedback from the resilience professionals addressed some major concerns around an outside organization leading the implementation process. This, along with the content analysis and interviews, informed the findings of this report. 1 See Appendix A for interview questions 2 See Appendix B for focus group agenda Finding Faults: Comparing Resilience Scorecards June 2019 Page | 17 Limitations The project has some major limitations due to time, participation, and the researcher's experience. Due to time, the project was not able to get feedback from all counties or regions of Oregon. This created a localized viewpoint from Lane County, which is not representative of the vast regional variations within Oregon. The participants generously gave insight into the scorecard process. Partially due to time and my inexperience, there were not enough participants to create empirically relevant data. Though those who did participate were experts in their fields and the topic of resilience, which does generate some validity to identified themes. These limitations should be considered when reviewing the following findings. FINDINGS These findings were gathered from the above methods and answer the four research questions by examining the resilience scorecard process, important categories to measure, barriers to implementation and completion of a scorecard, and specific considerations for Cascadia Prepared. Resilience Scorecard Processes A similar process was used for the majority of scorecards. The scorecards typically used working groups, or a single working group made up of community members and topic experts. These groups lead the process of stakeholder groups or data mining for the different sections measured (UN Disaster Resilience Scorecard; Resilient Communities Scorecard; Torrens Community Resilience Scorecard; Plan Integration Scorecard). An example is the Plan Figure 1: Plan Integration Scorecard Team Integration Scorecard, which visually shows Layout the four groups or "teams" required to complete this scorecard’s process (National Resource Institute, 2017). As seen in Figure 1, the leadership team oversees the policy, mapping, and engagement teams, which address the various steps required during implementation. This structure helps to ensure the scorecard is completed in a timely manner because there is invested leadership for each phase as well as an overarching project management team. Source: National Resource Institute, 2017 A variation to this community drawn leadership is the Resilience Performance Scorecard, where the Global Earthquake Model organization takes charge of the implementation (GEM, 2017). This scorecard is different due to the focus on a single hazard – earthquakes – and by not having the community in charge of the process itself. The scorecard has community stakeholder Finding Faults: Comparing Resilience Scorecards June 2019 Page | 18 groups for data and idea gathering, with the organization that created the scorecard – the Global Earthquake Model Foundation (GEM) – designated as the lead consultant. The foundation leads the community through the categories and has a patented GEM software to compute the scores for the communities (GEM, 2017). While this does not allow the community to take ownership of the process, it does help communities that do not have the technical capacity for implementing a scorecard. Different approaches and scorecard objectives led to different ranking systems, as we will explore below. Resilience Scorecard Content The layout of the sections and directions on the information necessary for each scorecard is slightly different, though most of the scorecards examined have pre-scored subsections that address the data variables needed for each overarching category. The rankings around the scorecards were typically a numerical rating of various categories. Among the scorecard, different emphasis was put on different variables. Each scorecard used different terminology and defined resilience around a theme of adaption and recovery from a disaster event with a community bouncing back stronger than before (UN Disaster Resilience Scorecard; Resilient Communities Scorecard; Torrens Community Resilience Scorecard; Plan Integration Scorecard; Resilience Performance Scorecard). I will examine three examples of different layouts to show some of the variations among these scorecards: the UN Disaster Resilience Scorecard, the Torrens Community Resilience Scorecard, and the Plan Integration Scorecard. UN Disaster Resilience Scorecard The UN Disaster Resilience Scorecard is one of the most detailed among the group examined. It starts each section with a definition and various topics that will be covered; an example is shown in the first statement of Figure 2. This example was taken from the “Understand and Strengthen Societal Capacity for Resilience” section (UNISDR, 2017). Next, there is information on the types of data needed to compute each sections' score, through a data list and the "Comments” column located within each subsection (UNISDR, 2017). The other columns give clear information and definition to the topic and data being measured, with each subsection being indicated by a number, such as 7.1, and each detailed topic of data required to measure these subsections being indicated by an extended number of the topic, subsection, and data topic, such as 7.1.1. Figure 2: UN Disaster Resilience Scorecard Example Finding Faults: Comparing Resilience Scorecards June 2019 Page | 19 Source UNIDR, 2017 The information format is shown differently within different scorecards' layout. The UN Disaster Resilience Scorecard and others have overarching themes on resilience that are then subdivided into important variables that measure a community's strength or weakness within the overall category. The main difference between the UN Disaster Resilience Scorecard and the Torrens Community Resilience Scorecard is the level of detailed guidance around the sections, with the former measuring more categories and subsections per category. Torrens Community Resilience Scorecard The Torrens Community Resilience Scorecard follows a similar format to the previous scorecard through the use of an overarching section, and pre-ranked subsections that include information on the data needed to compute those scores (Torrens Resilient Institute, 2012). This scorecard has four main sections that are prompted by a question, with Figure 3 showing part of the social resilience section. The question is then broken down into subsection questions that are ranked through different ranking levels determined by the Torrens Resilient Institute (2012). There is some information on the types of data and resources needed to compute these subsection scores, but it is not as detailed as the UN scorecard. Figure 3: Torrens Community Resilience Scorecard Example Source: Torrens Resilience Institute, 2012 Finding Faults: Comparing Resilience Scorecards June 2019 Page | 20 It is one of the earliest published scorecards from the list I examined. Instead of having various framing techniques, the plan uses questions to prompt the community to answer both the categories and subcategories. The questions give clear direction to a community on what topics are important to measuring the different sections, which helps to keep the number of pages and required data sources low. It also promotes communities to use the scorecard with their own individual needs in mind (Torrens Resilient Institute, 2012). The two previous scorecard examples show what is typical among the other scorecards, with the next example showing a different approach to measuring resilience. Plan Integration Scorecard The Plan Integration Scorecard uses content analysis of a community’s plans to determine resilience. The scorecard requires policy and planning experts to complete a content analysis of the community’s planning documents, which affects the format of the scorecard (National Resource Institute, 2017). Figure 4 shows some examples of the format and information given to communities using this scorecard. In the example, table 2.3 and 2.1 show some policies and plans that the scorecard gives as samples of what a plan content reviewer might experience. The scorecard assumes a certain level of expertise among its implementers and thus gives examples instead of directive questions or categories (National Resource Institute, 2017). These examples are not an exhaustive list because of the need for a digestible document, the effects of a policy can be different depending on the community, and resilience specific policies vary greatly between communities. Figure 5 is a typical layout used when conducting a content analysis of a single plan, which can be expanded to all the plans in a community. The policy is placed into the left column, then a reviewer marks if that policy applies to the different sub- districts that are mapped out by the community prior to reviewing the plans. The number of resilient policies and number of mentions indicates how resilient a community is and creates the end score (National Resource Institute, 2017). Figure 4: Plan Integration Scorecard Example Source: National Resource Institute, 2017 Finding Faults: Comparing Resilience Scorecards June 2019 Page | 21 Figure 5: Plan Integration Scorecard Content Analysis Format Source: National Resource Institute, 2017 This scorecard has more ambiguity around its examples and format because of how one conducts a content analysis and the variable effects of policies depending on the community. The experts' knowledge is leaned on heavily to complete the scorecard. These three scorecard examples, along with the other scorecards examined, all addressed the different categories that measure resilience. Wrap Up There were two main approaches to conveying data: the use of detailed and structured information or more of a process and example based approach. The Plan Integration Scorecard is more process based and open ended due to the nature of what it is measuring, the policies and their affects within various community plans. While the UN Disaster Resilience Scorecard and Torrens Community Resilience Scorecard had a prescriptive approach with detailed and structured information. The UN and Torrens scorecards were an example of the approach taken by the Resilience Performance Scorecard and the Resilient Communities Scorecard. This allows for more adaptability because laypeople can implement the scorecard. These approaches all depend on what a community wants to measure and what capacities it has to implement the various scorecards. Important Categories to address in Resilience Scorecards The resilience scorecards all incorporated various categories that measured a community's vulnerability holistically. These categories addressed the essential capacities that resilience assessments must determine – the physical, social, and organizational capacity of a community to identified hazards (Arbon 2012). Physical capacity is usually based around infrastructure that has more documentation, can be updated to certain engineering or other physically set Finding Faults: Comparing Resilience Scorecards June 2019 Page | 22 standards based on past experience, and is more static over time compared to social capacity. Physical capacity is better completed in a top-down fashion (Phone Interview 5.9.19) because it is based on plans and other government policies that are usually the responsibility of the government. Measuring the physical capacity of a community – such as the infrastructure's resilience - is more straightforward than measuring the social or organizational capacity. This has to do with the lack of official plans attached to social capacity, the subjectivity of the types of measurement systems, and the ever-changing situation around these more human-based capacities (Phone Interview, 5.8.19). These are further challenged by the lack of empirically valid data collection methodologies and difficulties in properly analyzing these informal networks (Cutter, 2016). Social and organizational capacity are based on individuals and the networks they create. Therefore it is best measured in a bottom-up approach, starting from the smallest unit, the individual. All are vital to creating a well-rounded resilience scorecard that can address the needs of a community to become more resilient. Though each scorecard used different terms to explain each category, they were placed into ten broad categories. These categories were then narrowed down to six critical ones through research, interviews, and the focus group. Below the table shows the “critical” categories and definitions that were gathered from the scorecards.3 Table 2: Critical Categories to Measure Resilience Category Definition Risk and Vulnerability Social and physical vulnerability Financial Capacity Financing capabilities; incentives; penalties Social Capacity for Resilience Interaction between formal and informal institutions (social networks) Infrastructure Resilience Water; energy; transportation; communication; education; food systems; healthcare; etc. Disaster Response Emergency plans; emergency supplies Disaster Recovery Plans Continuity; pre-event recovery plans; capital improvement plans 3 See Appendix C for the full list of ten categories. Finding Faults: Comparing Resilience Scorecards June 2019 Page | 23 As stated in the focus group, these categories address the critical post-disaster topics needed for the protection of life and property (5.29.19). Throughout the research, it was implied that people typically approach disasters with a post-event mindset. This mindset focuses on the event itself as well as the response to, and recovery from a disaster event (International Economic Development Council, 2019). The other mindset is pre-event, which approaches the categories from the frame of preparation and mitigation (International Economic Development Council, 2019). Two other categories were also mentioned by the focus group and interviews that are directly related to the pre-event mindset: urban development and the natural environment. Urban Development looks into how resilience is incorporated in the land use, building code, and zoning code, while the natural environment encompasses the conservation and use of the environment to enhance resilience. Both topics address the underlying structures that affect a community’s physical resilience, with urban development showing how the policies of a community affect how it is built and the natural environment addressing the large influence of the surrounding nature on resilience (Focus Group 5.29.19). These topics can directly impact a community's ability to mitigate the vulnerability by having stricter seismic building codes or put funds towards preparation actions such as backup generators, etc. This is important for creating resilient communities and something that resilience experts emphasized when analyzing the categories from a different viewpoint. Implementing and Completing a Scorecard The interviews and focus group mainly concentrated on barriers and feasibility of implementing and completing a scorecard. These opinions from Oregon resilience professionals informed the majority of this section. Barriers The interviewees and focus group participants cited several barriers to completing a resilience scorecard, including resources, finances, participation coordination, and time having a major effect on the process. These barriers will be addressed below in terms of the impact they have on implementing a scorecard within the scope of measuring the state of Oregon’s vulnerability to the Cascadia Earthquake. The identified barriers for this specific scope can be applied to other projects that involve resilience scorecards. Resources The term resources can mean many different things; for this report, we are defining it as the methodology and data needed to complete a scorecard. Resilience scorecards use a methodology to determine what to account for and how to measure it, which is typically not a problem for the community implementing it. It can become a problem when one measures multiple communities at one time because the terms used in resilience can mean many different things to different people or in different community contexts (Focus Group 4.29.19). An example is the different variations of the term resilience among the scorecards examined. Each scorecard has a similar basic understanding of the underlying framework but adds various Finding Faults: Comparing Resilience Scorecards June 2019 Page | 24 additions such as including "preservation and restoration of essential basic structures and functions" (UNISDR, 2017) or “thinks in the long term and is able to reorganize and renew itself” (Vermont National Resources Council, 2013). Thus terminology with consistent definitions behind each term are fundamental to the success or failure of implementation (Phone Interview 5.8.19; In-Person Interview 5.6.19). An interviewee suggested a solution to these issues, which was having the lead organization be aware of the different shades of meaning for each term and create an exhaustive definition that is conveyed to participants before any data is collected (In Person Interview 5.6.19). The next step will be to collect the data itself, which brings its own challenges. The collection of data is fraught with gaps due to differences between different data sources and differences among the communities themselves. It is well known that finding social data on community vulnerabilities is very difficult. This difficulty stems from the subjectivity of social data (Cutter, 2016), and the fact that vulnerable populations within communities are not easy to measure (In-person Interview, 5.10.19). For example, immigrant populations are very vulnerable to natural hazards because of their typically low financial cushion and lack of institutional protection. Collecting the necessary data on these populations is difficult because many do not want to be found by government organizations due to the potential of deportation and separation of families (In-person Interview, 5.10.19). Surveys are good measurements of social capacity because they address the smallest unit of measurement, an individual's capacity (Interviews 2019). For surveys, one has to be very diligent to account for self-selection and get information from those not typically willing to answer (Phone Interview 5.8.19). These gaps do not exist as much for the physical capacity of a community because it is more static in nature and not easily ignored. Local governments are also usually in charge of the maintenance for physical infrastructure, which leads to public works directors having basic information on the vulnerabilities and strengths (In Person Interview 5.7.19). However, the gaps in data do show up among different communities and levels of government. This is caused by the different financial capacity of communities to account for these topics, with wealthier and more prominent communities having more resources to measure them. These difficulties in data collection and methodology can affect the timeline and financial structure of a large scale project, as well as create a situation where the scorecard does not measure data in an “apples to apples” manner that results in categories not being easily comparable. Finances Finances are vital to conducting any project, especially for those with large geographic scopes such as this statewide project. Money is needed for every step of the process – such as compensating staff time, providing space for stakeholder groups, analyzing the data - as well as the need for incentives to participate. It is common knowledge that there are limited public funds to address all of the problems communities are facing. The research has shown that resilience is perceived as something that may or may not happen in the future, which makes it difficult to argue for committing funds towards only resilience measures because there are so many other issues happening in the present moment (In Person Interview 5.7.19). Finding Faults: Comparing Resilience Scorecards June 2019 Page | 25 For this process to be more than a point in time snapshot and to facilitate participation, there need to be dedicated funds towards completing projects that are informed by the findings of the scorecard (Focus Group 4.29.19; Phone Interview 5.8.19). With the geographical scope of this project being so great, finding enough financial incentives to help support communities during and after the scorecard process will be difficult. Finances are always a tricky barrier to overcome and can significantly affect how well participation happens in our busy worlds. Participation Throughout the research, an emphasis was put on the importance and dependence on multiple stakeholders to implement a scorecard. The multiple stakeholders are from within the community, state and federal governments, and topic experts such as economists and engineers. These stakeholders need to be committed to the process and have trust in the lead organization. This trust is very important and was emphasized throughout all of the interviews and focus group. Both the literature and focus groups highlighted common questions required in the process of building trust: - What are we measuring? - Why is it important to participate? - What are the benefits or rewards? - How is this information going to be used? - What is the level of confidentiality? These questions need to be answered to the satisfaction of all stakeholders in order to facilitate their participation and make the process as timely as possible (Focus Group 4.29.19). Another potential issue to participation is the focus on a specific hazard – the Cascadia Earthquake—, which will not cause the same physical damage across the entire state. The eastern side of the state will not experience the violent shaking that west of the I-5 corridor will have, which makes it more difficult to elicit participation. These communities’ concerns involve a different view of the scorecard categories and thus could affect the comparability across the state as well as the motivation to participate. There is a case for measuring the resilience of communities to incoming disaster refugees and emergency agencies that could solve this issue (OSSPAC, 2013). Therefore it will be imperative to make inquiries to communities individualized and constantly refer back to the questions above when interacting with participants. Also keep in mind that Oregonians have not experienced a Cascadia Earthquake event since the Native Americans were the sole inhabitants of the land. Thus, people do not have a good idea for how resilient they really are because they have never experienced it (In Person Interview 5.7.19). Some solutions to this issue around encouraging statewide participation would be to include multiple hazards, instead of focusing on just one, or to use other types of situations, such as severe storm power outages and other disruptive events, to measure the resilience of a community (Phone Interview 5.8.19). Finding Faults: Comparing Resilience Scorecards June 2019 Page | 26 Time A project that has multiple inputs and needs large amounts of data, like scorecards, is where time issues become complicated. Without committed participants, there will be a lot more time spent on getting all the necessary stakeholders to the table and collecting the types of data and expertise required by the project team (Arbon, 2012). As any project manager knows, the time frame of the project and how it changes throughout the implementation is a major factor that can make or break a project – especially if there are funding deadlines to contend with. The focus group and interviews had estimates of completing the scorecard that ranged from 3-6 months, 1-2 years, or never being completed because of not overcoming other barriers and running out of time. To solve this, there must be accurate expectations of the time frames for the various project sections as well as an experienced project manager to handle any issues that might lead to the process not being able to finish (Focus Group 5.29.19). By overcoming these barriers, and any others that come up during the process, the chance of a successfully completed resilience scorecard is attainable. Feasibility Throughout the research, focus group discussion, and interviews, people's general perception of resilience scorecards were as a point in time snapshot of all categories necessary to accurately assess a community’s vulnerability, which typically resulted in no long term improvement or follow up. Though resilience scorecards measure topics that are very important to determining resilience, they are all subjective in how and what data sources and measurements they use to determine a community's score. This subjectivity can create a false sense of security among government officials and community members because it can deem a community resilient to a hazard based on faulty information or misguided preconceived ideas around how a hazard might manifest (In Person Interview 5.8.19). The false sense of security can be exasperated by the point in time nature of most scorecard implementations. Many communities use it to assess vulnerabilities, but do not complete action items that can increase their resilience or monitor and follow up after the initial ranking (Focus Group 4.29.19; Phone Interview 5.8.19; In-Person Interview 5.7.19, etc.). Resilience is constantly evolving - positively and negatively - as the formal and informal networks around physical, social, and organizational capacities shift over time, which makes the issues around scorecards being a "one and done" solution very dangerous because it simplifies a complex solution and creates more hurdles for feasibly completing multiple scorecards in the long term. To implement a resilience scorecard successfully, there are a lot of different parts that need to be present and working. In Oregon, cities have home rule and are required by state policies to plan their comprehensive plan, municipal development code, and annual budget. While these plans are essential for resilience, many cities - chiefly smaller ones - depend on their county government to provide disaster response or other services related to infrastructure, finance, economic development, urban growth, etc. (Focus Group, 4.29.19). Counties also are already required by the federal government to create a Natural Hazard Mitigation Plan to receive pre and post-disaster funds, with cities and special districts creating plans that annex into the overall county plan (FEMA, 2019). The different plans, data sources, and feasibility for different Finding Faults: Comparing Resilience Scorecards June 2019 Page | 27 levels of government to have the capacity for creating them were compiled through research, the focus group and interviews. By using the process created by the FEMA Natural Hazard Mitigation Plans, we can infer that the county has a lot of the data needed for a resilience scorecard. The county could be a natural choice to focus these efforts on because there is already a process in place to include cities and special districts that are located within the county’s boundaries and have important information on the policies and plans that make communities resilient. For projects with a statewide scope, there needs to be a lead organization or entity that is in charge of getting a uniform message out, trusted by local governments, able to collect and synthesize the data gathered, and hopefully administer the incentive funds for future resilience projects. There could be various lead agencies – the state, Councils of Governments, or an outside organization such as Cascadia Prepared. Right now FEMA and OSSPAC are lead agencies on the resilience assessments being currently implemented, and could be good examples to examine for future implementation. No matter who is in charge, they will have to contend with some limitations that will be affected by an organization’s reputation as a neutral third party that has widespread recognition. These limitations include communities’ issue with giving highly sensitive or secure information, not having the right contacts with the people needed to complete the scorecard, and troubles around getting buy-in from participants. There would also need to be an incentive to partner with the organization, which might result in more of a financial commitment than possible in this current political and financial climate. Below are some recommendations for Cascadia Prepared. Considerations for Cascadia Prepared Cascadia Prepared has many different avenues available to support the communities of Oregon in its mission of improving Oregon’s resilience to the Cascadia Earthquake. These options are: to not produce a scorecard, to produce a scorecard that addresses either the physical and organizational resilience, or to address the social resilience of communities across the state. Option 1: Not Produce a Resilience Scorecard The first is to not produce a scorecard at all, which was not a finding that was expected at the beginning of the research project. Many different resilience assessments are already used within the state of Oregon, including the Oregon Resilience Plan and the Natural Hazard Mitigation Plan (NHMP). These two plans are both mandated and backed by governmental support, which gives them legitimacy among the stakeholders that participate in their process (OSSPAC, 2013; FEMA, 2019). A lot of extensive data was gathered initially for these plans, with the continuation of informed data being inputted and considered during every updated version of each plan. They both include all of the components necessary for a resilience scorecard, such as community buy-in, standardized processes, follow up, and the NHMPs have financial support through FEMA’s grant program (FEMA, 2019). Several interviewees commented that with the number of assessments already being completed by communities, adding another assessment onto the pile will not help those communities improve their resilience. The resources that would be put towards a scorecard could be put towards other actions, such as helping the Finding Faults: Comparing Resilience Scorecards June 2019 Page | 28 communities implement some of the action items already identified by other assessments. Most Natural Hazard Mitigation Plans within Oregon have action items around public education and awareness because it is mandated by FEMA through goal 5 “promote public awareness” (FEMA, 2019). Another goal is to “augment emergency services” (FEMA, 2019), which could include working towards identifying and upkeep on designated shelter locations and buildings. A lot of communities have limited resources and time to implement all of these action items. Cascadia Prepared could be a great partner to help with the implementation of these action items. Option 2: Produce a Resilience Scorecard This scorecard would address the recommendation offered by the Oregon Resilience Plan. The Oregon Resilience Plan acknowledged that regional resilience information is not as applicable for communities to implement due to the extreme variations among Oregon’s regional and county areas, therefore it recommended for the use of the framework in the Oregon Resilience Plan to produce an in-depth analysis at the community level (OSSPAC, 2013). This would address the physical and organizational capacities of resilience. The state of Oregon has a very diverse and wide range of environments and communities (OSSPAC, 2013). These communities vary widely and therefore require information that is tailored to their own specific vulnerabilities and strengthens around responding and recovering from the Cascade Earthquake. The communities would use the framework designed by the Oregon Resilience Plan, with a lot of the information being collected from resilience assessments already used in Oregon, like the community’s NHMP, and experts (OSSPAC, 2013; In-Person Interview 5.7.19). Natural Hazard Mitigation Plans (NHMPs) address the vulnerability of a community to multiple hazards, including the Cascadia Earthquake. The Oregon Resilience Plan framework would help communities narrow down their recovery timeframe for getting everything back to normal after the event happens. By having both of these plans, a community could have more data to back resilience based projects and give residents more awareness on how resilient they will have to be in the event of the earthquake. General Scorecard Process Recommendations There are some considerations when implementing a scorecard that apply specifically to Cascadia Prepared, which include the issue of being a relatively unknown non-profit and the need for financial backing. As stated above, to get participants involved in an assessment, the lead organization has to have legitimacy as an organization and incentives for participating. Cascadia Prepared is going to have a tough time answering the question of "Why should communities participate with them?" because many communities do not know of this nonprofit and therefore might not be able to be trusted in the eyes of the community as a holder of potentially sensitive emergency information (Focus Group 4.29.19). Therefore how they approach communities will be especially important during the initial stages and will require an emphasis on Cascadia Prepared being a partner instead of an outside watchdog group that is there to critique community preparedness (Interviews 2019). Finding Faults: Comparing Resilience Scorecards June 2019 Page | 29 This can be helped by financial incentives, but the funding of those incentives is going to be complicated due to the number of other resilience assessments and competition from other grant applications. By embarking on their mission, Cascadia Prepared should be focused on the triple bottom line of helping communities, expanding their organization, and creating self- sufficiency among individuals (Phone Interview 5.8.19). Hopefully, these considerations will help Cascadia Prepared determine the best route to pursue and improve their ability to overcome obstacles. Option 3: Address Communities’ Social Capacity Another option would address the social capacity of resilience within communities. This has a lot of potential to help communities, the growth of Cascadia Prepared, and resilience data sources. It involves Cascadia Prepared using one of the scorecards mentioned above or an outside organizational resilience diagnostic tool (such as the New Zealand Resilient Organizations Diagnostic Tool) to address something that is typically not represented well in resilience assessments: social capacity. Social capacity within the resilience field is a challenging topic to measure due to the subjectivity and barriers around collecting this highly variable and informal information (Cutter 2016). The organization could focus on the individual’s preparedness and scale up to the neighborhood and other social network levels, which would allow for the process to grow with the organization and create some on the ground change. Addressing social capacity would allow for the implementation of better overall community resilience (In Person Interview 4.30.19). This option is an exciting opportunity for Cascadia Prepared because they could focus exclusively on social capacity and give it the time and resources needed to do a comprehensive and individualized look into communities. This "next step" for resilience could be significantly addressed by outside organizations like Cascadia Prepared. The data could be used to inform neighborhood or community population resilience, especially around traditionally vulnerable communities that take more time and resources to measure accurately. As one might image, scorecards that are addressing the physical and organizational side of resilience as well, typically, cannot put the necessary resources towards social capacity because their scope has to address two other major components of resilience. Cascadia Prepared could be part of a process that would work on community action items around individual preparedness and education, while also creating and implementing diagnostic toolkits to measure and improve a community’s resilience. This could be accomplished through pre and post surveys or community toolkits that could create databases to bridge the gap between the general information typically stated in resilience plans and the actual individual data for each community (Phone Interview 5.8.19). Cascadia Prepared is in a prime position to implement this social capacity diagnostic and improvement plan due to their ability to specialize and position as a non-profit organization that doesn’t have to content with government limitations. Finding Faults: Comparing Resilience Scorecards June 2019 Page | 30 CONCLUSION Though there are already various types of resilience assessments currently being used to measure resilience, resilience scorecards are one assessment type that can help communities measure their vulnerabilities to natural hazards. An important takeaway is for more research to be done on the data sources and assessments already executed within communities, so resources can be spent wisely to further the accomplishment of actual community resilience through action item projects. If there are no sufficient plans, then there are some great scorecards that have been designed to meet a community’s specific resilience measurement needs - which you can review in this report. Otherwise, there is a real problem of resilience assessments being used to “check a box” and usually not followed up on to build actual resilience. This will hopefully inform the initial phases of a lead organization or community on their process to complete a resilience scorecard and will create less unhelpful redundancy among plans. This redundancy can be changed through the responsible allocation of resources and efforts towards resilient actions instead of just measurement. Finding Faults: Comparing Resilience Scorecards June 2019 Page | 31 RESOURCES: Cutter, S.L. (2016) The landscape of disaster resilience indicators in the USA. Nat Hazards 80: 741. https://doi.org/10.1007/s11069-015-1993-2 Cai, H., Lam, N., Qiang, Y., Zou, L. Correll, R., Mihunov, V. (2018). A synthesis of disaster resilience measurement methods and indices. https://www.sciencedirect.com/science/article/pii/S2212420918304618 Winderl, T. (2014). Disaster resilience measurements: Stocktaking of ongoing efforts in developing systems for measuring resilience. United Nations Development Programme. https://tinyurl.com/yb8ld2zz Arbon, P., Gebbie, K., Cusack, L., Perera, S., Verdonk, S. (2012). Developing a model and tool to measure community disaster resilience. Torrens Resilience Institute. https://tinyurl.com/ybotr3l8 Burton, C. G. (2015). A Validation of Metrics for Community Resilience to Natural Hazards and Disasters Using the Recovery from Hurricane Katrina as a Case Study. Annals of the Association of American Geographers, 105(1): 67-86, DOI: 10.1080/00045608.2014.960039. https://tinyurl.com/ybhxn8fh Fitzpatrick, T. (2016). Community Disaster Resilience. Planning and Response: 57-85. Khazai, B., Anhorn, J., Burton, C. (2018). Resilience Performance Scorecard: Measuring urban disaster resilience at multiple levels of geography with case study application to Lalitpur, Nepal. https://www.sciencedirect.com/science/article/pii/S2212420918303662 Vaughan, E. and Henly-Shepard, S. (2018). Resilience Measurement Practical Guidance Note Series 1: Risk and Resilience Assessments. Produced by Mercy Corps as part of the Resilience Evaluation, Analysis and Learning (REAL) Associate Award. https://tinyurl.com/ydcyt4ez Norris, F.H., Stevens, S.P., Pfefferbaum, B. et al. (2008). Community Resilience as a Metaphor, Theory, Set of Capacities, and Strategy for Disaster Readiness. Am J Community Psychol 41: 127. https://doi.org/10.1007/s10464-007-9156-6 Sherrieb, K., Norris, F.H., Galea, S. (2010). Measuring Capacities for Community Resilience. Soc Indic Res 99: 227. https://doi.org/10.1007/s11205-010-9576-9 Federal Emergency Management Agency [FEMA]. (updated February 2018). Local hazard mitigation planning fact sheet. https://www.fema.gov/media-library/assets/documents/5756 Finding Faults: Comparing Resilience Scorecards June 2019 Page | 32 Cascadia Prepared. Our mission and goals. Visited April 2019 https://www.cprep.org/mission.html Rowcliffe, P., Lewis, M., Port, A. (2000) The community resilience manual: A resource for rural recovery and renewal. Centre for Community Enterprise. FEMA. (2019) Hazard mitigation planning process. Visited April 2019 https://www.fema.gov/hazard-mitigation-planning-process National Resource Institute. (2017). Plan integration for resilience scorecard guidebook. Texas A&M University: Institute for Sustainable Communities Global Earthquake Model Foundation. (2017). Resilience performance methodology. Vermont National Resources Council. (2013). Resilient communities scorecard: A tool for assessing your community. United Nations Office for Disaster Risk Reduction [UNISDR]. (2017). Disaster resilience scorecard for cities. Torrens Resilience Institute. (2012). Community disaster resilience toolkit and scorecard. Oregon Seismic Safety Policy Advisory Commission [OSSPAC] (2013). The Oregon resilience plan. https://www.oregon.gov/oem/documents/oregon_resilience_plan_final.pdf International Economic Development Council. (Visited June 2019). Phases of disaster. http://restoreyoureconomy.org/disaster-overview/phases-of-disaster/ Finding Faults: Comparing Resilience Scorecards June 2019 Page | 33 APPENDIX A - INTERVIEW GUIDE Introduction & Oral Voluntary Consent We have briefly discussed the research topic, but to give you more detail this study is focused on comparing different resilience scorecards through the lens of a local non-profit's- Cascadia Prepared- requirements for a scorecard to help the state of Oregon measure its vulnerability to the Cascadia Subduction Earthquake. Resilience scorecards are a type of assessment that uses a "systems thinking approach" to quantify and identify the interconnections and vulnerabilities of an area (Cutter, 2016). I am looking at the feasibility of completing a scorecard, resilience related data within Oregon that is currently available, and the challenges of administering resilience scorecards. Creating or implementing a scorecard to measure Oregon’s vulnerability to the Cascadia Earthquake will help to funnel the limited resources to the areas of greatest need. It should be clearly stated that your participation in this study is completely voluntary. If at any time during the interview you feel uncomfortable and do not want to answer a question please tell me to pass and we can continue to another question. You may at any time stop the interview and request not to be involved. This interview holds minimal risk to participants; therefore, you will not be asked any questions that would put you in the greater physical or psychological risk of an ordinary day. All responses will be anonymous and will in no way be connected to your business or department. Additionally, your responses will not be used as a reflection of your business or agency's viewpoint. Knowing these terms, would you like to continue as a participant for this study? During the interview I’m going to ask you questions about your perceptions on the feasibility of scorecards, the barriers to conducting a scorecard, types of data that is currently collected in Oregon, and your perceptions on the completing a scorecard. I will be taking notes. Introductory questions  What is your job title and/or profession?  What does resilience mean to you?  Do you know what a resilience scorecard is?  If not, explain: A resilience scorecard is a type of assessment model that helps to measure a community’s vulnerability and risk to a specific hazard or multiple hazards. It typically looks at the physical, organizational, and social capital areas and how those topics interact with each other.  What are the most important factors/categories you would want to be measured in a scorecard?  What would be some issues in measuring those categories?  What level of government – city, county, or COG- would you measure that information at?  What are some of the barriers you see to completing a whole resilience scorecard? o What barriers did you deal with during the Oregon Resilience Plan? o What are some barriers to having Cascadia Prepared implement this scorecard?  How long do you think it would take to complete a resilience scorecard? Do you have any other thoughts you would like to add on this topic? Thank you for your time. Finding Faults: Comparing Resilience Scorecards June 2019 Page | 34 APPENDIX B - FOCUS GROUP AGENDA Date: April 29, 2019 I. Welcome and Introductions a. Looking over and signing the Consent Form b. Presentation of Background Information c. Icebreaker II. Dialogue around Different Questions a. Important scorecard categories b. Issues around measuring those categories i. Different levels of government ii. Data sources c. Process of implementing a scorecard i. What could the process look like? ii. Obstacles or barriers to implementation iii. Feasibility of data collection at different levels of government III. Summary and Wrap Up Prompting Questions for Each Section A. From the information given in the presentation and your knowledge of hazards, how would you rank the scorecard categories? o What are the 3 most important categories to include? - written o Are there any categories that are not as feasible to measure in Oregon?- written B. Please get out the comparison excel spreadsheet. What do you think the feasibility of measuring these categories, that we just ranked, at different levels of government? If you have any expertise in a certain government level please focus on filling out that column first and, if you have time, I would really appreciate your impressions of the other levels of government. o Do they have the ability to do financial incentives? o What data sources would show this information? o What are some of those data sources you might know of? – written o Why do you think there are gaps in the level of information? C. Process – now we are going to look into the process of how we could implement a scorecard in Oregon o What does the process of implementing a scorecard look like to you? -written  Who is handing over the data? Players needed to produce the data  How does this process look like? o What are some obstacles to completing a scorecard? – written  What are some obstacles that you see happening during this collection of data?  How would you incentivize this transfer of data? o How could you overcome them? - written o Knowing what you know of the data and obstacles to implementing a scorecard, What level of government would you pick to implement the scorecard at? – written o Could either go into a specific type of government level or be general Finding Faults: Comparing Resilience Scorecards June 2019 Page | 35 APPENDIX C - RESILIENCE SCORECARD CATEGORIES Category Definition Risk and Vulnerability Social and physical vulnerability Financial Capacity Financing capabilities; incentives; penalties Social Capacity for Interaction between formal and informal institutions (social Resilience networks) Infrastructure Resilience Water; energy; transportation; communication; education; food systems; healthcare; etc. Disaster Response Emergency plans; emergency supplies Disaster Recovery Plans Continuity; pre-event recovery plans; capital improvement plans Urban Development Land use; building codes; zoning Natural Environment Conservation; use of the environment to enhance resilience Institutional Capacity for Public awareness and public education around disasters and Resilience hazards Protection of Cultural Historic buildings; cultural vulnerability Heritage Finding Faults: Comparing Resilience Scorecards June 2019 Page | 36