A fuzzy model to assess the resilience of Protection and Civil Defense Organizations

Resilience is the intrinsic ability of a system to adjust its functioning prior to, during, or following changes and disturbances, so that it can sustain required operations under expected and unexpected conditions. Protection and Civil Defense Organizations (PCDOs), communities and cities deal with disaster management involving routine, non-routine and even unpredictable/unforeseen situations with varying degrees of complexity. It is important that such organizations continually assess their resilience, enable them to learn on their weaknesses and real capacities to cope with emergency situations. This research aimed the development of an Organization Resilience Indicator System (ORIS) based on a fuzzy model to enable PCDOs self-assesses their resilience. Based on a literature review on organizational and community’s resilience, a system of resilience indicators was defined. This system was validated by experts using fuzzy set theory to aggregate opinions in the development of a resilience ideal pattern. Then, the resilience of four PCDO organizations was self-evaluated. The results were accordingly to maturity level of the organizations evaluated, indicating that the ORIS is valuable to measure PCDOs resilience.


Introduction
Disasters initiated by extreme weather conditions hitting vulnerable populations due climate change are dramatically increasing in poor and developing countries around the world. In Brazil, landslides due heavy rain in the mountain region near Rio de Janeiro caused around 900 deaths in 2011, and recently, in 2019 more 30 people died when buildings collapse in vulnerable areas of the Rio de Janeiro city, again after heavy rains.
The Hyogo and Sendai Frameworks [UNISDR (2005), (2015)] for Disaster Risk Reduction indicated the need to build more resilient communities. The Hyogo Framework for Action-HFA defines five priorities for actions and guidelines that, if followed by the communities' government, would improve their resilience against disasters. The goal of both frameworks is to reduce loss of lives and the overall resources needed to cope with disaster management. The Hyogo and Sendai framework to build more resilient cities is based in high level normative concept capturing basic ideas concerning obligations that governments should adopt to fulfil the planned goals. In contrast, system resilience is a descriptive concept, about the functioning of a system. Holling and Gunderson (2002), define resilience as the disturbance level a system can absorb without changes in its structure. In an organization, resilience is related to the activities to anticipate and circumvent disturbances that threat their existence and/or their primary goals (Hale and Heijer 2006). Van de Vorm et al. (2011) applied the resilience concepts at organizational, team and individual levels. They proposed a framework to connect those levels, where organizational resilience deals with the high organizational level including management commitment to safety, just culture (Dekker 2012), necessary resources to cope with unexpected events (Sheridan 2008). At team level, team resilience enable people to deal system variability to prevent potential damage (Morel et al. 2008) as showed by Carvalho et al. (2005) when nuclear power plant operators' multi-disciplinary team solve "micro incidents" using tacit and implicit knowledge, rather than to follow strictly rules and procedures. At individual level, resilience can be viewed as an individual construct that involves personal skills including ability to learn, respond and anticipate problems (Van de Vorm et al. 2011).
However, modern PCDOs are organizations are conceived to support countries disaster risk reduction or resilience strategies. These strategies are related to the dimensions of the Hyogo protocol (UNISDR 2005) such as (CDEM 2019): • promote management of hazards in a way that contributes to safety and wellbeing; • encourages communities participation in the risk management process; • provides for planning and preparation for emergencies, and for response and recovery; • requires local authorities to coordinate reduction, readiness, response and recovery activities; • provides a basis for the integration of national and local planning and activity; • encourages coordination across a wide range of agencies, recognizing that emergencies are multi-agency events affecting all parts of society.
Therefore, the resilience of a PCDO cannot be viewed and measured in the same way as the resilience of an industrial organization, mainly because PCDO's resilience cannot be viewed apart from tasks and actions made by other organizations involved in the risk reduction and resilience strategies. PCDOs must be aware on their own resilience potential as an organization, and on the overall level of resilience of the environment or community in which they are inserted, in order to know what to do to promote the resilience against disasters and to support the government's National Strategies, which is the focus of the indicator system proposed in this article.
In the disaster management scientific literature, most of the theoretical development in the resilience domain is related to Community Resilience issues, as well as ways to measure the community resilience level in disaster situations (Cutter et al. 2010(Cutter et al. , 2008Yoon et al. 2015;Dasgupta and Shaw 2015;Horney et al. 2016;Saja et al. 2018;Rapaport et al. 2018). However, regarding PCDOs, the public organizations responsible for implementation of disaster risk management actions, there is no specific research to assess their resilient potential, which is very important especially in underdeveloped countries. To cope with recommendations of Hyogo and Sendai Frameworks for Disaster Risk Reduction building more resilient cities, PCDOs have an increasing interest to improve their own resilience to deal with risks and disasters. To do so, they need a tool that helps them to assess and perceive their resilient potential to act systematically and integrally in all phases of the risk and disaster management cycle.
Most of indicators used for community resilience in disaster management measure what happened in past situations (e.g. number of fatalities, events, and people in risk areas). Sudmeier et al. (2013), in their empirical research to develop practical guidance for assessing community resilience, recognized that a focus on outcome indicators was a limitation on their research, as they found process or leading indicators that were not adequately treated in their research. Considering organizational resilience as an organizational process (Hale and Heijer 2006) or a functional concept (Holling and Gunderson 2002), indicators pointing to the past may not be enough to capture the resilient potential of an organization. Several authors already argue that leading or trending indicators are more adequate to measure the resilient potential of organizations (Reason 1997;Woods 2006;Grecco et al. 2014).
The objective of this research was the development of an integrated system of indicators to be applied in PCDOs to enable a self-evaluation of the resilience of these agencies in relation to risk and disaster management. The resilience indicator system developed in this paper, combines multiple indicators at the different phases of the disaster management cycle (prevention, response, mitigation, recovery) that measure a range of underlying properties of resilience to produce a single metric of resilience (Tate 2012). The system was based on indicators of organizational, government and institutional resilience categories of systems for assessing community resilience conjugated with indicators from systems for assessing the resilience of organizations.
In a practical sense, this paper points to the possibility of the managers of the PDCOs apply day to day the developed resilience indicator system to monitor their resilient capability, as a continuous process of self-learning, to correct any existing problem and plan new actions in order to increase the organizational resilience in the PDCOs and, consequently, the community resilience, and the urban resilience at all. It is according to the Hollnagel's (2010)  A Fuzzy Set Theory (FST) approach was used to deal with the uncertainty and imprecision in the score weighting processes of indicators and to aggregate opinions to ensuring consistency among different evaluators. The final indicator system was tested comparing the PCDOs self-evaluation of resilience made by their managers of 4 major Brazilian cities. It is expected that such system of resilience indicators allow PCDOs to self-monitor their resilience, to better know and reflect on its practices, and to learn and evaluate what are real issues on risk and disaster management regarding resilience.
The research design and development were based on the grounded theory (Glaser and Strauss 2017), using participatory methods for data collection and validation largely applied on scientific areas as ergonomics, social ecology, social design, and so forth. The participatory ways of constructing an organizational resilience indicator system, which will be managed by part of the participants who were involved in its development enable the construction of a more sustainable system along the time. This research also contributes to provide an indicator system addressed to PCDOs' of developing countries, which lack of rational and managerial tools and ways to manage risks and disasters and their own resilience. Cutter et al. (2010) define community resilience as a set of capacities that can be promoted through interventions and policies that help to build and enhance the community's capacity to respond to and recover from disasters. On the other hand, for Hale and Heijer (2006) resilience in an organization relates to the management characteristic of activities to anticipate and circumvent the threats to their existence and to their primary goals. Resilient capacities in organizations are demonstrated in the ability to manage severe pressures and conflicts between safety and the organization's primary production or performance goals. As it can be seen from these two approaches, the term resilience refers to a wide variety of definitions, which are employed in different contexts, such as in communities exposed to risks and disasters, and in organizations that need to innovate to survive.

Resilience and disaster risk management
According to the United Nations International Strategy for Disaster Risk Reduction-UNISDR (2009), "resilience is "the ability of a system, community or society, exposed to risks, resisting, absorbing, adapting and recovering from the effects of a danger in a timely and efficient manner, including by preserving and restoring its essential basic structures and functions." In disasters, such as great earthquakes, tsunamis and floods, the collective mobilization of a whole community is a particular challenge, because, in some situations, previous public contingency plans become overloaded by the size, scale and magnitude of the event, as well as by the infrastructure destruction (Comfort 2016;Funabashi and Kitazawa 2012). Therefore, more and more emphasis is placed on mitigation and preventive actions, which are undertaken before the disaster, in order to reduce losses. According to Nakamura et al. (2017), community education in disaster prevention, coupled with effective relationships and communications within communities, helps to establish a system in which the community can gain a strong awareness of disaster prevention and, for example, evacuating risk areas on a voluntary basis.
Another important issue related to community resilience to disasters is that the overall city organizations, especially those responsible for the services/infrastructure, need to be resilient, as they provide goods and services that enable the economic and general wellbeing of the communities in which they are located. When these organizations are hit by disasters, they may be negatively impacted and unable to provide the goods and services for which they are proposed (Sadiq and Graham 2016). McManus et al. (2008) add that the resilience of organizations involved with disaster management issues contributes directly to the speed and success of community recovery, because without critical services such as energy, water, sanitation, transportation, health, etc., communities will find it more difficult to respond and recover from the effects of a disaster. This highlights that to be considered resilient, PCDOs workers' must be aware and committed to the process of building community resilience, addressing the disaster risks, as well as the ways to respond to emergency situations, and to recover from those events.
This awareness is reflected in some studies in the scientific literature, whose purpose is to assess the resilience of organizational systems. Rose and Krausmann (2013) proposed a framework, based on economic theory and empirical studies, with the aim of creating a resilience index for the economic recovery of companies after a disaster. Lee et al. (2013) have developed a tool to measure and compare the resilience of organizations to identify the strengths and weaknesses of resilience and to help organizations understand how resilient they are to develop strategies for improvement. Chan et al. (2014) developed a study on indicators to improve government strategies for disaster prevention in an urban area. Huber et al. (2012) were based on the concepts of Resilience Engineering to propose a method of developing indicators and a tool to evaluate organizational resilience. Labaka et al. (2015) have provided a framework on resilience policies that must be implemented by critical infrastructure organizations to improve the resilience of these organizations. Sadiq and Graham (2016) discussed the requirements of organizational preparedness for disasters. Carvalho et al. (2016) used a fuzzy system to propose a participatory model for assessing maturity in relation to Disaster Risk Reduction, using indicators aligned with the Hyogo Framework of Action (HFA). Gomes et al., (2014) analyzed the resilience of team performance in a nuclear emergency response exercise; however, they did not describe the resilience indicators used. Rabbani et al. (2016) analyzed preventive strategies to mitigate the consequences of disasters and make resilient organizations. The aim was to find the most appropriate strategy according to the characteristics of the organization. Jung and Song (2015) investigated how the structural arrangements for collaboration within emergency management networks influence disaster resilience, finding that interorganizational collaboration is critical to community resilience. UNISDR (2009) defines DRM Disaster Risk Management-DRM as the systemic process of using administrative guidelines, organizations skills and operational capabilities to implement strategies, policies and improve capacities, to reduce adverse impacts and the risk of disasters. This is done through prevention, mitigation and preparedness activities and measures. However, the studies do not consider many of the practices that guide DRM as proposed by UNISDR. The literature review indicated that they don't address indicators to measure the organizational resilience related to government organizations that deal with DRM.
In Brazil, the DRM is described in the National Policy of Protection and Civil Defense-NPPCDE (Brasil 2012). The National Policy comprises the actions of prevention, mitigation, preparation, response and recovery. According to CEPED/UFSC (2014), prevention, mitigation and preparation actions should happen all the time (when there is no disaster situation) and involve: risk identification, assessment and generation of knowledge, prevention and mitigation of future risk, creation of a culture of prevention and the preparation and improvement of the response system of organizations and society. Disaster Management (DM), in turn, takes place in the post-disaster situation and comprises the planning, coordination, and execution of response and recovery actions. Response actions are classified into relief actions and assistance to affected populations. Recovery actions begin after the stabilization of the situation due to response actions, and are classified in reconstruction and restoration actions.
The Protection and Civil Defense Organizations (PCDOs) of Brazilian cities are responsible to develop the actions according to the National policy, which follows the Disaster Risk Management phases as proposed by the United Nations International Strategy for Disaster Risk Reduction. Like any other organization, PCDOs are also vulnerable to risks and disasters, and with them the people who work there. These organizations, in addition to caring for the populations at risk, need to take care of their own staff and equipment to be able to provide the support to the communities vulnerable to risks and disasters. The workers of PCDOs work in periods without disasters, conducting actions in the prevention and mitigation of risks, preparing themselves and the cities, and also in the phases of disaster response-that is critical-and recovery. It is worth highlighting that urban resilience is dependent on the resilience of these organizations, taking into account the phases of the risk and disaster management cycle: prevention, mitigation, preparation, response and recovery. PCDOs, however, do not generally have a tool or other resource that makes them aware of their resilient potential or actual performance in addressing the risks and disasters in cities. Organizations' self-knowledge about their own resilience helps them to cope with risks and disasters and planning, with a view to improving their own resilience and, consequently, urban resilience. Therefore, it is important that PCDOs can continuously assess their resilience to know their weaknesses and capabilities to make better decisions and actions. A system to evaluate organizational resilience, expressed through indicators, can contribute to these organizations to have a continuous self-knowledge on their resilience and to elaborate their plans and actions according to this learning process, improving their resilience to deal with risk and disaster management in their cities.

Method: using fuzzy set theory for development of PCDOs indicators
The method proposed includes the following steps: 1. Development the Organizational Resilience Indicator System (ORIS), an indicator framework for each phase of disaster management cycle-prevention, mitigation, preparation, response and recovery-based on literature review; 2. Multiprofessional validation: determination of an ideal resilience pattern based on experts' evaluation of ORIS using fuzzy set theory; 3. Situated Validation: self-evaluation of the resilience of four Brazilian PCDOs to test the system, comparing the PCDOs managers' evaluations with the ideal resilience pattern.
The indicators for PCDOs resilience were based on the perceptions experts who work in Protection and Civil Defense Organizations of different Brazilian cities have on the relative importance of each indicator to achieve the goals of disaster risk reduction strategies. The scores assigned for each indicator depends on personal knowledge and data available/ perceived. The set of indicators developed-the indicator system-can be used to evaluated in which extend their organization accomplish with each indicator metric, enabling the PCDOs self-evaluation of their resilience.
Score weighing is based on human reasoning through fuzzy information, coming from people mental models and the available information on the situation/environment, resulting in inexact and vague data sets. In Fuzzy Logic, the true value of a variable can be any real number between 0 and 1. It is employed to handle inexact data, considering the concept of partial truth, where the truth value may range between completely true and completely false (Zadeh 1965;Zimmermann 1996). These models have the capability of recognizing, representing, manipulating, interpreting, and using data and information that are vague and lack certainty (Zadeh 1996). A fuzzy number is expressed as a fuzzy set defining a fuzzy interval in the real number. Because the boundary of this interval is ambiguous, the interval is also a fuzzy set. Among the various shapes of fuzzy number, triangular fuzzy number is the most popular one (Pedrycz 1994). A triangular fuzzy number Ã can be denoted by (a, b, c) (see Fig. 1) and its membership function is: In fuzzy set theory variable values are described by linguistic variables. If Age is a linguistic variable, its values are linguistic rather than numerical, i.e., young, not young, very young, quite young, old, not very old and not very young, etc. Linguistic variables provide ways to characterize phenomena which are too complex or too ill-defined to be described by numbers. In the proposed PCDOs' indicator system, the linguistic variables are described by 4 terms (Unimportant, Little Important, Important and Very Important) represented by triangular fuzzy numbers. Experts in the domain (civil defense) should assign the importance of each indicator; therefore the indicator system reflects the actual knowledge and experience of each expert who participate in the process (Ishikawa et al. 1993). In this research the Hsu Chen method was used to aggregate (or discard) opinions of individuals to form a group consensus opinion (Hsu and Chen 1996).

Results
In the next subsections a detailed description of each methodological step and the respective results are presented.Ã

Development of ORIS through literature review
A systematic review of the scientific literature on disaster resilience indicators was carried out initially in databases containing Brazilian journals in Portuguese language (Scielo) and in other databases that contained international journals (Science Direct, Web of Science, Scopus) in English language. After an initial search and owing to the objective of this study-to identify organizational indicators and organizational indicators systems of disaster resilience that could be applied in a Protection and Civil Defense Organization in order to assess their organizational resilience-we realize that the Scopus database included most of the articles important for the research. Therefore the systematic literature review uses only the Scopus database.
So, the methodological procedure consisted, firstly, in combining the keywords: resilience indicators, resilience measurement, and resilience index with the keyword disasters, using the search feature "And" as follows: resilience indicators and disasters (first search), resilience measurement and disasters (second search), and resilience index and disasters (third search). Two criteria were used to select the articles: • Articles on indicators for assessing the resilience of communities inserted in a disaster context. It was chosen because organizational indicators, themselves, are rare and it was supposed that some community resilience indicator could be considered as an organizational indicator as well; • Articles on indicators for evaluating the resilience of organizations inserted in a disaster or crisis context.
Promptly, a new refinement of the research was made, considering the focus of this work in the category of so-called organizational, institutional and governmental indicators. Besides the academic papers, we also used in documents related to UN World Conferences on Disaster Risk Reduction (DRR) (UNISDR 2005(UNISDR , 2009(UNISDR , 2015 and National Protection and Civil Defense legislation (Brasil 2012). Table 1 shows the indicators candidates chosen from the literature review for each phase of the disaster management cycle.
Based on Table 1, the authors developed the Organizational Resilience Indicator System-ORIS with 34 indicators, divided into the 4 phases of the disaster risk management cycle as shown in Table 2. The operationalization or a way to measure an indicator is called a "metric". A metric denotes how the indicator is measured, whereas an indicator denotes something that one wishes to measure with the use or more metrics. (Grecco et al. 2014). The corresponding indicator metrics are presented in Table 3.

Multiprofessional validation
A systematic approach for the multiprofessional validation of ORIS, a system with 34 indicators comprising the four phases of disasters (dimensions), and the concepts of fuzzy set theory used are described in this section. This approach aimed to obtain from experts in PCDOs the relative importance of each ORIS indicator. The relative importance of the indicators as defined by the experts, with their respective importance values, forms a resilience pattern, showing how an organization can reach it resilient potential. The Resilience Pattern resulting from the multiprofessional validation can be compared with the actual resilience organization level, indicating how to improve the PCDO resilience.   To aggregate multiple experts' opinions and to deal with their consistency, the similarity method developed by Hsu and Chen (1996) was used. The method for multiprofessional validation was carried out, following the steps: 1. Selection of experts; 2. Calculation of the relative importance of the experts; 3. Choice of linguistic terms and membership functions; 4. Determination of the importance degree of the indicators; 5. Aggregation of the fuzzy opinions; 6. Definition of the resilience pattern based on experts opinions.
The selection of experts is a critical factor, because the reliability and quality of the results depend on the quality of the experts. However, in general, all people with recognized knowledge and experience who are or have already been involved in PCDOs are candidates for the evaluation process of the indicators. The experts selected were  3. There are own means running disclosure, such as official websites, social media and leaflets to engage the public during the planning efforts for disaster risk reduction as well as to disseminate information relating to disaster risk, and ongoing/planned operations 1.4. Alarm and warning systems are installed and operate 24 h/day 1.5. There is a database for storing information relating to the risks of disasters and disasters on data logging, recording damage and losses 1.6. There are reserve funds for institutional strengthening and the resources available for investment in the structure and internal organization of the institution have been sufficient 1.7. Agents and managers have adequate experience in protection and civil defense 1.8. There are skilled workers for the use of new technologies (software, alerts and monitoring applications, drones, rescue technologies) for the prevention of impacts of disasters 1.9. The DRR actions are planned with other institutions, public agencies, political leaders and other sectors/segments of society (members of vulnerable communities, community groups, volunteers and Red Cross, universities, professional councils, religious institutions, private institutions, NGOs) 1.10. The civil defense and protection agents are encouraged and supported to practice their creativity and innovation capabilities to help in planning and strategy 1.11. There is a process of participatory decision-making and leadership delegating decision making to subordinates 1.12. Agents fulfill decisions correctly 1.13. There are protocols for standardized actions, such as plans, programs, forms and reports 1.14. The number of agents and managers with effective job protection (PCDO employees) is higher than the ones with temporary assignments 1.15. The institution demands works of prevention and recovery to the competent organs/ departments of government 1.16. The municipal body for protection and civil defense is linked directly or indirectly with the mayor and community leaders to develop, implement and carry out municipal protection policies and civil defense in the county. Table 3). The relative importance of the experts depend on his/her profile such as experience, work time in PCDOs, and so forth. A questionnaire was used to identify the profile of each expert. The relative importance of an expert (RIE) E i (i = 1, 2, 3… k) is defined by Eq. 1, where ts i is the total score of the expert i. 3. There is a communication plan and good relationship with the community and the press 2.4. There are teams in 24 h on duty every day of the year 2.5. The training and preparation of simulated exercises are performed with the participation of vulnerable communities 2.6. The training and preparation of simulated exercises are performed with the participation of members of school communities, hospitals, health units, industry organizations and trade 2.7. The protection and civil defense agents participating in training programs for disaster response, performing regularly, training and drills 2.8. The simulated field and table are performed with scenarios reflecting realism participants 2.9. There is stockpiling of materials for emergency humanitarian assistance 2.10. There is collaboration with other agencies/departments in defining guidelines for the recovery plan for the areas affected by disasters 2.11. Plans and strategies are reviewed and are updated/redesigned based on experiences. Response 3.1. During the response actions the institution makes every effort to use human and material resources (vehicles, equipment, tools and machinery) predefined in the contingency plan 3.2. There are warning and alarm systems which cover risks. Also, there are technologies/equipment and trained personnel to access hard to reach areas 3.3. There are warning and alarm systems which cover risks. Also, there are technologies/equipment and trained personnel to access hard to reach areas 3.4. The team meets at the end of the answer to evaluate the actions and compare them to what was predicted and share lessons learned. Recovery and restoration 4.1. Plans and standards for recovery and restoration of infrastructure and affected buildings are influenced so that the competent bodies adopt building standards/rebuild better than before the disaster 4.2. There mobilization to ensure financial resources for the recovery and restoration of affected areas 4.3. There are meetings with the affected community and the bodies responsible for recovery and restoration to influence the decisions, so that they consider disaster risk reduction principles

professionals of several Brazilian Protection and Civil Defense Organizations (see
The experts used the linguistic terms, U (Unimportant), LI (Little Important), I (Important) and VI (Very Important) to evaluate the relative importance of each indicator, weighting each indicator according the respective metric. Table 4 shows the importance degrees and triangular fuzzy numbers for linguistic terms and the graphic representations of membership functions for the linguistic terms U (Unimportant), LI (Little Important), I (Important) and VI (Very Important) are shown in Fig. 2.
The group opinion aggregation method (Hsu and Chen 1996) was used to combine the experts' opinions. The agreement degree (A) between expert E i and expert E j is determined  In case of completely different opinions of 2 experts, resulting in zero agreement, should be adjusted with other methods like reach in order to reach a consensus, a common intersection at a fixed α-level cut (Lee 1996). After the calculation of agreement degrees between experts the agreement matrix (M) gives an insight into the agreement between the experts.
The relative agreement of an expert (RAE) E i (i = 1, 2, 3… k) is given by Eq. 3.
The fuzzy number Ñ (Eq. 6) that combine expert's opinions is the fuzzy value of each leading indicator, which is also represented by a triangular fuzzy number. In Eq. 6, ñ i , represents triangular fuzzy numbers of the linguistic terms.
The ideal resilience pattern is calculated normalizing the importance degree (NI) of each leading indicator (Eq. 7). The normalized importance (NI) of leading indicators is given by defuzification of respective triangular fuzzy number Ñ (x i , y i , z i ), where y i is the importance degree.
The team of experts selected for determination of the resilience pattern included 11 PCDOs professionals of the 5 important Brazilian states capitals: 1 from Belo Horizonte, 1 from Curitiba, 2 from Recife, 3 from João Pessoa, and 4 from Natal. The relative NI i = y i greater value of y importance score assigned for each expert was determined using a questionnaire with 6 questions, each question with items associate with a score. Table 5 shows the relative importance of each expert calculated by Eq. 1.
The calculations made for each indicator was done in the same way as the one made for the leading indicator 1.5 (database) of the phase 1 (prevention and mitigation) as described below. The triangular fuzzy numbers of opinions data are shown in Table 6. The experts' agreement degrees were calculated using Eq. 2. The intersection area and the total area of two different opinions of experts ("Important" and "Very Important"), used in this calculation, are shown in Figs. 3 and 4. The agreement matrix between the experts E i and E j is shown in Table 7. In Table 8 there is the relative importance, the relative agreements, the relative agreement degrees, and the consensus coefficients of the experts.

Situated validation: application and test of the proposed method
The situated validation of ORIS was done in a pilot test, by asking managers of 4 Brazilian PCDOs to assess to what extend their organizations comply with the ORIS and its  metrics. The 4 cities were the same mentioned above, without the participation of João Pessoa city. To keep the anonymity of the evaluated organizations, they are identified here as PCDO1, PCDO2, PCDO3, and PCDO4. In PCDO4 the organizational resilience assessment questionnaire was applied directly to the PCDO4 manager by one member of the research team. In the other PCDOs, the questionnaires were sent by e-mail and answered by the respective managers, who agreed to participate in the research, without the presence of the researchers.
The method aims to compare the perceptions of PCDOs experts on the resilience of his/her organization with the resilience pattern. In this phase, linguistic terms are used to assess the values of the leading indicators in the 4 cities mentioned above. The experts used the linguistic terms, SD (Strongly Disagree), PD (Partially Disagree), NAND (Neither Agree Nor Disagree), PA (Partially Agree), SA (Strongly Agree). Table 10 shows  Table 7 Agreement matrix between experts E i and E j in the evaluation of the leading indicator 1.5 the attendance degrees and triangular fuzzy numbers for linguistic terms. We used the membership functions proposed by Lee (1996). The procedure used in the Situated Validation phase had 3 steps: 1. Choice of linguistic terms and membership functions; 2. Application of the questionnaire; 3. Calculation of the PCDO resilience level compared with the pattern.
In Fig. 6 we show the graphic presentations of membership functions for the linguistic terms SD (Strongly Disagree), PD (Partially Disagree), NAND (Neither Agree Nor Disagree), PA (Partially Agree) and SA (Strongly Agree).
Using center of area defuzzification method (Yager and Filev 1993) the attendance degree (A) to the resilience pattern was calculated using Eq. 8, where a j is the attendance degree of the leading indicator j of the phase of disaster i in the PCDO.
The evaluation of the organizational resilience in the PCDOs of 4 cities (Recife, Belo Horizonte, Curitiba and Natal) was performed by one professional of each city, the highest ranking professional available in this research. Table 11 shows the attendance degrees of each phase of disaster compared to the ideal resilience pattern of the 4 cities named PCDO1, PCDO2, PCDO3 and PCDO4. We consider satisfactory a compliance degree greater than 0.75, because this value already represent a partially agreement with the ideal resilience pattern. This represents a α-cut at 0.75 of the fuzzy set "Phases of disaster".
As shown in Table 11, PCDO1, 2 and 3 achieve good results in all disaster phases, which is according to the experience and actuation of these PCDOs in the last years. However, PCDO 4 show the lower ranking, below the satisfactory compliance degree in Prevention and Mitigation, Preparedness, and Response (a very low level of 0.56). In Recovery and Restoration PCDO 4 achieved the highest rate, which is influenced by several restoration actions this city has to do after a huge landslide due heavy rain that occurred in 2014.

Discussion
One of the strategic goals of the Hyogo Framework Action, is "the development and strengthening of institutions, mechanisms and capacities at all levels, in particular at the . Later on, the Sendai Framework established as one of its principles that "it is necessary to empower local authorities and local communities to reduce disaster risk, including through resources, incentives and decision-making responsibilities, as appropriate" (UNISDR 2015, p. 13). And also established as one of its priorities that it is necessary "strengthening disaster risk governance to manage disaster risk" (UNISDR 2015, p. 14). According to this view, the indicator system ORIS provide ways for the  Table 9 show the final result, the ideal resilience pattern. It is represented by NI values Overall, the results of the Situated Validation indicated that PCDO1 and PCDO3 were, in general, more resilient institutions than PCDO2 andPCDO4, which is the less resilient institution among those surveyed. The overall results are consistent with the maturity level of each PCDO. Despite all these organizations have already dealt with disasters in these Brazilian cities, they have different levels of maturity. PCDO1, 2 and 3 belong to major Brazilian cities and have similar capacities. They constantly deal with heavy rain situations and their consequences such as floods and landslides. They have operating contingency plan developed with the participation of several stakeholders, such as other municipal departments, fire corps, and police. On the other hand, PCDO 4, at the time of the survey, did not have a contingency plan. The preventive actions of this agency appeared uncoordinated, because there was no planning, and the actions were in fact corrective (Silva et al. 2015). The main types of disasters that PCDO 4 deals with are those originated from heavy rain, i.e., landslides, and flooding. Nowadays PCDO 4 is articulated with the local Federal University to build the contingency plan, as well as update and map disaster risk in the city, and it will use ORIS in order to improve its resilience.
However, it is worth emphasizing that the reliability of results, provided by the Organizational Resilience Indicators System (ORIS), as they are applied here for the Situated Validation process, depends on the managers' knowledge and perceptions on each resilience indicators/metric on the organizations they manage. In a daily basis, ORIS system can be applied by PCDOs in different ways to ensure more reliable results. For example, there are indicators in which the perception evaluated in ORIS can be compared with data recorded (e.g. number of drills). Moreover, the evaluation can be done not only by managers, but by the agents that deal directly with the different phases of disaster. These organizations may choose to use all the indicators of the system, or only those that they consider to be more pertinent with respect to their context of performance.
The main advantages or contributions of ORIS are: • Provide to PCDOs a resilience indicators system addressed specifically to risk and disaster management agencies; • Enable the evaluation of organizational resilience in all phases of the risk and disaster management cycle, following (in an integrated way) the guidelines and principles oriented by the scientific literature, documents of the International Strategy of Disaster Reduction-ISDR and Brazilian laws; • The involvement of people interested (stakeholders) on the development in a participatory way; • The possibility to be used for self-evaluation and self-learning of the organization, enabling the organization to know its weaknesses and potentialities; • It is driven to create strategies, policies, programs and actions, and to take decisions to face risks and disasters, enabling organizations to continuously increase their resilience.
There are also limitations depending on the way it will be applied. The measurement of resilience through the self-assessment may not reflect the full reality of these organizations. It is also known that this type of evaluation is most successfully applied in "mature" organizations that have a conscience and a self-learning practice. In practical terms, the large majority of municipalities in Brazil do not even have protection and civil defense agencies, and in many localities these organizations are still seeking a minimum political and organizational structuring to implement municipal civil protection and defense policies.
In general, the limitations described here do not eliminate the contributions of this work, in both the academic and practical sense. From the theoretical point of view, the study made it possible to fill the literature gap on the lack of resilience indicators for organizations dealing with risk and disaster management. From a practical point of view, it makes available to those organizations a tool for evaluating resilience that has been validated by specialists, agents and managers of protection and civil defense agencies, and applied in some organizations of this nature. Its use helps the PCDOs to acquire knowledge on their deficiencies and resilience potential and capabilities, assisting them in decision-making and establishing strategies to continuously improve their resilience to risk and disaster management in their cities.

Conclusions
In this paper we described a research in which a participatory method for the measurement of organizational resilience in Protection and Civil Defense Organizations was proposed and applied. The method uses indicators and concepts and properties of fuzzy sets theory (FST). Based on literature review a set of indicators was defined. Then an ideal resilience pattern was developed in a participatory way by experts in the domain, using the similarity aggregation method to aggregate fuzzy individual expert opinions in which we consider the difference in the importance of each expert. As shown in the results section, all 34 indicators were considered Much Important (MI) or Important (I) by the professionals, validating the previous indicator selection. Finally, the indicator system was applied in a selfevaluation of the resilience of 4 Brazilian PCDOs. The resilience evaluations compare the perceptions of the PCDOs' managers with the resilience pattern. The managers evaluated in what extent their organizations comply (or not) with the resilience indicators and their metrics as proposed by ORIS. The results are accordingly to maturity level of the organizations indicating that the ORIS is valuable to measure PCDOs resilience.
This study PCDOs showed that the method offers interesting perspectives for the implementation of resilience engineering principles for PCDO's management. In fact this method, using indicators for the different phases of disaster management cycle, provided a basis for identification of potential problems in each phase of disasters. The results of selfevaluations indicated that more mature civil defense organizations, with more support from city governments achieve higher resilience levels. In some sense it is related to the toplevel commitment principle-in this case the government compromises with DRR strategies-which is always present in safety culture and resilience engineering. These problems are also related to lack of flexibility and autonomy managers of Brazilian PCDOs have to use financial resources, especially regarding new staff with professional experience in the domain. Further evaluation on the workers' perceptions may shed light on internal organization processes, e.g. huge differences in workers and managers perceptions may indicate that managers are not compromised with the resilience as they supposed to be.
This method can be applied in any safe-critical organization (e.g. nuclear industry. aviation. pharmaceutical) with adjustments in the leading indicators and its metrics that should be developed according to the characteristics of these organizations.
Future researches can be done in the development of a computational system in order to automate the use of the method in order to assesses an organization's resilience online; in the periodic application of method to estimate how new corrective actions change resilience levels; or in the use of the method in other organizations.