Can Your Electrical Infrastructure Weather a Natural Disaster? White Paper Document 1910DB1611

by C  had Kennedy

Executive summary Contingency planning for continued business operations is a multi-faceted risk management function. While natural disasters cannot be avoided, their impact may be somewhat lessened if businesses are better prepared. This paper identifies pre-planning exercises companies can complete to help restore electrical distribution and control equipment efficiently and safely.

Can&your electricaloverview infrastructure weather a natural disaster Oil Gas solution

Table of contents

2

Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Natural Disaster Definition & Statistics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Three Steps to Electrical Disaster Recovery Planning

Step 1: Knowledge of the Electrical System. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4



Step 2: Develop (or Update) an Electrical Safe Work Practices Policy. . . . . . . . . . . . . . . . . . . . . . . 5



Step 3: Electrical Emergency Action Plan. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

Developing an Electrical Emergency Action Plan. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

Schneider Electric white paper

Can&your electricaloverview infrastructure weather a natural disaster Oil Gas solution

Introduction

3

Businesses are under increasing pressure to maximize profits and minimize downtime. Therefore, it is extremely important to have a contingency plan for continued operations in the event of a natural disaster or emergency. Actions taken during the first 24 to 48 hours of a disaster are critical in determining whether or not a business fully recovers. As many as 50% of businesses close down following a disaster according to the latest research1. While natural disasters cannot be prevented, having a detailed emergency recovery plan can limit the financial and personal havoc they can cause. A good starting point is to address the following key areas:

• • • • • •

Ensure electrical equipment is properly maintained Identify the electrical equipment that is critical to operations Be aware of the most current natural disaster recovery codes and standards Know the effects of water damage to electrical equipment Develop a safety plan that incorporates emergency procedures Develop an electrical emergency action plan

The National Fire Protection Agency (NFPA) and the Occupational Safety and Health Association (OSHA) provide guidelines to develop disaster recovery, emergency response, and safety plans. This paper will incorporate those guidelines to help in the creation of both short-term and long-term restoration plans. The number one priority for both plans is to safely restore power.

Natural Disaster Definition and Statistics

Natural disasters can be broken down into four categories, as shown below. Each type of disaster has a unique severity level. Category Type of Disaster Geophysical Earthquakes, tsunami, volcanic eruption Meteorological Storm, lightning Hydrologic Flood, mass movement Climatologic Extreme temperature drought, forest fire Since the early 1990s, total economic losses from natural disasters in the United States have averaged tens of billions of dollars per year2. These disasters cause death and injury, property damage, business interruptions or downtime and loss of revenue. Natural disasters can damage electrical distribution equipment in multiple ways (fire, vibration, water, etc).The most frequently-occurring events in the U.S. involve water in the form of hurricanes and storms, with subsequent flooding. Since water and electricity do not mix, restoring power to water-damaged equipment can be a dangerous undertaking. As an example, in May, 2010, Nashville, Tennessee experienced an unprecedented 500-year flood killing more than 30 people and devastating property with over $2 billion of damage. It took two years for some companies to re-open for business; some never did. There is increasing evidence that climate change is leading to more frequent and severe weather events, which points to an increased natural catastrophe risk. Figure 1 illustrates U.S. natural disasters for the past 35 years. In some cases, the increasing numbers can be attributed to better technology to identify natural disasters. However, climate changes are considered to have a significant impact on natural disaster occurrences.

Schneider Electric white paper

Can&your electricaloverview infrastructure weather a natural disaster Oil Gas solution

Figure 1: Number of Natural Disaster Events in the United States (1980 - 2015) Relevant Codes & Standards NFPA 1600 is the primary document and overarching standard on disaster recovery, emergency management, and business continuity. For workplace safety and planning, OSHA references, and cites to, NFPA 70E, Standard for Electrical Safety in the Workplace. Chapter 32 of NFPA 70B, Recommended Practice for Electrical Equipment Maintenance, is entitled, “Electrical Disaster Recovery”. The National Electrical Manufacturer’s Association (NEMA) has published “Evaluating Water-Damaged Electrical Equipment” as well as a guide for “Evaluating Fire- and Heat- Damaged Electrical Equipment.

Accounted events have caused at least 1 fatality and/or produced normalized losses > $3.0M. Source: 2016 Munich Re. NatCatSERVICE

For additional information on your location, the Federal Emergency Management Agency (FEMA) publishes disaster and emergency declaration by year and state.

Three Steps to Electrical Disaster Recovery Planning

Step 1: Knowledge of the Electrical System As part of an electrical disaster recovery plan, businesses should: • Have a current single-line drawing of their electrical distribution system • Identify which electrical equipment is critical to the electrical infrastructure • Understand which equipment must be replaced and that which can be reconditioned, as shown in Figure 2 Electrical equipment exposed to water can be extremely dangerous if re-energized without proper reconditioning or replacement. Assessing the damage goes beyond visual water indications. Moisture and weather exposure can affect the equipment’s integrity. Water can be contaminated with sewage, chemicals, salt, debris, and other substances. Foreign debris could possibly remain inside the electrical enclosure. These contaminants may cause loss of dielectric spacing and can be a hazard upon re-energization. Whether the equipment is to be replaced or reconditioned, all services should be performed by qualified personnel familiar with the equipment’s operation and construction. It is

Schneider Electric white paper

4

Can&your electricaloverview infrastructure weather a natural disaster Oil Gas solution

5

important to note that the ability to recondition equipment will vary; it may include the repair or replacement of internal components. Reconditioned equipment should be tested per ANSI / IEEE standards prior to re-energization. Figure 2: NEMA Recommendations on Evaluating Water-Damaged Electrical Equipment

It is important that restoration activities are performed by qualified personnel as defined by OSHA and NFPA. Hiring outside contractors to assist or perform electrical work does not relieve the facility owner (host employer) from being liable for safe work practices. NFPA 70E states that known hazards must be communicated to the contracted worker(s).

Step 2: Develop (or Update) an Electrical Safe Work Practices Policy An Electrical Safe Work Practices (ESWP) policy is a written document created by the employer that covers all areas of the company’s electrical safety practices. The ESWP policy is not a one-size-fits-all policy to cover a company with multiple locations. For example, voltage, energy level, circuit conditions and hazard levels are different in each facility. The policy should identify detailed information specific to that location. This may include the location of the safety policy for employee (or contractor) access, who is the onsite authority having jurisdiction (AHJ) for decisions, etc. Developing and auditing an ESWP policy is critical to business continuity and disaster recovery. Since creating a comprehensive program may seem overwhelming, it may be helpful to use the following guidelines to get started. 1) Facility - Includes company policies and systems regarding: • Equipment maintenance • Testing • Clearance requirements

• Tools • Repairs • Safe working conditions

2) Personnel - Focuses on actual work practices and addresses: • Qualified and unqualified personnel • Proper care and use of PPE • Job preparedness • Training and continuing education

Schneider Electric white paper

Can&your electricaloverview infrastructure weather a natural disaster Oil Gas solution

6

3) Procedures - References on the job procedures and includes (but not limited to): • • • •

Performing energized work Lock-out / Tag-out Arc flash hazard analysis Reporting safety concerns

• • • •

De-energizing and re-energizing Job planning Equipment labeling Recordkeeping

The employer must verify on a regular basis that each worker is complying with the safetyrelated work practices required by NFPA 70E. Managers and supervisors must embed ESWP as part of their DNA, attend electrical safety training classes, and ensure all employees practice safe work practices everyday. Finally, should a natural disaster strike, employers and employees should be very familiar with emergency procedures and work practices.

Step 3: Develop an Electrical Emergency Action Plan The purpose of the electrical emergency action plan (EEAP) is to understand the electrical assets, critical operational infrastructure, risks, and short- and long-term power restoration execution plans. Some of the benefits include: • Reduces time to restore short-term and long-term power most quickly and safely • Reduces uncertainties when a disaster occurs • Increases understanding of electrical assets, available emergencies services, and replacement market availability • Being able to know the immediate financial implications should a disaster occur NFPA 1600 provides a framework for a site wide disaster recovery program regardless of the risk type. Although it does not include specific guidelines for any particular system or hazard, the provisions within help users manage potential disasters by having a common set of guidelines for managing all disaster recovery plans for a facility. From an electrical system perspective, NFPA 70B should be a first step in the process for the electrical system recovery plan. NFPA 70B provides guidance on creating a maintenance program for the electrical system in a facility which will contain information and data essential to the electrical system recovery plan. NFPA 70B - Chapter 32, “Electrical Disaster Recovery”, helps users determine potential catastrophic events and categorize potential impacts to the system. For now, electrical disaster recovery planning is based on experiences of the individual or team who has electrical responsibilities. One purpose of this whitepaper is to provide guidelines on developing an electrical emergency action plan and examples how it saved or could have saved facilities from excessive downtime. Two real industry examples follow. Downtime, loss of revenue and headaches could have been reduced from a robust EEAP program. Note: Facility names will remain anonymous.

Schneider Electric white paper

Can&your electricaloverview infrastructure weather a natural disaster Oil Gas solution

7

Example 1: Transformer Disaster Recovery – Industrial Facility Background Transformers are considered by corporate risk managers and insurers as the most critical equipment in facilities because of the large quantity of oil in contact with high voltage elements. Service entrance transformers are essential to operations. Many facilities would experience downtime or lack of productivity if the service entrance transformer is being serviced or a disaster hindered its performance. Furthermore, transformers can be customdesigned for utilities and facilities which makes finding a replacement or spare parts complicated, especially on an emergency basis. What Happened? A manufacturing facility that produced over $1 million per day in product was struck by lightning. The strike caused extensive electrical equipment damage, which included catching a 25 MVA transformer on fire. The fire was not contained in time and spread to an adjacent transformer. No standby generators were available. (Even so, standby generation at most industrial facilities would only be able to power vital processes, data rooms, and key emergency services.) Without standby generators and spare transformers, the facility experienced a complete blackout. To get back online as soon as possible, the team assessed the damages, identified an available emergency transformer, coordinated logistics and financial approvals, acquired the additional electrical equipment needed, and hired an experienced electrical disaster recovery team. Lessons Learned 1. An EEAP should provide details to understand immediate actions, the sequence of engagement with recovery teams, available immediate spares at the facility, primary, secondary market and the financial impact to communicate to senior management. 2. To restore power as quickly as possible, identify critical electrical assets and the nearest available spares, rates, and terms for both short- and long-term requirements. For example, temporary 25MVA transformers are hard to find and once located, lease for approximately $35,000 per month (excluding delivery charges). In addition, lead times on new replacement equipment can take months. 3. Outside resources may be required to help manage power restoration, should another disaster occur. Partner with organizations that have the qualifications, resources and experience to handle the job. Items such as pre-negotiated commercials terms and conditions, defined scope of work, equipment pricing, lead times, availability, and service agreements streamline the power restoration process.

Example 2: Generator Disaster Recovery - Commercial Building with Businesses and Data Center Background On May 1-2, 2010, Nashville, Tennessee received a record 13 inches of rain, which caused a historical 500-year flood. Thirty people were killed and damages exceeded $2 billion. Also known as “Music City USA”, Nashville is a key tourist area for music lovers around the world. Due to its proximity to the Cumberland River, downtown Nashville was hit hard and

Schneider Electric white paper

Can&your electricaloverview infrastructure weather a natural disaster Oil Gas solution

8

floodwaters damaged many commercial buildings. This example focuses on a class A commercial high-rise in downtown Nashville that houses multiple businesses and a data center. What Happened? The following timeline of events depicts how the flooding affected the commercial high-rise building. Times shown serve as general guidelines. Sunday, May 2 9:00 pm

Municipal power was no longer able to serve power to downtown Nashville. The commercial building automaticly switched to emergency power and only served vital data center operations. Teams were deployed to the site to closely monitor data center and weather conditions.

Monday, May 3 12:00 am

Metropolitan police ordered a mandatory evacuation in downtown, which left data center operations and security vulnerable.

4:30 am

Selected personnel were allowed back into downtown to perform key operations, which included refueling generators. Although rain had stopped, the Cumberland River was steadily rising.

10:00 am

A decision was made to start soft shutdown actions, but it was too late. Generators flooded, which began a 31-hour data center outage. Through a key relationship, a temporary generator was located 20 miles south the city.

Tuesday, May 4 4:30 pm

The data center was brought back on-line.

Wednesday, May 5

Still with no lights, air conditioning, or restroom facilities, a 24-hour staff rotation was set up to monitor the data center and the condition of the temporary generator. The water level was still too high to assess damage to the building’s generator.

Thursday, May 6

The building’s generator was drained and the damage was assessed. Spare parts were not immediately available. Five days later the generator was repaired and power switched back to the building generator. However, the temporary generator was retained as a back-up for a few additional weeks.

Thursday, May 13

Schneider Electric white paper

Nashville Electric Services (NES) restored power to the building and switched from generator to municipal power.

Can electrical infrastructure Optimize power system reliabilityweather a natural disaster Oil &your Gas solution overview

Lessons Learned Figure 3 summarizes three lessons learned from the aftermath of this historical flood. 1. Defined Priorities: The EEAP plan must have a clear definition of what constitutes an emergency and when to execute the EEAP. In this case, only three hours were available before the mandatory evacuation order was issued. Prioritizing critical functions is essential to efficiently and safely restoring power. 2. Emergency Service Contracts: Searching for temporary equipment after a disaster occurs slows down the power restoration process and can be very expensive. The EEAP should include emergency service contracts to guarantee disaster recovery team response time, critical equipment pricing, lead times, and a deployment strategy with details on setting up and operating a command center to meet an organization’s needs. 3. Critical Spare Parts: Having a current single-line electrical diagram of the power distribution system is crucial to efficiently restoring power. Use the drawing to pinpoint the electrical equipment critical to business operations. Identify the equipment’s critical spares parts availability, pricing and lead times for custom-made parts. Figure 3: Timeline of Events

In some cases, EEAP programs should be signed off by customers and facility management companies. Why? Consider the following example: A customer leases multiple floors (one floor being a data center) from a large building management company. The customer’s IT manager may have no control over the equipment feeding power to the office space and data center. In this type of situation, clear roles and responsibilities, work scopes, and agreed upon emergency actions are vital to a long-term building owner and customer relationship.

Schneider Electric white paper

9

Can&your electricaloverview infrastructure weather a natural disaster Oil Gas solution

Developing an Electrical Emergency Action Plan

10

The development of an EEAP may seem overwhelming. Below is a step-by-step guide created by Schneider Electric Services to help customers be prepared in the event of an emergency. 1. Define the Criteria of an Emergency Lighting strikes on service entrance transformers and flooding of backup generators clearly indicate operational ‘states of emergency’. Are nuisance tripping on circuit breakers or power quality issues considered emergencies? When productivity is impacted, it can be confusing as to when a state of emergency should be declared. The EEAP should clearly define what constitutes an electrical emergency. 2. Identify Electrical Equipment that is Critical to Business Operations On the single-line diagram, trace the power from the incoming utility source to every piece of electrical equipment feeding critical business operations. Perform an analysis on each of these critical assets to include availability in the market, lead times, and a plan of action if and when the equipment is no longer functional. For most critical assets, it is necessary to understand cost of temporary rentals, logistics, contacts, and testing requirements for both temporary and permanent power restoration. For example, a special transformer’s spare parts may not be available in the primary market, i.e., the local electric municipality. Therefore, it is a good practice to determine available suppliers, acquire budgetary pricing, and lead times on the secondary market. 3. Selection of Outside Vendors and Pre-Negotiated Commercial Terms & Conditions In the aftermath of a disaster, it may be too late to negotiate pricing or lead times for the required resources to restore electrical power. Without pre-negotiated emergency service contracts, companies may suffer from overpricing and insufficient support. In addition to pre-negotiated normal and emergency rates, due diligence should include estimated response time and procedures for large-scale project coordination (see Step 4). The selected vendor(s) should have the depth and experience to handle major disasters. 4. Define Internal and External Responsibilities The EEAP should clearly define ‘who has responsibility for what’ in restoring power to the facility. They also have details on the customer’s responsibility such as providing electrical single-line diagrams, energization procedures, and coordination and communication activities between multiple vendors through a central contact. Third-party vendors should also provide a clear procedure on how they will approach an emergency at a facility that includes assessing damage, mobilizing resources, appointing a project manager, and establishing a command center. All parties involved should fully understand the safety plan that is put in place. 5. Define the Equipment and Service Scope This section will further define the equipment and associated work scope. For example, an electrical distribution service company can define the equipment scope to be from the utility service entrance (13.8 kV) to low voltage switchgear (480 V). The associated work scopes include equipment installation and commissioning in both temporary and permanent scenarios. 6. Emergency Contract Terms Emergency contracts should be written for a specific time period, with an expiration date. It is also recommended to add expiration dates to the EEAP plan for self-auditing purposes. A suggested best practice is to audit the EEAP when the ESWP is being audited which is every three years according to NFPA 70E.

Schneider Electric white paper

Can&your electricaloverview infrastructure weather a natural disaster Oil Gas solution

11

7. Contact Information This section of the EEAP should include the latest contact information for anyone (internal or external) who has a defined responsibility in restoring power due to an emergency.

Conclusion

When a natural disaster strikes, its impact on individuals, communities and businesses can be devastating. Restoring electrical power is a crucial part of the recovery process. Regardless of the industry or facility type, having a detailed Electrical Safe Work Practices (ESWP) policy and an Electrical Emergency Action Plan (EEAP) can help recovery efforts. Multiple standards exist from OSHA, NFPA and NEMA to serve as guidelines for businesses to help them understand and develop a contingency plan in the event of an emergency or natural disaster.

References 1. ServPro - http://www.servpro.com/ready 2. Lloyd’s Insurance Report, ‘Managing the escalating risks of natural catastrophes in the United States (October 2011) 2. Munich Re. NatCatSERVICE (2016) 3. NFPA 70B: Recommended Practice for Electrical Equipment Maintenance

Schneider Electric white paper