ANSI/BICSI 002, Data Center Design and Implementation Best Practices

Rick Ciordia, PE, RCDD, DCDC, RTPM, CT ETTG, PSC. CALA District Chair - BICSI

BICSI International Standards Program

• Develop standards within all facets of Information & Communications Technology (ICT) infrastructure design and installation • Details: – Over 450 member worldwide – Accredited by ANSI – Develops international open to use/“royalty free” standards and best practices

BICSI Standards Within ICT

IEEE Defines the message and transmission characteristics

ISO/IEC & TIA Defines the transmission media and system specifications

BICSI Defines how to design solutions using transmission media and systems

Standards Development Cycle Industry Needs Technology Trends Designers Suppliers Clients

Idea to Consensus Body

Use

Draft

Consensus Ballot

Review of Results

BICSI Published Standards • Building/Facilities – K12 / Primary Education Facilities (001) – Data Centers (002)** – Healthcare Facilities (004)

• Systems – ESS (005)* – DAS (006)**

• Installation – Cable Installation (NECA/BICSI 568) – ICT Bonding and Grounding (NECA/BICSI 607)*

• Other – BIM for ICT (003)

* Available in Spanish ** Translation into Spanish planned or in progress

Current Activities New Standards • DAS Design – Moving to publication

• ICT Systems for Intelligent Buildings • OSP Installation and Construction – General Principles – Aerial Pathway

Revision Activities • Revision to BICSI 005, ESS – IP Fire Alarm

• Revision to BICSI 001, Educational Facilities – Expanded scope to include secondary, graduate, post-graduate facilities

About ANSI

(American National Standards Institute) • Promotes standards use within United States • Accreditation body – Standards Development – Credentialing Bodies (ISO 17024) – Testing Laboratories (ISO 17025)

• Ensures open and unbiased standards development processes • Does not create standards

Reach of BICSI Standards

ANSI/BICSI 002-2014 Data Center Design and Implementation Best Practices

Data Center Standards ISO/IEC 24764 (2010)

ANSI/TIA-942-A (2012)

Information technology — Generic cabling systems for data centres

Telecommunications Infrastructure Standard for Data Centers

44 pages

118 pages

CENELEC EN 501735:2007 / A2:2012

ANSI/BICSI 002-2014

Information technology — Generic cabling systems - Part 5: Data centres

Data Center Design and Implementation Best Practices

534 pages

48 pages

CENELEC EN 50600 (2012-)

ASHRAE TC9.9 (2012)

Multiple Documents

150 pages

Information technology — Data centre facilities and infrastructures

Thermal Guidelines for Data Processing Environments, 3rd edition

Comparing the Data Center Standards

• Data Center Design – BICSI 002, CENELEC 50600

• Computer Room and Cabling Infrastructure – TIA 942-A

• Cabling Infrastructure – ISO/IEC 24764, CENELEC EN 50173-5

• Environmental Conditions – ASHRAE TC 9.9

• Others – Local / Country – Certifications

Design Standard - CENELEC 50600

EN 50600-1 (2012) General Concepts 32 Pages

EN 50600-2-1 (2014) Building Construction 36 Pages EN 50600-2-2 (2014) Power Distribution 40 Pages EN 50600-2-3 (2014) Environmental Control 32 Pages EN 50600-2-4 Telecommunications Cabling Infrastructure

EN 50600-2-5 Security Systems EN 50600-2-6 Management and Operational Information

BICSI 002, Data Center Design and Implementation Best Practices • More than just cabling • Expands requirements from other standards • Covers additional topics not within other standards

BICSI 002-2014 Breakdown Security 13%

Other 7%

Cabling & Networks 21%

Site & Building 18%

Electrical 29% HVAC & Mechanical 12%

Represents 408 pages of normative content across 13 sections

Content Revision and Expansions

• • • • • • •

Availability Classes Modular Data Centers* Hot/Cold Aisles Mechanical Systems* DCIM* Circuit Maps and DC Power Cabinet Airflow and Cabling Capacity • “Green” / Efficiency*

• Building Structure • Site Hazards • Data Center Services Outsourcing Model* • Bonding & Grounding • Commissioning • Network Security* • Telecommunications Cabling • (And More …)

* Indicates all new content to this edition

BICSI Design Classes and Selection Methodology

Finding the Right Design • Methodologies include: – Loss prevention vs. initial build cost – Total Cost of Ownership (TCO) – Desired availability level

• All provide a starting point for design requirements

BICSI Design Class Determination

• Based on three questions 1. How much downtime per year will be allowed for maintenance? 2. During scheduled operation, what is the maximum allowed downtime? 3. What is downtime’s impact to operations?

• Answers will indicate design class for starting point of requirements

Interaction of Answers Define Operational Requirements

Define Operational Availability Requirements

Identify Required Availability Class

Define Impact of Downtime

BICSI DC Design Classes • Class 0: Single path, and fails to meet one or more criteria of Class 1 • Class 1: Single path • Class 2: Single path with redundant components • Class 3: Concurrently maintainable & operable • Class 4: Fault tolerant

Availability Class Prefixes • Class Fx:

Facility (Electrical & Mechanical)

• Class Cx: Cable Plant • Class Nx:Network Infrastructure • Class Sx: Data Processing and Storage Systems • Class Ax: Applications

Class F1 Electrical Example Utility

Transfer Switchgear

Alternate Power Source

Electrical Distribution

UPS Maintenance Bypass

Static Bypass

Mechanical Switchgear

PDU

Critical Loads

Non-Critical Loads

Mechanical Loads

Class F2 Electrical Example Utility

Transfer Switchgear

Alternate Power Source

N

Alternate Power Source

+1

Electrical Distribution

Static Bypass

UPS

UPS

N

+1

Mechanical Switchgear

Maintenance Bypass

PDU

Critical Loads

Non-Critical Loads

Mechanical Loads

Electrical Class F3 Alternate Power Source

N

Alternate Power Source

+1

Transfer Switchgear

Utility

Electrical Distribution

Electrical Distribution

Static Bypass

UPS

UPS

N

+1

Mechanical Switchgear

Mechanical Switchgear Mechanical Loads

Maintenance Bypass

Mechanical Loads

Output Distribution Switchboard

Alternate Switchboard

Critical Mechanical Switchgear

PDU

Non-Critical Loads

Critical Mechanical Switchgear

Critical Fans/Pumps

Critical Loads

PDU

Non-Critical Loads

Electrical Class F4 Alternate Power Source

N

Transfer Switchgear

Alternate +1 Power Source

Transfer Switchgear

Utility

Static Bypass

UPS

N

+1

Mechanical Switchgear

Maintenance Bypass

Mechanical Switchgear

Mechanical Loads

UPS

N

+1

Static Bypass

Output Distribution Switchboard Critical Mechanical Switchgear

Non-Critical Loads

UPS

Maintenance Bypass

Output Distribution Switchboard

PDU

Alternate +1 Power Source

Electrical Distribution

Electrical Distribution

UPS

Alternate N Power Source

Critical Mechanical Switchgear

Critical Fans/Pumps

Critical Loads

PDU Non-Critical Loads

Class F4 Electrical “Catcher” Utility

Utility

High Voltage Receiving 2

High Voltage Receiving 3

High to Medium Voltage Transformer 2

Switchgear

High Voltage Receiving 1

High to Medium Voltage Transformer 1

Alternate Power Source

+ Alternate Power Source 1

N

Medium Voltage Distribution 1

Medium Voltage Distribution 2

Medium Voltage Electrical Distribution 1

+1

N

Main

To Mechanical Systems See Section 10 for configuration of mechanical systems

PDU-A

Source 2

Normal UPS

Stand-by

Overlap Switch (Manual)

Non-Critical Loads

Standby UPS

Static Bypass Input

N

Source 2

Source 1

Normal UPS

Source 1

Source 2

+1

Source 2

Static Bypass Input

Main

Standby UPS

Source 1

Source 1

Source 2

Static Bypass Input

N

Source 1

Source 2

Source 1

Normal UPS

Medium Voltage Electrical Distribution 2

Standby UPS +1

Stand-by

Overlap Switch (Manual) PDU-B

Critical Loads

Non-Critical Loads

Telecommunication Classes Service Provider

Class C1 C3 C2 C4

Service Provider

Service Provider

Service Provider

Maintenance Hole

Maintenance Hole

Entrance Room

Entrance Room

Offices, Operations Center, Support Rooms

Computer Room

Telecomunicaciones Room

Main Dist Area

Horiz Dist Area

Main Dist Area

Horiz Dist Area

Zone Dist Zone Dist Area Area

Horiz Dist Area

Horiz Dist Area

Zone Dist Area

Equip Dist Area

Equip Dist Area

Equip Dist Area

Equip Dist Area

Equip Dist Area

Equip Dist Area

Equip Dist Area

Equip Dist Area

ITE

ITE

ITE

ITE

ITE

ITE

ITE

ITE

Only required for large computer rooms due to port counts or distance limitations

Air-Cooled Condenser (Dry-Cooler) Chiller

Air-Cooled Condensers (Dry-Coolers)

• Redundant critical components • All power feeds from common upstream distribution • Only redundant components able to be maintained under load

Chiller

Chiller CRAH CRAH

Mechanical Class F2 F1 • No redundancy present • All power feeds from common upstream distribution • No ability to be maintained under load

Air-Cooled Condensers (Dry-Coolers) Air-Cooled Condensers (Dry-Coolers)

Chiller Chiller

Mechanical F3 Class F4 • Redundancy in all areas • Equipment and controls with redundant systems fed from different electrical distribution • Maintainable when actions do not reduce cooling capacity below "N+1"

Chiller Chiller Chiller

CRAH CRAH

• Redundant equipment and piping for maintenance • Power feed so that cooling capacity does not drop below "N" when maintaining mechanical or upstream electrical distribution • Maintainable when actions do not decrease cooling capacity below "N"

Class N0/N1 and N4 Network

Class S4 System and A4 Application

Availability and Multi-Site Data Centers

• Prior to virtualization, subclasses aligned through data center • Today, a single data center may not have alignment • Availability class methodology can in discussions about using multiple data centers to achieve availability target

Example: Class 3 Availability Using Three Class 2 Data Centers

Modular Data Center Design • Modular may describe: – How a data center is segmented – Method of construction/implementation

• Both can assist in allowing for different availability needs for one site

Modular/Containerized Data Center Space Adjacencies

F4 F3 F2

Modular Design Planning Aides

BICSI-002, 2014 Supplement

Are BICSI & Uptime Similar? • ANSI/BICSI 002-2014 This standard provides a reference of common terminology and design practice … a framework for the process to determine facility criticality and to develop optimum design & implementation solutions

• Uptime Tiers “Only data center benchmarking system developed by and for data center owners Performance-based on fundamental concepts – Not a checklist, design menu, or cookbook” Source: Uptime Institute: Tier Classification System & Operational Sustainability presented by Dana Smith, Director of Development, Uptime Institute at BICSI Andino 2012

Standards REGULATIO NS TIA

IEC

ISO

BICSI

ASHRAE

CENELEC

ieee NFPA

Colombia o NTC-6064 - Tecnología de la Información. Cableado Genérico para instalaciones de clientes.  Especifica el cableado genérico para uso dentro de instalaciones que pueden comprender una o varias edificaciones en un predio (campus).  Comprende el cableado balanceado y el cableado de fibra óptica.  Referencia: ISO/IEC 11801

Colombia (Cont.) o RITEL - Reglamento Técnico para Redes Internas de Telecomunicaciones  Establece las medidas relacionadas con el diseño/ construcción y puesta en servicio de las redes internas de telecomunicaciones.  Referencia: ISO/IEC 11801

Costa Rica o Colegio Federado de Ingenieros y

Arquitectos  Todos los planos de instalaciones eléctricas y de telecomunicaciones, de voz y datos deberán cumplir donde corresponda con:  El NEC Código Eléctrico Nacional (NFPA 70) en su última versión en español.  NFPA 70 E “Norma para la seguridad eléctrica de los empleados en los lugares de trabajo”, en su última versión en español.

 Las últimas revisiones y adenda aprobados de las normas ANSI/EIA/TIA 568, 569, 570, 606, 607.

Mexico o NMX-I-108-NYCE-2006, Telecomunicaciones – Cableado – Cableado estructurado – Puesta a tierra en sistemas de telecomunicaciones o NMX-I-132-NYCE-2006, Telecomunicaciones – Cableado – Cableado estructurado – Especificaciones de las pruebas de cableado balanceado – Parte 01: Cableado instalado o NMX-I-154-NYCE-2008, Telecomunicaciones – Cableado – Cableado estructurado – Cableado genérico residencial o NMX-I-248-NYCE-2008, Telecomunicaciones – Cableado – Cableado estructurado – Cableado de Telecomunicaciones para edificios comerciales – Especificaciones y métodos de prueba o NMX-I-279-NYCE-2009, Telecomunicaciones - Cableado Cableado estructurado - Canalizaciones y espacios para cableado de telecomunicaciones en edificios comerciales

Mexico (Cont.) o NMX-I-14763-1-NYCE-2010, Telecomunicaciones – Cableado – Cableado estructurado – Implementación y operación de cableado en edificios comerciales – Parte 1: Administración o NMX-I-24764-NYCE-2013, Tecnología de la información – Sistema de cableado genérico para centros de datos o NMX-J-364/I-ANCE-2011, Instalaciones eléctricas – Parte 1: Principios fundamentales, planeación de características generales, definiciones o NMX-J-C-I-489-ANCE-ONNCCE-NYCE-2014, Centros de datos de alto desempeño sustentable y energético – Requisitos y métodos de comprobación o NOM-001-SEDE-2012, Instalaciones Eléctricas (Utilización) ANCE       - Asociación Nacional de Normalización y Certificación del Sector Eléctrico NMX       - Norma Mexicana NOM       - Norma Oficial Mexicana NYCE       - Normalización y Certificación Electrónica ONNCCE - Organismo Nacional de Normalización y Certificación de la Construcción y Edificación

Standards In Use Country

NEC

TIA

Chile Colombia Costa Rica El Salvador Guatemal a Honduras Mexico

RL

X

RLN

R

RN

Panama Peru Puerto Rico Republica

R

ISO

BICSI

X

X

R

X

X

X

RN

X

X

RN

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

Uptim Nation ICREA Other e al

X

DC Standards Organizatio n Standard

BICSI

TIA

ISO

BICSI ISO TIA 942A 002 24764 (2012) (2014) (2010)

US Worldwi Worldwi Recognition /Canada de de (Int’l)

Electrical Ar Mechanic ea al

s Telecom Co Environm ve ent re Site d

Yes

CENELEC ASHRAE EN 501735 / EN 50600

ICREAASHRAE Std-131TC9.9 2013

EU (Int’l)

Int’l US/Cana (21 da (Int’l) Countrie s)

Yes

Yes

Yes Yes

Yes

Yes

Yes

Yes

Yes

ICREA

Yes

Draft Yes

Yes Yes

Yes

Planned Interoperability Government Regulations Electrical Code Building Code

Data Center

(BICSI 0022014)

Fire Code Electrical Standards Cabling Standards

Cabling Spaces Standards Environmental Conditions Building System Standards Security Standards

Interoperability Demonstrated

(BICSI 002-

2014) Colombia Canada United Rica Costa States

Industry Canada, Labour Government Government FCC, OSHA Regulations Program CSA Electrical NFPA C22.1 70 (NEC) Code (CEC) International Building National Building CodeCode of Building Government Code Canada  (NBC) (IBC) National Fire Code of Government, NFPA Fire 1,Code NFPA NFPA 72 Canada (NFC)

Electrical IEEE, IEEE, CSA, NFPA, Standards NFPA IEEE NECA Cabling RITEL, TIAStandards 568 ISO Cabling Spaces TIA ISO 942Standards Environmental ASHRAE Conditions Building TIA 862, System ISO AHSRAE Standards 13 NFPA, RCMP, Security Government, Government ASIS, Government, Standards Government ISOISO

Why BICSI 002? “If you fail to plan, you are planning to fail!” “¡Si fallas en planificar, estas planificando fallar!” ●

- Benjamin Franklin

Thank You! Rick Ciordia, PE, RCDD, DCDC, RTPM, CT ETTG, PSC. [email protected]