Substation Communications Design - Legacy to IEC 61850

Substation Communications Design Legacy to IEC 61850 Part 3/3: Reliability & Security Tim Wallaert Chris Jenkins Agenda • Substation to the Contro...
Author: Noel Dixon
9 downloads 0 Views 4MB Size
Substation Communications Design Legacy to IEC 61850 Part 3/3:

Reliability & Security Tim Wallaert Chris Jenkins

Agenda •

Substation to the Control Room Communications − − − −



?

How to Build a Redundant Network − − − − − −



Legacy networks Networking today CIP Hardened equipment

RSTP MRP Routing Router Cellular PRP/HSR

How to Lock it Down −

Firewalls



VPN Port Security Authentication

− −

© 2014 Belden Inc. | belden.com | @BeldenInc

2

Legacy Utility Networks 1200 baud

SCADA Master

Administration

Modem Bank Dial-in

Leased Lines

IED

RTU

IED

IED

RTU

IED

IED

Distributed Substations

© 2014 Belden Inc. | belden.com | @BeldenInc

3

RTU

IED

Today’s Digital Networks RTU Management & Provisioning

Master Site

SCADA/ EMS

PBX GW

HMI

Switch

Term Server

Video Monitoring

Video Storage

PoE

HMI

Wide Area Network

PoE

VOIP Enet IED

SerialSerial Serial IED IED IED

VOIP HMI

PoE Enet IED

SerialSerial Serial IED IED IED

Sub A Sub C

VOIP Enet IED

SerialSerial Serial IED IED IED

Sub B © 2014 Belden Inc. | belden.com | @BeldenInc

4

New networks have to be compliant… What does “compliance” mean? • FERC/NERC-CIP − Federal Energy Regulatory Commission − North American Electric Reliability Corporation • http://www.nerc.com/pa/Stand/Pages/CIPStandards.aspx • Critical Infrastructure Protection (CIP) • Version 1 enforced in 2008 • Currently enforcing Version 3 • Version 4 has been approved, April 2014 deadline • Still to be approved Version 5 is being pushed to replace Version 4 • Version 5 has MANY changes including: • Encryption • Multi-Factor Authentication •

© 2014 Belden Inc. | belden.com | @BeldenInc

5

Federal Regulation • Critical Infrastructure Protection (CIP) is a group of standards enforced by NERC • NERC does not certify equipment • There is no such thing as a “CIP certified router” • Belden provides equipment that enables customers to design and implement networks that are CIP compliant • Constantly monitoring NERC for changes in standards and listening to feedback from our customers

© 2014 Belden Inc. | belden.com | @BeldenInc

6

“I need to upgrade my network and stay compliant, but how?”

© 2014 Belden Inc. | belden.com | @BeldenInc

7

Turn to a Trusted Leader

© 2014 Belden Inc. | belden.com | @BeldenInc

8

Solutions Portfolio

© 2014 Belden Inc. | belden.com | @BeldenInc

9

Important Specs to Consider •

Experienced, Reliable Vendor



Standards Based Equipment



Environment Extended Temp ranges  Noise Immunity  IEEE1613  IEEE61850-3 

© 2014 Belden Inc. | belden.com | @BeldenInc

10

Important Specs to Consider IEEE 1613

IEC 61850-3

Contact

8kV

Air

15kV

ESD

Radiated RF

35V/M I/O ports

Fast Transient

Oscillatory

Dielectric Strength Operating Temperature

4kV

Power ports (HV and LV)

4kV

I/O ports

2.5kV

Power ports (HV and LV)

2.5kV

HV power ports

3kV

LV power ports

2kV -40 to +85+ C

ESD (61000-4-2)

Contact

8kV

Air

15kV

Radiated RF (61000-4-3) Fast Transient (61000-4-4)

Surge (61000-4-5)

Conducted RF (61000-4-6)

20V/M I/O ports

4kV

Power ports (HV and LV)

4kV

I/O ports

4kV

Power ports (HV and LV)

4kV

I/O ports

10V

Power ports (HV and LV)

10V

Magnetic Field (61000-4-8) Voltage Dips & Interrupts (61000-4-11)

© 2014 Belden Inc. | belden.com | @BeldenInc

30A/m

HV power ports

11

Pass

Important Specs to Consider •

Redundant Power options − Low

and High voltage options − Dual power supply features − Hot Swappable

PS1

PS2

© 2014 Belden Inc. | belden.com | @BeldenInc

12

Important Specs to Consider

• Redundancy features - Serial Port - RSTP - MRP - RIP, OSPF, BGP - VRRP - Cellular redundancy - PRP/HSR

• Security - Encryption - Authentication - Firewalls - Detection © 2014 Belden Inc. | belden.com | @BeldenInc

13

Redundancy Features

© 2014 Belden Inc. | belden.com | @BeldenInc

14

Serial Redundancy RTU Management & Provisioning

Master Site

SCADA/ EMS

PBX GW

HMI

Switch

Term Server

Video Monitoring

Video Storage

PoE

HMI

Wide Area Network

PoE

VOIP Enet IED

SerialSerial Serial IED IED IED

VOIP HMI

PoE Enet IED

Sub A

Sub C

VOIP Enet IED

SerialSerial Serial IED IED IED

SerialSerial Serial IED IED IED

Sub B © 2014 Belden Inc. | belden.com | @BeldenInc

15

Serial Redundancy Poll

Master Site

SCADA/ EMS

SCADA/ EMS

HMI

PoE

HMI

Wide Area Network

VOIP Enet IED

Term Server

Term Server

Video Monitoring

HMI

PoE

SerialSerial Serial IED IED IED

VOIP

Sub B

RESP RESP SerialSerial

PoE

RESP

© 2014 Belden Inc. | belden.com | @BeldenInc

SerialSerial Serial IED IED IED

Sub C

Serial IED IED IED

Enet IED

Video Monitoring

VOIP Enet IED

Sub A

Backup Site

16

Ethernet Redundancy Protocols Protocol Parallel Redundancy Protocol (PRP)

Current Standard

Typical Re-Config

Topology

Available since

IEC 62439-3:2012-07 0mS

Any topology/mesh

2010

IEC 62439-3:2012-07 0mS

Pure Ring Only

2010

IEEE 802.1D-2004

5-20mS per switch

Any topology/mesh

2004

Media Redundancy Protocol (MRP) IEC 62439-2:2010

200mS worst case, 50 switches max

Pure Ring Only

1998/2007

Routing Information Protocol (RIP & RIP 2) RFC 1723

~30sec

small networks

1988/1994

Open Shortest Path First (OSPF)

RFC 2328

seamless

Any topology/mesh

1987/1998

Border Gateway Protocol (BGP)

RFC 4271

seamless

Any topology/mesh

1989/2006

High-Availability Seamless Redundancy (HSR) Rapid Spanning Tree Protocol (RSTP)

© 2014 Belden Inc. | belden.com | @BeldenInc

17

RSTP Redundancy

© 2014 Belden Inc. | belden.com | @BeldenInc

18

Rapid Spanning Tree Protocol (RSTP) Bridging RTU Management & Provisioning

Master Site

SCADA/ EMS

Video Monitoring

PBX GW

Switch

Term Server

Video Storage

BPDU

HMI

HMI

PoE

HMI

PoE

VOIP Enet IED

SerialSerial Serial IED IED IED

Sub A

VOIP

IED

BPDU Enet IED

SerialSerial Serial IED IED IED

SerialSerial Serial IED IED IED

Sub C

Sub B © 2014 Belden Inc. | belden.com | @BeldenInc

PoE

19

Rapid Spanning Tree Protocol (RSTP) Bridging RTU Management & Provisioning

Master Site

SCADA/ EMS

Video Monitoring

DATA DATA DATA

PBX GW

Switch

Term Server

Video Storage

BPDU BPDU HMI

PoE

~5 -15ms recovery

HMI

PoE

VOIP Enet IED

SerialSerial Serial IED IED IED

Sub A

HMI

VOIP

IED

BPDU BPDU Enet IED

SerialSerial Serial IED IED IED

SerialSerial Serial IED IED IED

Sub C

Sub B © 2014 Belden Inc. | belden.com | @BeldenInc

PoE

20

MRP Redundancy

© 2014 Belden Inc. | belden.com | @BeldenInc

21

Media Redundancy Protocol (MRP) RTU Management & Provisioning

Master Site

SCADA/ EMS

PBX GW

Video Storage

HELLO HELLO HELLO

Ring Switch HMI

Switch

Term Server

Video Monitoring

Ring Manager

PoE

*10mS recovery VOIP

HMI

Ring Switch

HMI

PoE

PoE

VOIP VOIP

IED

IED

SerialSerial Serial IED IED IED

Sub C

IED

SerialSerial Serial IED IED IED

Term Server IED

IED

Sub C

Sub B * 50 switches per ring max © 2014 Belden Inc. | belden.com | @BeldenInc

22

Routing Protocol Redundancy

© 2014 Belden Inc. | belden.com | @BeldenInc

23

Routing Protocols RIP, OSPF, BGP Master Site

SCADA/ EMS

PBX GW

Switch

Term Server

Video Monitoring

HMI

RTU Management & Provisioning

Video Storage

PoE

HMI

Wide Area Network

PoE

VOIP Enet IED

SerialSerial Serial IED IED IED

HMI

PoE

VOIP Enet IED

Sub A

SerialSerial Serial IED IED IED

Sub C Enet IED

SerialSerial Serial IED IED IED

Sub B © 2014 Belden Inc. | belden.com | @BeldenInc

24

Routing Protocols RIP, OSPF, BGP SCADA/ EMS

PBX GW

Master

HMI

Substation C is over here Video

Term Server

Video Monitoring

Substation A is over here

RTU Mgmt

Storage

PoE PoE

HMI

HMI

PoE PoE

VOIP Enet IED

SerialSerial Serial IED IED IED

HMI

PoE

VOIP Enet IED

Sub A

SerialSerial Serial IED IED IED

Sub C Enet IED

SerialSerial B is Substation Serial IED IED IED over here

Sub B © 2014 Belden Inc. | belden.com | @BeldenInc

25

Routing Protocols RIP, OSPF, BGP Master Site

SCADA/ EMS

PBX GW

Term Server

Video Monitoring

HMI

RTU Mgmt

PoE

Video Storage

Wide Area Network

HMI

PoE

VOIP Enet IED

SerialSerial Serial IED IED IED

HMI

PoE

VOIP Enet IED

Sub A

SerialSerial Serial IED IED IED

Sub C Enet IED

SerialSerial Serial IED IED IED

Sub B © 2014 Belden Inc. | belden.com | @BeldenInc

26

Router Redundancy

© 2014 Belden Inc. | belden.com | @BeldenInc

27

Virtual Router Redundancy Protocol (VRRP) RTU Management & Provisioning

Master Site

SCADA/ EMS

Video Monitoring

PBX GW

Term Server

Switch Video Storage

Slave Router(s)

Master Router

Wide Area Network

© 2014 Belden Inc. | belden.com | @BeldenInc

28

Virtual Router Redundancy Protocol (VRRP) RTU Management & Provisioning

Master Site

SCADA/ EMS

Video Monitoring

PBX GW

Term Server

Switch Video Storage

Slave Router(s)

Master Router

Wide Area Network

© 2014 Belden Inc. | belden.com | @BeldenInc

29

Virtual Router Redundancy Protocol (VRRP) RTU Management & Provisioning

Master Site

SCADA/ EMS

Video Monitoring

DATA

PBX GW

Term Server

Switch Video Storage

Slave Router(s)

Master Router

Wide Area Network

© 2014 Belden Inc. | belden.com | @BeldenInc

30

Virtual Router Redundancy Protocol (VRRP) RTU Management & Provisioning

Master Site

SCADA/ EMS

Video Monitoring

DATA

PBX GW

Term Server

Switch Video Storage

Slave Router(s)

Master Router

Wide Area Network

© 2014 Belden Inc. | belden.com | @BeldenInc

31

Cellular Redundancy

© 2014 Belden Inc. | belden.com | @BeldenInc

32

Before Cellular RTU Management & Provisioning

Master Site

SCADA/ EMS

Video Monitoring

PBX

DATA ATA

GW

Switch

Term Server

Video Storage

Wide Area Network

HMI

PoE

DATA VOIP Enet IED

SerialSerial Serial IED IED IED

Sub A

© 2014 Belden Inc. | belden.com | @BeldenInc

33

Before Cellular RTU Management & Provisioning

Master Site

SCADA/ EMS

PBX

ATA

GW

Switch

Term Server

Video Monitoring

Video Storage

Wide Area Network HMI

PoE

VOIP Enet IED

SerialSerial Serial IED IED IED

Sub A © 2014 Belden Inc. | belden.com | @BeldenInc

34

Cellular Backup RTU Management & Provisioning

Master Site

SCADA/ EMS

PBX

DATA DATA

GW

Switch

Term Server

Video Monitoring

Video Storage

Verizon Wireless Internet

Wide Area Network

HMI

VOIP Enet IED

PoE

DATA DATA SerialSerial Serial IED IED IED

Sub A © 2014 Belden Inc. | belden.com | @BeldenInc

35

PRP/HSR Redundancy

© 2014 Belden Inc. | belden.com | @BeldenInc

36

Parallel Redundancy Protocol (PRP)Protocol) Zero failover with Network Redundancy

“0ms” recovery

DAN Dual Attached Node SAN Single Attached Node

SAN

SAN

SAN

SAN DAN

DAN

SAN

SAN

Two redundant networks By doubling the packets no data loss if one packet fails PRP-Redundancy-Box = bidirectional splitter and combiner

© 2014 Belden Inc. | belden.com | @BeldenInc

37

High-Available Seamless Redundancy (HSR)

HSR Packet

HSR Packet

No packet loss

© 2014 Belden Inc. | belden.com | @BeldenInc

38

Security

© 2014 Belden Inc. | belden.com | @BeldenInc

39

Today’s Networks RTU Management & Provisioning

Master Site

SCADA/ EMS

PBX GW

HMI

Switch

Term Server

Video Monitoring

Video Storage

PoE

HMI

Wide Area Network

PoE

VOIP Enet IED

SerialSerial Serial IED IED IED

HMI

PoE

VOIP Enet IED

Sub A

SerialSerial Serial IED IED IED

Sub C Enet IED

SerialSerial Serial IED IED IED

Sub B © 2014 Belden Inc. | belden.com | @BeldenInc

40

Defense in Depth Wide Area Network

HMI

PoE

Enet IED

Enet IED

Serial Serial Serial IED IEDIED

SCADA

VOIP

Enet IED

Serial Serial Serial IED IEDIED

Serial Serial Serial IED IEDIED

Enet IED

IT

Engineering

Sub

© 2014 Belden Inc. | belden.com | @BeldenInc

41

Firewalls

Public NonSecured Network

Private Secured Network



Integrated or standalone device that can be configured to allow or block specific traffic and users



DX/10Series Routers offer integrated Firewall features



Eagles are standalone devices

© 2014 Belden Inc. | belden.com | @BeldenInc

42

Firewalls •

Firewall with Stateful Packet Inspection (SPI) − Both

IP and MAC address filtering supported



Network Address Translation (NAT)



VPN support

10XTS, 10ETS, 10RX

DX Eagle

© 2014 Belden Inc. | belden.com | @BeldenInc

43

Eagle 20 Tofino – Firewall + DPI •

Firewall with Stateful Packet Inspection (SPI)



Layer 2 Bridge with No IP Address − No

disruption to existing network design

− VERY



secure

Content Inspection filters traffic at the protocol level (Deep Packet Inspection) − Modbus/TCP − others



to follow

Simple deployment, configuration and management

© 2014 Belden Inc. | belden.com | @BeldenInc

44

Tofino™ Modbus TCP Enforcer LSM: Content Inspector for Modbus •

Protocol ‘Sanity Check’ blocks any traffic not conforming to the Modbus standard



Control engineer defines list of allowed Modbus commands, registers and coils



Automatically blocks and reports any Modbus traffic that does not match your rules

© 2014 Belden Inc. | belden.com | @BeldenInc

45

VPN

© 2014 Belden Inc. | belden.com | @BeldenInc

46

Virtual Private Networks RTU Management & Provisioning

Master Site

SCADA/ EMS

PBX GW

Switch

Term Server

Video Monitoring

Video Storage

Encrypted using IPSec HMI

PoE

Wide Area Network

HMI

PoE

VOIP Enet IED

SerialSerial Serial IED IED IED

HMI

PoE Enet IED

SerialSerial Serial IED IED IED

Sub A Sub C Enet IED

SerialSerial Serial IED IED IED

Sub B © 2014 Belden Inc. | belden.com | @BeldenInc

47

Security - VPN •

Hardware and software encryption



Multiple tunnel support



Pre-Shared Key (PSK) or X.509 Certificates



IPSec



DX/10Series Routers offer integrated VPN features



Eagles are standalone devices

© 2014 Belden Inc. | belden.com | @BeldenInc

48

Port Security

© 2014 Belden Inc. | belden.com | @BeldenInc

49

Port Security •

Default with all ports “administratively set to DOWN”



Some devices support “no tail ending”. Port is locked after being unplugged. Must be enabled by administrator



Physical port security devices



Unusual port connectors provide a small level of security

© 2014 Belden Inc. | belden.com | @BeldenInc

50

MAC Based Port Security •

Secures physical ports by applying a MAC based filter on a per port basis which allows only the authorized MAC address to forward traffic from the given port.

© 2014 Belden Inc. | belden.com | @BeldenInc

51

IP Based Port Security •

Secures physical ports by applying a IP based filter on a per port basis which allows only the authorized IP address to forward traffic from the given port.

© 2014 Belden Inc. | belden.com | @BeldenInc

52

Authentication

© 2014 Belden Inc. | belden.com | @BeldenInc

53

Authentication •

Switches and Routers

support RADIUS Authentication − Protects

access to the

1

console ports − Authenticates

4

users to

Network

the network -

Helps satisfy CIP

authentication requirements

2

3

Radius Server

© 2014 Belden Inc. | belden.com | @BeldenInc

54

Authentication •

Secure Access Servers − Subnet

Control Center Engineering Access

Solutions

Secure Access Manager

− CrossBow − Cooper

Power RSA



Satisfies CIP access record keeping requirements

Network

Communications Gateway IED

IED

RTU

IED

IED

RTU

© 2014 Belden Inc. | belden.com | @BeldenInc

55

Summary •

Substation to the Control Room Communications − − − −



?

How to Build a Redundant Network − − − − − −



Legacy networks Networking today CIP Hardened equipment

RSTP MRP Routing Router Cellular PRP/HSR

How to Lock it Down −

Firewalls



VPN Port Security Authentication

− −

© 2014 Belden Inc. | belden.com | @BeldenInc

56

Top Three Takeaways



Multiple Redundant protection schemes to pick from when designing/upgrading a network



Security Features that support CIP compliant networking requirements



Belden - Industry Leading Product Depth and Experience

© 2014 Belden Inc. | belden.com | @BeldenInc

57

Additional Resources & Assistance Obtain further Substation Communication resources from our website:

1.  

www.belden.com/power-td/ This webpage includes substation communication diagrams and other useful tools

Contact a Belden representative for assistance:

2. 

Call 510-438-9071 if you are in the U.S. or Canada



Or complete the form at www.belden.com/contact/

Thank you for your interest in this presentation!

© 2014 Belden Inc. | belden.com | @BeldenInc

58

Belden.com

|

@BeldenInc

© 2014 Belden

Suggest Documents