Putting SCADA Security to the Test: Why you need a lab and how to get one

Chris Sistrunk, PE Sr. Engineer Entergy – Jackson, MS 8th Security Summit Portland, Oregon

Why do we need a lab, Chris? 8th Security Summit Portland, Oregon

What happens when you use nmap on an Industrial Control System

http://securityreactions.tumblr.com

8th Security Summit Portland, Oregon

Why do we need a lab? With a lab, you can • Test relay and RTU settings on a replica of production systems • Test new firmware before issuing to field • Perform root-cause analysis – Why is this device locking up once a month?

• Try out new equipment from a vendor 8th Security Summit Portland, Oregon

Why do we need a lab? Save time & money by • Creating standard settings templates • Find problems before they are widespread (Not having to recall units with firmware issues) • Develop and test equipment pilots in-house rather than hiring a company to do it • Use lab equipment as emergency spare 8th Security Summit Portland, Oregon

Why security testing? • Not all SCADA/relay vendors do negative or security testing at their factories • Even if they did, they can’t test equipment the EXACT way that you use it • Test your own equipment before hackers or some drive-by malware does it for you • Use the results to mitigate vulnerabilities 8th Security Summit Portland, Oregon

What kinds of testing?

8th Security Summit Portland, Oregon

What kinds of testing? • • • • • • •

Factory/Site Acceptance Testing (RTU system) Firmware/Software Testing (new or patches) Protocol Testing (DNP3, Modbus, etc) Protocol Fuzzing (custom or off-the-shelf) Penetration Testing (Metasploit, etc) Physical security testing (cabinet locks etc) DOCUMENT! DOCUMENT! DOCUMENT! 8th Security Summit Portland, Oregon

What would be your stuxnet? • • • •

Be a hardhat hacker Think like an attacker who has your prints! Build your systems with layers of defense If you find a vulnerability, let your vendor know (they might even have a patch)

“To make things work well, you must break them!” 8th Security Summit Portland, Oregon

How I Audit SCADA Systems

http://securityreactions.tumblr.com

8th Security Summit Portland, Oregon

OK, how do I get a lab?

8th Security Summit Portland, Oregon

OK, how do I get a lab? • Ask your boss! Ask the CIO! Ask Ask Ask! • If you are the boss, ask your best people what they want in their lab and go buy it! • Put together a plan or a business case! – Add it to NERC/CIP compliance budget (big driver)

• Go get spare equipment and make a rack! • Start small and add to it. – Mine started as 2 relay racks in my cubicle 8th Security Summit Portland, Oregon

Some ideas

8th Security Summit Portland, Oregon

Still can’t afford it?

8th Security Summit Portland, Oregon

Can’t afford one, don’t have the manpower, don’t have the expertise? • 3rd party testing such as Enernex, Digital Bond, Kinectrics, Cimation to name a few • The US Gov’t has the Idaho NL National SCADA Test Bed, Pacific NW NL, & Sandia NL • Colleges such as Louisiana Tech, Mississippi State, Jackson State have power, SCADA, and security equipment in their labs • Farm out the testing and work with them to get the results you want & capitalize the test costs 8th Security Summit Portland, Oregon

Engineering Truth

“Engineering isn't about perfect solutions; it's about doing the best you can with limited resources.” -Randy Pausch, The Last Lecture 8th Security Summit Portland, Oregon

To be the best, you need the best tools!

8th Security Summit Portland, Oregon

Entergy THQ Virtual Lab Tour

8th Security Summit Portland, Oregon

Transmission HQ Labs • • • •

Transmission HQ moved from NOLA to Jackson Business continuity after Hurricane Katrina Brand new building in Fall of 2009 5 large rooms designated for lab space – – – – –

Relay & SCADA Lab Communications & Security Lab Real-time Power System Simulator Lab Mississippi Grid Lab High Voltage Lab 8th Security Summit Portland, Oregon

Relay & SCADA Lab

8th Security Summit Portland, Oregon

Relay & SCADA Lab NO LAB RATS OR CYBERATTACK SQUIRRELS ALLOWED

8th Security Summit Portland, Oregon

Relay & SCADA Lab

8th Security Summit Portland, Oregon

Relay & SCADA Lab • Cubicle: 2 racks >> Old Break Room: 7 racks • New THQ: 15 bolted racks, 10 rolling racks – – – – – –

40+ Protective Relays (7 different standard panels) Digital Fault Recorder 8+ RTUs, 3 Communication Processors Substation Grade LAN & Corp Network GPS Clock (IRIG-B), HMI Screen & Keyboard Toolbox, O-Scope, Multimeter, Cables, Workstations, Chip Burner, Relay & RTU Test Sets, etc 8th Security Summit Portland, Oregon

Relay & SCADA Lab • • • • • • •

THE LAB OF MY DREAMS! We can replicate almost any substation Test new configurations Test problematic field configurations Test new firmware & software Test drive new equipment Train relay & RTU technicians and engineers 8th Security Summit Portland, Oregon

Communications & Security Lab

8th Security Summit Portland, Oregon

Communications & Security Lab • • • • •

Substation Hardened Router & Switch Radios of different bands and technologies Six-sided PSP for simulating CCA sites Several field firewalls Wurldtech Achilles Fuzzer – – – –

Test network robustness of devices Fuzzing DNP3, Modbus, & IEC 61850 Test new RTU & Relay firmware patches Will network storm affect control outputs? 8th Security Summit Portland, Oregon

Power Real-Time Simulator Lab

8th Security Summit Portland, Oregon

Power Real-Time Simulator Lab

8th Security Summit Portland, Oregon

Power Real-Time Simulator Lab “Hypersim is the only real-time digital simulator with the power to simulate and analyze very large-scale power systems with more than 2000 three-phase buses.” - http://www.opal-rt.com • Simulate different fault scenarios – Will the Relay A, B, C have a misoperation? – Will relay fault activity affect comm (vice versa)?

• R&D & commissioning tests 8th Security Summit Portland, Oregon

Mississippi Grid Lab

8th Security Summit Portland, Oregon

Mississippi Grid Lab • Multipurpose type lab used by Entergy Mississippi T&D Grid Engineers • Inspecting/repairing equipment • Pre-test new panels before field installation • Spare parts inventory

8th Security Summit Portland, Oregon

High Voltage Lab

8th Security Summit Portland, Oregon

High Voltage Lab • The Hi-VARC (High Voltage AC Resistive Current) test set provides rapid, automatic evaluation of MOV arresters and polymer insulators using AC voltages up to 132kV.” http://www.jmxservices.com • Inspection & root cause of failed insulators, HV circuit breaker components, etc 8th Security Summit Portland, Oregon

Last but not least…

8th Security Summit Portland, Oregon

Go make stuff…Go break stuff

8th Security Summit Portland, Oregon

A Few Thoughts SCADA Security isn’t easy • Doing the best we can with what we have

SCADA, Relay, & Security Labs • Having a lab is so valuable for testing, troubleshooting, breaking & fixing stuff • Yes I have a fuzzer and I’m not afraid to use it

DNP3/IP Secure Authentication v5 • Please tell your vendors you want NEED it 8th Security Summit Portland, Oregon

Dream BIG!

8th Security Summit Portland, Oregon

Questions?

[email protected] Follow @chrissistrunk

8th Security Summit Portland, Oregon