Electricity for Free? The Dirty Underbelly of SCADA and Smart Meters

generation (DCS)

transmission (EMS)

distribution (DMS, AMR, AMI)

© Copyright Red Tiger Security – Do not print or distribute without consent.

1

next hour of your life… •! High level review of how the electric grid works •! the role SCADA systems play in the generation, transmission, and distribution of electric power •! typical network diagrams and data flow for each part of the process

•! SCADA VA Assessment Methodology 101 •! 6-layer approach to ensure all system components are checked •! “point-click-scan” == system shutdown

•! SCADA and Smart Grid Vulns •! SCADA Vulnerabilities from a data set of over 38,000 vulnerabilities from live SCADA assessments •! Vulnerabilities with new SMART Grid Technology

© Copyright Red Tiger Security – Do not print or distribute without consent.

2

random thoughts •! Obama was quoted as stating a figure of $1 trillion lost last year to cybercrime—a bigger underworld than the drugs trade •! Banks and other companies do not like to admit how much data they loose. In 2008 alone Verizon recorded the loss of 285 million personal-data records, including credit-card and bankaccount details, in investigations conducted for clients. •! SCADA systems that control all critical functions required for civilization (water, food supply, power, fuel, etc…) are far less secure than Enterprise IT systems.

© Copyright Red Tiger Security – Do not print or distribute without consent.

3

…more random thoughts •! SCADA system administrators can no longer hide behind obscurity •! Most common SCADA protocols are one Google-click away – try it at home for yourself “Modbus Protocol Specification” •! Fuzzing / hacking SCADA is simplistic •! Malware writers are already targeting SCADA •! July 15th: “Trojan makes database queries that point towards the WinCC SCADA system by Siemens” •! Smart meter hacking kits rumored to be released in other countries…

•! Remember that little APT (Advanced Persistent Threat) problem… well that impacts SCADA too

© Copyright Red Tiger Security – Do not print or distribute without consent.

4

So are we winning the cyber war?

…hard to when we are asleep…

© Copyright Red Tiger Security – Do not print or distribute without consent. 5

…especially with important stuff… © Copyright Red Tiger Security – Do not print or distribute without consent.

6

so let’s learn how the stuff works •! power generation •! power transmission •! power distribution •! Disclaimer:: Any network diagrams contained here are only typical / common architectures and not reflective of any one particular system.

© Copyright Red Tiger Security – Do not print or distribute without consent.

7

power system overview

© Copyright Red Tiger Security – Do not print or distribute without consent.

8

first choke point DCS firewalls typically weak and often contains realtime ICCP connections to other systems DCS operator stations, servers, and applications typically not hardened. DCS controllers prone to weak stacks.

© Copyright Red Tiger Security – Do not print or distribute without consent.

9

first choke point EMS firewalls typically weak. EMS systems integrate data from many diverse systems, including Internet sources

RTUs and IEDs at the substation level can drop load to sections of entire cities

© Copyright Red Tiger Security – Do not print or distribute without consent.

10

first choke point DMS firewalls typically weak.

AMR systems often integrated with DMS systems and Enterprise IT applications for billing, etc..

DMS systems integrate data from many diverse systems, including Internet sources. DMS also have load shedding capabilities. RTUs and IEDs can drop load to entire neighborhoods

Smart meters have known flaws © Copyright Red Tiger Security – Do not print or distribute without consent.

11

•! Cyber security standards and regulations, including NERC CIP and ISA S99, are calling for specific perimeters or security levels •! Access between security levels must be controlled, monitored, and logged. •! At least now we have the start of a model to go by for securing critical infrastructure. © Copyright Red Tiger Security – Do not print or distribute without consent.

12

SCADA VA Methodology 101 •! Over the past 10 years, we have built a proven process for safely conducting security assessments on live SCADA systems •! Follow a logical 6-layer approach that covers all system components in the following areas: 1.! Physical Security 2.! Network Infrastructure 3.! SCADA DMZ 4.! Control Room Assets (Servers/Hosts) 5.! SCADA Communications and Protocols 6.! Field Devices

© Copyright Red Tiger Security – Do not print or distribute without consent.

13

© Copyright Red Tiger Security – Do not print or distribute without consent.

14

© Copyright Red Tiger Security – Do not print or distribute without consent.

15

quick keys to a successful SCADA assessment •! Know thy tools •! Actively scanning SCADA systems and devices can cause system outages •! Use a passive approach •! Capture traffic, config files, and data for offline analysis •! If possible scan secondary systems, test systems, or development environments

© Copyright Red Tiger Security – Do not print or distribute without consent.

16

SCADA Vulnerability Trends •! Our team has performed over 120 security assessments of critical infrastructure facilities such as electric power generation plants, transmission energy control centers, chemical plants, water plants, and oil/gas production, refining, and pipeline systems •! Vulnerability analysis and classification conducted under research project facilitated by INL and funded through the DHS Control Systems Security Program contract #240704 •! ISA99 architecture model used to classify where the vulnerabilities were discovered in the systems

© Copyright Red Tiger Security – Do not print or distribute without consent.

17

© Copyright Red Tiger Security – Do not print or distribute without consent.

18

data source – what was collected? •! From mid-2002 to 2008, vulnerability data was stripped of any client information and the raw vulnerabilities were captured in a database •! Vulnerability ID (auto-numbered from entry number 1) •! Vulnerability Title (title for the vulnerability) •! Security Zone or Location (location based on the ISA99 model where the vulnerability was located) •! Disclosure Date (date when vulnerability was disclosed) •! Discovery Date (date when vulnerability was discovered by the team and entered into the database) •! Days Between Disclosure and Discovery (time between disclosure and detection) •! Vulnerability Detailed Description •! Vulnerability Suggested Remediation Steps © Copyright Red Tiger Security – Do not print or distribute without consent.

19

We Don’t Need No Stinking SCADA 0-Days •! avg. # of days between vulnerability disclosure and discovery •!

all field data was exported from the database to an excel spreadsheet containing over 38,000 rows, and much of the analysis had to be performed manually

•!

since we captured when the vulnerability was disclosed in the public, and also captured when the vulnerability was discovered and entered into the database, we were able to perform a simple diff against these two fields

•!

vulnerabilities that were never disclosed in the public were thrown out of this particular exercise since negative or zero entries would throw off the calculations

•!

the maximum number of days between when a vulnerability was disclosed in the public and when it was found during an assessment was over 3 years!

•!

the average was 331 days, or close to 1 year. this means that on average most SCADA and process control environments contained latent vulnerabilities, probably with known already-compiled exploits, and were not discovered until almost a year later, and would not have been discovered had they not decided to have a security assessment performed on their system.

© Copyright Red Tiger Security – Do not print or distribute without consent.

20

where are most of the vulnerabilities being discovered?

© Copyright Red Tiger Security – Do not print or distribute without consent.

21

Operational or SCADA DMZ most vulnerable •! Almost half of the total vulnerabilities were found in the DMZ between the Enterprise IT and SCADA systems •! Often we find that SCADA system owners struggle with which group in their company has the ownership and responsibility for maintaining the systems in this part of the network •! The Operational DMZ network is the first stepping-stone from the Enterprise IT network, and is the most common threat vector for attacks against SCADA systems •! Now we have found the “Perfect Storm” whereby the most connected area of the SCADA system also contains the most vulnerabilities, and is often overlooked by system administrators

© Copyright Red Tiger Security – Do not print or distribute without consent.

22

why is this so important… •! The Operational or SCADA DMZ network is the last line of defense before any traffic hits the SCADA and Industrial Process Control systems •! In many cases, the servers, workstations, and applications in this middle area are all authorized and trusted by the SCADA systems •! By dissecting the vulnerabilities in this level of the network, we can determine how the vulnerabilities at this level in the architecture can be exploited

© Copyright Red Tiger Security – Do not print or distribute without consent.

23

systems impacted at the Operations DMZ zone

© Copyright Red Tiger Security – Do not print or distribute without consent.

24

SCADA, how can I own thee, let me count the ways

© Copyright Red Tiger Security – Do not print or distribute without consent.

25

© Copyright Red Tiger Security – Do not print or distribute without consent.

26

workstation HMI vulnerabilities ranked by OS

© Copyright Red Tiger Security – Do not print or distribute without consent.

27

want to run that critical plant on windows?

© Copyright Red Tiger Security – Do not print or distribute without consent.

28

only logged 105 controller LAN vulnerabilities, but QnX showed up as the most typical source

© Copyright Red Tiger Security – Do not print or distribute without consent.

29

interesting security findings on control system networks •!

VOIP (Voice over IP) Systems

•!

Network Video Recording Devices

•!

!!

Software license cracking executables (CD-key generators)

!!

Torrent client software on Supervisor HMI LAN

!!

Paging Software Server (i.e. Air Messenger Server connected to both the SCADA and Internet for SMTP relay out)

!!

America Online Clients

!!

MP3 Music and Video Playing Software including iTunes

!!

Streaming Music and Radio software with vulnerabilities

!!

BitTorrent Clients (for peer-to-peer file sharing)

Network Surveillance Equipment and Software

•!

Adult Video Directory Scripts

•!

Online Dating Service Databases

•!

Advanced Forensics Format (AFF) archives

•!

Gaming Software Servers •!

aGSM - a freeware game server info monitoring utility

•!

Alien Arena 2006 Gold Edition

•!

Counter Strike

•!

Brood Wars

•!

Battlefield 1942 Server and Clients

!!

MSN and other IM chat clients

•!

Quake 2 and Quake 3 Game Servers found in Supervisor HMI LAN

!!

•!

Soldier of Fortune II

Anonymous FTP Servers running waiting for connections

© Copyright Red Tiger Security – Do not print or distribute without consent.

30

but wait…there’s more •!

Apache Web Servers and Linux hosts un-patched for over 2 years

•!

APC Battery Backup UPS systems with vulnerable Web Interface

•!

•!

•!

•!

Several web blog site engines running in control system DMZ Office grade Linksys, Belkin, and D-Link WiFi devices on Supervisory HMI LAN IM clients found installed and contained vulnerabilities on Supervisory HMI LAN Windows 95 found installed on hosts in Supervisory HMI LAN (no longer supported by MS)

!! Windows NT found installed on hosts in Supervisory HMI LAN (no longer supported by MS) !! Windows Vista found used as OS for operator consoles in Supervisory HMI LAN !! IRC Chat Servers found installed on hosts in the Operational DMZ LAN !! Nintendo Entertainment System (NES) Game Simulator !! Netscape Browser vulnerabilities detected in Supervisor HMI LAN !! Multi-function Printer/Fax/Scanner device vulnerabilities !! Botnet code and Remote Command/ Control malware

© Copyright Red Tiger Security – Do not print or distribute without consent.

31

SCADA Vulnerability Summary / take away points •! 331 = the average time in days between when a vulnerability was disclosed in the public versus when it was discovered in an industrial control systems assessment •! the intermediate Operations DMZ network that sites between the Enterprise network and the industrial control systems had the most vulnerabilities attributed to its zone •! web server and back-end database vulnerability findings comprised the largest number of vulnerabilities found in these Operations DMZ network – we need more web app testing! •! number of client workstation vulnerabilities also increased deeper into the real-time operations networks, thus proving we still have a patch problem with SCADA systems •! vulnerabilities with Windows operating systems or Windows applications also accounted for the overwhelming majority of vulnerabilities for systems in the Supervisory HMI LAN •! almost every assessment uncovered unnecessary software installed on the SCADA systems, and in some cases this included botnet and malware code

© Copyright Red Tiger Security – Do not print or distribute without consent.

32

still think we’re doing a good job at securing our critical infrastructure?

© Copyright Red Tiger Security – Do not print or distribute without consent.

33

but wait… we have NERC CIP to save the day !

© Copyright Red Tiger Security – Do not print or distribute without consent.

34

controls are often bypassed…

© Copyright Red Tiger Security – Do not print or distribute without consent. 35

so are we collectively feeling better now? •! NERC CIP at least forces the hand of Electric Power Utilities to implement a common baseline set of minimum physical and cyber controls •! Most controls in NERC CIP are logical, make sense, and map back to other existing best practices and International standards •! So then why are so many Utilities declaring ZERO (0) critical assets, or playing games with wording in the standard to make loopholes to de-list critical assets? •! Now even if everyone was playing fair and in compliance, now insert shiny new nifty Smart Grid technology into the mix…

© Copyright Red Tiger Security – Do not print or distribute without consent.

couple of thoughts about Smart Grid tech… •! Ami / AMR systems have similar vulnerabilities as SCADA Systems •! History is about to repeat itself •! Old Threats, New Impacts •! Attacks at the communication layer •! Attacks at the device layer

© Copyright Red Tiger Security – Do not print or distribute without consent.

we’ve seen this before… •!

Perimeter issues > these systems are interconnected with business applications (billing, work-order, account management systems, etc..), AND also often connected to operational SCADA and Energy Management systems for load shedding and remote tripping

•!

Back-end Server/Application issues > similar web and database app vulns as business applications, less secure implementation of protocols, and old versions of application frameworks

•!

Too much trust in the Protocol > Most AMI / AMR vendors are simply trusting that the 802.15.4 protocol security implementation will save them, and have not given much thought about scenarios when a communications mote is compromised

•!

End Devices have limited resources / weak stacks > The meters themselves do not typically have the resources to handle security features. Basically, the hardware cannot handle more computationally demanding processes, like upgrading their encryption handling capabilities once deployed. Limited tamper-detection capabilities cited, but not found operational in testing.

© Copyright Red Tiger Security – Do not print or distribute without consent.

field life of 15-20 years – Déjà Vu •! Due to high implementation costs, most AMI / AMR projects have long ROI cost recovery models, and are designed to operate for up to 20 years without requiring system upgrades •! Combine this with patching and firmware upgradability issues, and we are building into place the conditions that created much of the issues with SCADA and Process Control Systems Security •!

“Once these devices get deployed, they aren't going to get upgraded due to cost unless there is a major, crippling vulnerability found in them, and people are shamed into fixing it.” – quote by Jacob Kitchel (security researcher)

•!

“All it will take is someone to get bored and go shut a city down by telling all the communication motes that everyone didn't pay their bill, then half flash the firmware and brick them all.” - quote by Nick DePetrillo (security researcher)

© Copyright Red Tiger Security – Do not print or distribute without consent.

old threats, new impacts •! Data Enumeration (read-time grid data) •! Host Enumeration (what systems can we connect to?) •! Service Enumeration (what services are exposed?) •! Change Data on the fly (can the data be manipulated in flight?) •! Steal accounts and passwords (system admin access anyone?) •! Damage core system components (cause meters to fail…) •! Denial of Service (PING FLOOD, Malformed Packets, etc…)

© Copyright Red Tiger Security – Do not print or distribute without consent.

Man-in-Middle Packet Capture

© Copyright Red Tiger Security – Do not print or distribute without consent.

Write over any data in the stream (real time)

Change usage or billing data…

last24KWh=250;

last24KWh=25.0;

© Copyright Red Tiger Security – Do not print or distribute without consent.

sometimes it is not even that hard…

© Copyright Red Tiger Security – Do not print or distribute without consent.

43

Bricking PLCs and RTUs are relatively easy… “Smart Meters” have similar stack issues

PING Flood Often Results in Faulted PLC Processor. PLC Looses Configuration, and Must be Connected Locally with a Serial Cable to Upload Configuration.

© Copyright Red Tiger Security – Do not print or distribute without consent.

simple ping flood attack…

© Copyright Red Tiger Security – Do not print or distribute without consent.

45

denial of access •! Embedded device has a Login/Write Access password option •! 16 character limit

•! Vendor specific Modbus/TCP function code •! Password stored in the Flash of the controller

•! “This procedure cannot be undone if you forget the password. The device must be sent for repair”

© Copyright Red Tiger Security – Do not print or distribute without consent.

46

thirsty for more? training is available… •! Week of 6 September: •! http://www.redtigersecurity.com/uk-fall-2010/ •! Kent, UK •! If you click on the link provided on our web site, students can register for the course online

•! Week of 7 October: •! http://www.sans.org/eu-scada-security-summit-2010/ •! London, UK •! This event is being handled through SANS, so students have to register through the SANS web site. There is a register button on the page, and students can pick and select which optional training they are interested in. Our course is the one listed as: SPECIAL HOSTED: SCADA Security Advanced Training Oct 7th-11th

© Copyright Red Tiger Security – Do not print or distribute without consent.

47

last few comments… •! Power generation, transmission, and distribution systems all require functioning SCADA, EMS, and DMS systems that are available with 24x7 uptime •! In the past, these systems were isolated systems that used serial protocols and obscure system components •! Recently, SCADA and Control Systems have evolved to the point where they are deployed with network infrastructure components used by Enterprise IT networks (Cisco, Juniper, 3COM, etc.. They also leverage the same Microsoft operating systems and .NET application frameworks •! However, they do not have the security features taken for granted on Enterprise IT systems

© Copyright Red Tiger Security – Do not print or distribute without consent.

48

last few comments… •! The NERC CIP security compliance regulations is making a small dent in the problem, but not enough to hold back a determined attacker •! With the advancement of the “Smart Grid” and AMR systems, without the proper security precautions, the electric grid is now more vulnerable than ever •! Vulnerability Assessments are still the best first step in securing critical infrastructure •! The research that we, and other independent security firms, have performed on SCADA, Smart Meters, and AMR systems expose vulnerabilities that can lead to a situation whereby electricity is free… for those who have the intent and motivation

© Copyright Red Tiger Security – Do not print or distribute without consent.

49

contact info / q & a

Jonathan Pollet, CAP, CISSP, PCIP Founder, Principal Consultant Red Tiger Security, USA office: +1.877.387.7733 mobile: +1.281.748.6401 fax:

+1.800.864.6249

[email protected] www.redtigersecurity.com

© Copyright Red Tiger Security – Do not print or distribute without consent.

50