Defense in Depth

Why a single firewall security strategy can expose the plant control system to cyber attack

One afternoon in the summer of 2005 the management of a major car company was shocked to discover that over a dozen of their auto manufacturing plants had been shut down by a simple Internet worm. Even though there was a professionally installed firewall between the company and the Internet, the worm had made its way onto the control network, forcing more than 50,000 assembly line workers to cease work during the outages and costing the company an estimated US$14 million.

Interestingly, the company had a sophisticated firewall in place, based on a common strategy known as the Bastion model, where vulnerable systems are hidden behind a single firewall. Unfortunately, this design could not prevent this incident because the problem appeared to have originated from inside the control system, bypassing the firewall completely. And once inside, the worm was able to migrate from one control network to another with ease, as there was no firewalls between the companies manufacturing cells.

Many companies base their plant floor/SCADA security solution on a single firewall between the business network and the control system network, believing that this firewall will be the ultimate security filter and prevent anything evil from ever getting to the control system. Unfortunately as this automotive company discovered, nothing could be further from the truth.

The Bastion model results the possibility of a single point of failure which, with the inevitable help of Murphy’s Law, will eventually either be bypassed or will experience some sort of malfunction. When it does, the plant floor will be left wide open to security issues.

Too many back doors

To understand why the Bastion model fails, it is helpful to look at another Internet worm – the Slammer Worm – and study how it has affected control systems since its creation in 2003.

According to records in the Industrial Security Incident Database (ISID), this one worm has been responsible for more documented incidents of process disruption than any other source. A few of its “achievements” include interrupting power distribution SCADA systems, infecting the safety parameter display system (SPDS) in a nuclear plant, and curtailing oil production operations in the Gulf of Mexico.

What is particularly interesting is that the Slammer worm has used at least five different pathways to get to its control system victims.

In one case, it got into a petroleum control system via a maintenance laptop that was used at home (and infected) and then brought into the plant. In another case, it infected a paper-machine HMI via a dial-up modem that was used for remote support.And in a third case, it passed right through a poorly configured firewall. In all these examples there were firewalls in place, but the worm either bypassed them or it took advantage of some flaw in the firewall’s deployment.

Slammer is just one example – an analysis of seventy-five security incidents against control systems between 2002 and 2006 shows that over half the external attacks come through secondary pathways such as dial-up connections, wireless systems and mobile devices. In these cases, the firewall did its job, but the security strategy failed.

In other cases, the firewall appears to allow attacks and viruses to go right through it. Even this isn’t usually the fault of the actual firewall, but rather the way it is configured.

A typical IT-style firewall requires considerable expertise to design, commission and maintain and the complexity of this task is often underestimated. In a classic 2004 paper on firewall configuration errors, Dr Avishai Wool showed that many IT firewalls in major corporations are enforcing poorly written rule sets and are vulnerable to attack.

In the study, Dr Wool defined 12 serious firewall configuration errors (each very general in nature) and then inspected the firewall configurations of 37 major corporations. He found on average seven serious errors per firewall, with some having as many as 12 errors. In the words of Dr. Wool:

“Almost 80 percent of firewalls allow both the ‘Any’ service on inbound rules and insecure access to the firewalls. These are gross mistakes by any account.”

If these statistics apply to firewalls guarding the plant floor, it is little wonder that cyber issues eventually get through to the control system.

Trouble in the system

Once a misguided employee, virus or hacker does get past the main business or control system firewall, the typical control system is an easy target for attack. Poorly patched Windows-based computers abound and anti-virus software is the exception rather than the rule.

For example, during a security survey conducted at a major refinery, we discovered that only 55 percent of the Windows 2000/XP Machines in control rooms had the patch that prevented Blaster infections and even fewer (38 percent) had the patch for the Sasser Worm installed. Yet both these patches had been available for over two years and were approved by the control system vendor at the time of the survey. Even the most inexperienced hacker could have taken over this control system in a matter of hours.

The actual control devices, such as the programmable logic controller (PLC) or distributed control system (DCS) are even softer targets than the un-patched PCs. In a study by CERN, Europe’s laboratory for high energy physics, 25 industrial control devices (mostly PLCs) were tested using standard IT security tools (such as Nessus and Netwox) that can be downloaded for free by the novice attacker. Almost s third of the devices failed the tests, usually due to communications failures, system crashes and unprotected services.

For experts in the field these results were not all that surprising – the vast majority of the PLCs and DCSs currently in use offer no authentication, integrity or confidentiality mechanisms and can be completely controlled by any individual that can “ping” the device. Nor can they be easily updated or have security features added to them.

Building defenses

#1 – Defining the Perimeter
Sound security strategy, regardless of whether it is military, physical or cyber security, is based on the concept of “Defense in Depth”. Effective security is created by defining the boundaries and then layering multiple security solutions, so that if one is bypassed, another will provide the defense.

It is also based on not over-relying on any single technology. Firewalls are a not bad technology – in fact they are a fantastic tool in the security tool box, but industry has misused them by believing they will solve all security ills.

Evolving industry standards, such as ANSI/ISA-99, call this strategy a “zone and conduit” security model. An industrial facility plant is first divided in to a number different security zones, based on the control function, typical users and the potential consequences of failure. Next, security conduits are used to interconnect the zones, with small firewalls or encryption devices managing the conduits.

For example, in a refinery, a safety integrated system (SIS) might be in one zone, the process control system in second zone, the data historian in a third zone, and the IT network in a fourth zone. Security breaches in each of these systems could result in different consequences, so it makes sense to handle each individually.

Defense in depth design for a zone starts by creating a proper electronic perimeter around the control system zone and then hardening the devices within. The security perimeter for the zone is defined by both policy and technology. First, policy sets out what truly belongs on the zone’s network and what is outside. Then, the control system firewall acts as the choke point for all traffic between other zones and the control system devices in the zone.

Proper design and deployment of a control system firewall is critical – ideally it should be deployed using rules described in guidelines like the CPNI Good Practice Guide on Firewall Deployment for SCADA and Process Control Networks.

Often this is not the case – as Dr Paul Dorey of BP noted in his keynote speech at the Process Control Security Forum (PCSF) in June 2006, comments like “My networks aren’t connected, my server uses a separate network card to connect to the PCN and the corporate network” do not indicate a secure network design and are simply a great way to infect multiple business and control system networks at the same time. Similarly, using routers or switches in place of true firewalls is generally not acceptable.

The requirements in the ANSI/ISA-99.00.02 standard calls for the separation of the Industrial Automation and Control System (IACS) from other systems, including the corporate IT systems. Not meeting standards like these can expose plant and corporate management to both increased liability and in some industries, regulatory penalties.

#2 – Hardening the Plant Floor

Once the electronic perimeter of the zone is secured, it is necessary to build the secondary layers of defense on the control system devices themselves. For those control components – such as HMIs and data historians – that are based on traditional IT operating systems such as Windows and Linux, this should take advantage of the proven IT strategies of Patch and Anti- Virus (A/V) Management.

Many controls engineers mistakenly believe that patching or anti-virus deployment isn’t possible on control systems. While one cannot blindly deploy new A/V signatures or patches into the industrial control environment, the safe deployment of anti virus software or patches on control systems is very achievable.

A number of leading companies have demonstrated that careful A/V and patching policy can balance the need for system reliability with the need for system security. For example, two years ago at ISA EXPO 2006, industry giants Dow Chemical, Proctor & Gamble, and AstraZeneca each described how they successfully deployed anti-virus technology and patch management on their control systems.

In addition, most the major control equipment vendors now offer guidance on both patch management and A/V deployment for their control products. Thus there is little reason for control systems not to have hardened computers on the plant floor through good patch and A/V programs.

Protecting the crown jewels

For critical devices like PLCs, RTUs and DCS controllers where patching or anti-virus solutions are not readily available, the use of industrial security appliances is recommended. This rapidly evolving security solution is based on the use of low cost security modules deployed directly in front of each control device (or group of devices) that needs protection.

Industrial security appliances provide local protection for critical control devices, similar to the way personal firewalls (such as the Windows Firewall) and anti-virus software provide local protection for desktop computers. This way, if the hacker or virus manages to get through the electronic perimeter firewall, it will still be faced with an army of control-focused security devices that need to be breached before any damage can be done.

Two examples of this type of security solution are the Honeywell C300 firewall and the MTL Instruments Tofino Security Solution. Because of their focus on protecting a small number of critical devices rather than a whole network, both of these appliances can be tuned to meet the security needs of the device it is protecting.

The first is a small module that is preconfigured to protect Honeywell controllers from possible attack. It focuses on a specific control device and its industrial design results in a firewall solution that is simple for field personnel to install correctly.

The Tofino Solution is equally simple to install, avoiding the complexity and configuration issues of the typical IT firewall. Field technicians simply connect it between the control device and the rest of the network, apply power and walk away; yet it can also be configured, monitored and managed from a Central Management Platform (CMP) located somewhere on the corporate network.

No single point

“If you entrench yourself behind strong fortifications, you compel the enemy to seek a solution elsewhere.” The words of influential 19th century German military strategist Carl von Clausewitz.

Similarly, industrial security designs that assume all evil traffic will conveniently flow through a single choke point are succumbing to a dangerous set of beliefs. Depending on a single firewall is building a security solution based on a single point of security failure.

Only a proper defense in depth design where the control devices and systems are both individually and collectively hardened can provide reliable security for the plant floor.

========================================================================

Firewall Rules

Ideally, a process control or SCADA network would be a closed system, accessible only by trusted internal components such as the HMI stations and data historians. But realistically, the need for external access from both corporate users and selected third parties exists.

Implicitly, this means that some network paths exist from the outside, untrusted world to internal control components. The goal of the firewall, simply stated, is to minimize the risk of unauthorized access (or network traffic) to internal components on the process control network (PCN).

According to UK’s Centre for the Protection of National Infrastructure (CPNI) guidelines, the current industry best practices call for the process control network to be clearly separated from enterprise/IT network segments using a three-layer firewall architecture. The top layer is the enterprise/IT network, the bottom layer is the control network, and in between is a middle network – typically called the Demilitarized Zone (DMZ) – for shared enterprise/control assets such as data historians

By placing enterprise-accessible items in the DMZ, no direct communication paths are allowed from the enterprise network to the plant floor and each network effectively ends in the DMZ. Most sophisticated firewalls can allow for multiple DMZs and specify what type of traffic is forwardable between zones. Once the firewall in place, the following is a partial list of recommended practices for developing firewall rule sets:

1. The base rule set should be Deny All, Permit None.

2. Ports and services between the control system environment and an external network should be enabled and permissions granted on a specific case-by-case basis. There should be a documented business justification with risk analysis and a responsible person for each permitted incoming or outgoing data flow.

3. All “permit” rules should be both IP address and TCP/UDP port specific, and stateful if appropriate.

4. All rules shall restrict traffic to specific IP address or range of addresses.

5. All traffic on the PCN is typically based only on routable IP protocols, either TCP/IP or UDP/IP. Thus any non-IP protocol should be dropped.

6. Prevent traffic from transiting directly from the PCN/SCADA network to the enterprise network. All traffic should terminate in the DMZ.

7. Any protocol allowed between the PCN and DMZ is explicitly NOT allowed between DMZ and enterprise networks (and vice-versa).

8. All outbound traffic from the PCN to the enterprise network should be source and destination restricted by service and port using static firewall rules.

9. Control devices should not be allowed to access the Internet and control networks shall not be directly connected to the Internet, even if protected via a firewall.

10. All firewall management traffic be either via a separate, secured management network (e.g. out of band) or over an encrypted network with two-factor authentication. Traffic should also be restricted by IP address to specific management stations.

By following these guidelines a clear separation can be maintained, with no traffic directly flowing between enterprise and PCN/SCADA networks.