The problem with ICS security improvements

In the BruCON workshop on ICS and SCADA security (which you can read about here), we learned how to hack ICS systems using Wireshark. In the progress of which we also came to some conclusions about problems with ICS security in real life.

One problem with ICS and SCADA systems is the limited hardware. Old PLCs run very limited chipsets, not at all comparable to the hardware that now runs our small devices. These systems were only designed to handle a very small set of data and logic actions. Even if you wanted to include them, there just isn’t any room for advanced software features like proper authentication or encryption.

Furthermore, many industrial facilities are now using unsupported PLCs. Upgrading security would mean replacing all of these old devices.

Then, if you decide to do so, making changes in the PLC environment is not so simple, when the environment is live. The focus of the business is on uptime. Awareness of ICS security is too low, so security improvements are not held in great value by the business, compared to the cost of stopping a production line.

Workshop ICS Pentest 101: A Junior’s View

Last month I attended Brucon 2017, the leading Belgian security conference held at the University of Ghent. The conference offers talks by renowned speakers and you can get your hands dirty in a variety of workshops. I attended ‘ICS Pentest 101’ by Arnaud Soullié and Alexandrine Torrents from Wavestone, a French consulting company. Since I had no prior knowledge of SCADA and Industrial Controls Systems (ICS) whatsoever, I was on unfamiliar ground.

I tried to imagine beforehand what pentesting ICS and SCADA systems would look like: some Hollywood style hacker plots came to mind, like scenes from ‘Mr. Robot’. Hackers can easily take control of machine rooms, robots, elevators and ventilation systems with rather unrealistic hacking methods.
I did not realise those scenes weren’t too far from the truth.

First, Alexandrine gave an introduction about how these systems actually work. She showed us how Programmable Logic Controllers (PLCs) of known manufactures, like Siemens and Schneider, function. She provided us with virtual machines to experiment with and we played around with simulated PLC software.
In Mr. Robot, when Elliot wants to take control over a factory or power plant, he simply needs to physically plug in on the network where all the PLCs are placed, to gain control over the machines. Because there is no data encryption or any form of authentication, he can simply intercept the traffic and make changes on the fly to tamper with the network traffic that is being sent to the machines.
This is exactly what we experienced during the workshop. We intercepted network data with Wireshark and edited it to change a machine’s behavior. We also learned to attack the unpatched PLC itself to take control of the system.

If these systems were critical for safety, holy *% !

The colour of the hacker’s hat

On October 25 the CIOforum, ISACA Belgium and Antwerp Management School organise a CIO Speaker’s Café on cybersecurity in Technopolis, Mechelen. Access is free, and I’ll be there to talk about hackers and IT/OT (aka Information Technology and Operational Technology). Interested in a sneak preview?

Hackers with different hats

First on the agenda: the different kinds of hackers. A black-hat hacker is a hacker who is aggressive in nature and violates computer security, for example for personal gains. A derived version thereof is the red-hat hacker, who is employed by a government agency. His purpose is to hack into computer mainframes of other governments in order to disable or cripple them. There are also grey-hat hackers who can be both helpful and harmful, for example the Anonymous collective. White-hat hackers are security specialists you hire to verify the security of your networks and applications. They are your partners in crime to improve your level of security. At Toreon, we see ourselves as white-hat hackers.

And then there’s you. You might not mean to, but the largest cybersecurity risk comes from you. An analysis of the data from 1982 to 2010 found that the type of security incidents affecting control systems breaks down as follows: 50% of incidents were accidental, 30% were due to malware infections. Only 20% was the result to external or internal attacks.

Assessing the risks

If an incident occurs in an ICS environment, the impact on your business, on human safety and on the environment can be huge. Luckily you don’t have to just sit there and wait for an incident to happen. There are ways to proactively improve your cybersecurity. A security team, an inventory of your IT/OT environment, a verification of your security levels, good governance and awareness in your organisation are key.

Want to hear the rest? Join me and amongst others Ronald Verbeek (Director CIO Platform Nederland), Jethro Cornelissen (Cyber Security Incident Response Team Rabobank) and Yuri Bobbert (CISO, lector and Researcher Antwerp Management School) at the CIO Speaker’s Café! More information can be found here. Or you can send me an email.

How secure is your remote management solution?

When we perform security assessments for ICS (industrial constrol system) customers, we often notice that several different remote access paths for suppliers are used for remote management purposes. Most of these are established through a separate DSL line. Makes sense, right? A solution like this makes it easier for the vendor to provide remote maintenance. The setup is simple for the customer and the IT department can be bypassed.

I disagree! Let me explain why this type of setup is worrysome, how it can be improved and why better designed remote access solutions are important.

Security and maintenance

There’s a reason that a security conscious IT department is usually very reluctant to set up remote access paths that bypass the regular network.

Sometimes those DSL lines have their own security measures built in. But usually, they don’t have any at all. This leaves the burden of securing these lines with the IT department again. And with added security comes complexity and an increase in maintenance efforts – usually for both vendor and client.

A vendor that has some operational responsibility over the installation, should be more concerned with the security of his devices. They are more easily accessible from insecure networks through the DSL. The vendor will have to put more effort into patching and updating them.

A risk-aware IT department will usually add some extra security measures to the mix. A firewall and a VPN solution are the bare minimum. Those require maintenance, patching and monitoring. If those measures are not added, the alternative is usually to limit the window of opportunity by only enabling the DSL line when the vendor needs access. This adds to the responsibilities of IT staff: a strict procedure has to be followed. If not, lines may be kept open indefinitely, exposing critical ICS systems to the world! This happens more often than you would expect and it is something we definitely look for when scanning for security problems..

To avoid breaches, our client has to verify the security of their ICT infrastructure on a regular basis. This can be done using recurring penetration testing and system analysis. Every remote connection in use has to be tested, be it DSL line, leased line or modem connection. That’s a lot of work. If weaknesses are discovered, the lack of monitoring on these access points makes it hard to figure out whether an intrusion may have already occurred.

A secure solution

There IS a manageable solution to this problem that is also secure.

A proper and secure remote management solution for process environments should be:

  • centralised
  • time based
  • source IP controlled
  • using strong user authentication
  • monitored and logged

Centralised: all remote maintenance connections are made to a central VPN infrastructure that is regularly tested on several security aspects. This infrastructure is managed by the company itself. From that infrastructure, secure tunnels are setup to the different components that need to be managed by the vendor.

Time based: you don’t want to leave the connection open at all times. Connections can be granted access for a certain time (with a maximum) after which the connection is automatically closed.

Source IP controlled: because you don’t want just anybody on the Internet to connect to your remote maintenance connection, you should control the source IP addresses inside the VPN connection.

Strong authentication: every vendor that wants to have access should have their own set of user credentials to logon to the environment. Passwords have to follow company standards but using strong authentication devices (tokens) is very much preferred.

Logged: all acces to and from the remote management solution and to/from a hop station should be logged. Logs should tell you who connected, from where, what they did on which systems and how long the connection lasted. This way you are not only able to protect your environment from external theats, but it allows you to trace all actions in case something went wrong.

‘Jump stations’ should be mandatory as well to facilitate a good remote access solution for 3rd parties. The vendor logs onto a jump station and can acccess the right equpment from there, using the tools pre-installed for him. The use of jump stations allows you to further segregate your network and refine access. It becomes easier to close parts of your process environment to the individuals performing maintenance and limits the tools usable by the maintenance provider tot the ones made available on the jump station. The station can have user monitoring based on screenshots that allows you to see exactly what the support engineer is doing (or has done) on your environment. This enables tighter control at all times and full auditing capabilities after the fact.

Do you have any other ideas on how to safely implement secured remote management for process environments? Let me know in the comments below.

Why data integrity is crucial for safety in industrial environments

It is well known that the most important factor within the process industry is the availability of the systems and environment. The plant must be running at all times. That’s why most security improvement efforts are made in that area.  However, the integrity of the information and data within the industrial control systems environment can’t be neglected. Not only because we need correct data to let the process run correctly, but also because data integrity failures can lead to safety issues. More precisely to human and environmental safety.

That’s why it is absolutely crucial that the information you are sending to and receiving from your industrial environment (Human-Machine Interfaces (HMI), Programmable Logic Controllers (PLC), Remote Terminal Units (RTU), application servers, etc.) is 100% correct. That way you can make the correct decisions to control your environment or to know how to respond to events.

Imagine that information is not 100% correct. Do you think that this isn’t a (big) problem? Then read the following potential scenario’s that abuse information integrity:

  • What if somebody is able to change the values that are sent from the various components (PLC’s, RTU’s, etc.) to the application servers or HMI? That would mean operators get false information and might take actions that cause severe damage to the industrial installation (for example opening some valves that should remain closed while operating or starting some pumps that should remain off during a normal situation). What if a person is at that time standing next to the installation that is going haywire? What if this happens within a chemical plant or nuclear power plant?
  • Image that somebody tampers with the images that are displayed onto your HMI’s? They might show you that things are going wrong within the installation while in fact your plant runs smoothly. Acting on this wrong information could make your plant not run that smoothly anymore … So in this case the integrity of the images stored on the engineering system is important.
  • What if somebody is able to sabotage a fire detection system so that alarms are no longer passed onto the physical alarms and/or DCS environment to indicate there is a problem? You can argue that this is not that bad, until an actual fire breaks out and no detection, warning and/or evacuation signal is given. This could lead to an impact of people safety if people are around and/or within the building on fire.

These examples clearly show that the integrity of the information within an industrial control system can’t be neglected in any way. Integrity issues can cause downtime, environmental safety issues and human safety issues and should be treated as an important item on your security checklist, next to (or just below) availability.

If I got you curious, make sure to read “Blackout” from Marc Elsberg (only available in Dutch and German), a fiction story in which the integrity of a process environment is impacted, and they way that affects the environment and social life.

How do you build a (modest) ICS testing & training lab?

Part of training people into becoming ICS security specialists is providing them the opportunity to test or train certain things in a ‘safe’ environment. Which means you’ll need a (modest) ICS testing & training lab. There are some great labs out there (Idaho National Labs for example), that offer every test set-up you can think of. But not all companies have the resources to build a lab of that scale.  For most companies or organisations a testlab environment limited to simulations of their own processes is just fine. There’s quite a bit of information you can find online about building such a lab. And usually it’s suggested that building it is fairly easy. Is it really?

You need ‘real’ stuff…

To build a representative lab you should have ‘real’ stuff in it. And with ‘real’ stuff I don’t mean having one or two Progammable Logic Controllers (PLC) combined with some Remote Terminal Units (RTU). A real ICS lab should contain everything that is present in a real environment. To simulate a power plant for example, you would need an actual Distributed Control System (DCS) that connects to several PLC’s, RTU’s, sensors etc. so that those devices can be taken into account during a simulated test/training. It is also important to have a real (physical) process running behind it so that physical impact of attacks can be observed properly.

…and network environments

And that’s just the process side of the testing lab. Your ideal lab should contain more than just that. This means that supporting environments for the process side should be included (think about the fire detection system, physical access systems, PI/OPC gateways, etc.). And don’t forget to include a simulated office environment with all regular business services in it (e.g. email server, file sharing, intranet, etc…) as well as a “simulated Internet”. These should be included to simulate different attack scenario’s and investigate different possible attack paths. You can play the role of an attacker on the “simulated internet” and try to get in or you can assume that you have physical access to the corporate offices and put yourself into the simulated office environment.

Where to start?

You have several options, depending on how much you are willing to spend on your lab. You can check eBay and buy several old or decommissioned RTU’s/PLC’s, or you can look out for decommissioned plants and buy their equipment. Another option is to procure training kits from big(ger) SCADA vendors such as Siemens, Phoenix, ABB, Scheider Elecric, etc. Or you can build things yourself with small devices such as Raspberry PI or other alternatives. Don’t be afraid to experiment and combine several things/vendors together. Make sure you also have servers (for virtualisation purposes), switches, hubs, routers, firewalls etc. and learn how to use them. By building your own testing environment you will certainly learn a lot on basic networking skills.

Testlab procedures

Once your lab is ready to be used, make sure to create some procedures. Take especially good care of staging, as this is often forgotten. Nothing is as frustrating as losing a lot of time to get your lab back to its original state after a successful ‘attack’. At this stage you will be very happy if you have a (tested) staging procedure which gives you an easy step-by-step guide to revert to a known good state of the lab.

Got any experiences with building or using ICS test labs? Then let us know via the comments below.

How do you become an ICS Security Specialist

During one of my recent lectures on ICS Security one of the students asked me where he had to start to become an ICS Security Specialist. Since I couldn’t give a clear answer right away, I put some thought into the subject and tried to gain more insights on the most important requirements and potential career paths.

OT versus IT

For starters, I need to explain the difference between Operational Technology (OT) and Information Technology (IT).

OT refers to computing systems that are used to manage industrial operations. Operational systems include production line management, mining operations control, oil & gas monitoring etc.

IT refers to computing systems that are used for other more administrative tasks. IT systems include e-mail, ERP solutions, office suites, internet access, etc.

It’s important to know the difference, as there are several differences between the tasks and responsibilities of both environments. Which means they have other priorities and concerns.

DIY: Do it yourself

Since there are no real studies combining bot topics, you will have to acquire the necessary skill sets and gain knowledge on the subject yourself. Training courses such as SANS ICS training curriculum, Red Tiger ICS training, Scadahacker ICS trainings can help you along the way.

When you start your career, you will most likely not start as an ICS Specialist. Which means you might end up on either the IT or the OT side of a company. Being an ICS Security Specialist implicates that you are ‘on top’ of both. So you need to get acquainted with ‘the other side’. Make sure to talk to people working in both environments, since they have tons of knowledge and experience to share. Don’t be afraid to ask questions and be eager to learn. Accept that you might have different views on security however.

Personally, I am an IT person who stepped into the OT scene several years ago. My career path led me from being a master in electronics, towards a job as security engineer. Later I became a security consultant and penetration tester (https://www.corelan.be/index.php/2015/10/13/how-to-become-a-pentester/) until I ended up as an ICS Security Specialist. That would not have been possible without tasting both sides, with an open mind and attitude.

Join the other side

In the past, OT environments used to be island. Today that’s no longer the case, as they get integrated with IT networks. Business requirements and mobile data access are the most common drivers, next to manageability and even security (although that might seem to be counterintuitive). This makes the need for IT/OT security experts more pressing than ever. So if you’re thinking about becoming an ICS Security Specalist, you  will not have to worry about finding a job.

Triggered by the IT/OT challenges? Then listen, learn and join ‘the other side’. Let us know what you think is important to become an ICS Security Specialist in the comments.