Transcript
This webinar is the result of a growing partnership between GE and Dragos. Started a couple months ago as we met each other at a conference and realized that GE and Dragos are doing some complementary activities. We decided to start off trying to work out some of our joint ideas and philosophies. And now we're starting to share some equipment and some cyber intel and exploring additional partnerships. So this webinar was kind of the collection of all of this and we decided to share it out and see what people think. See if people have additional questions and ideas that we should be working through. The webinar is scheduled for about an hour and we'll try to reserve the last 20 or so minutes for questions. So we'll start by introducing ourselves. Reid, maybe if you could go ahead and advance the slide. Sure. Thanks. I'm Kenneth Crowther. I joined GE about seven months ago. I started my career off working at Honeywell. I was a chemical engineer at a chemical plant. Went to grad school, studied quantitative risk analysis. Worked as a professor for a couple of years and then went and worked for the MITRE Corporation for the last eight years. And came to GE a couple weeks ago. I'm a product security leader for GE Global Research and part of a product security incident response team or PSIRT here at GE. And I'm Reid Whiteman. I'm a senior vulnerability analyst with Gregos. My background is working a lot as a consultant in the industrial space. I actually got started in the industrial space working for Schweitzer Engineering Labs where I did a lot of negative testing on digital protection relays and other electric power equipment. Since then, I've been doing a lot more for a few years. Did a lot of globe trotting working for both Digital Bond and IOActive, two pretty big consultancies that have presence in the industrial space. I did a lot of time looking around at various plants, not only looking at the security of individual products, but looking a lot at network architecture issues. And at this point, I work for Gregos' intelligence group. So I specifically focus on looking at vulnerabilities, but I also do spend some time looking at what threat actors are doing. Both malware analysis and basically their tools and techniques that they're using for spreading around people's networks. So there are three main ideas that we want to talk about during this webinar. And each of these ideas corresponds with one of the white papers. And this has been a good back and forth between GE that designs, builds these industrial systems and Gregos that understands in detail the ICS systems and the adversaries and the tactics and techniques that they're using to get into ICS systems. So the first idea is that security provides the most value when it's built to support the processes. And so we'll go into this a little bit in a little bit more detail. But when I first start talking to cybersecurity experts who sometimes come from the IT security world, their first kind of reaction is, if it's really important, then let's disconnect it. Let's disable remote control. Let's focus on segmentation and access restrictions. But what we want to emphasize is that the process and the information that supports the process is what derives value. And so security needs to be built around that process. So that's kind of the first point that we're going to make. But as we are connecting things up and enabling remote control, it means that we have additional protection requirements. But it needs to be built around those information flows. So that's kind of the first point. The second point is that once we have identified that critical information and the connections and activities and devices that support it, then the most cost-effective way to tailor our security is by understanding the capabilities of adversaries, the processes, the techniques that they're using to exploit those systems. And then we would know where to target and what type of security measures to put in place and how to piece those security features together. And then the third point is to talk about this evolution. Sometimes the cyber community can focus on device hardening and understanding networks. But in ICS, we need to understand this in the context of the process and production systems, as well as the organization that's enabling those production systems to continue. And so we wanted to emphasize some features of this evolution as it applies to these industrial control systems. So I'll start with this first topic. And Reid, you can go ahead and go to the next slide. And I'll just start with when I first started working at a – it was a caprolactam plant. Basically makes toluene and ammonia byproducts to make this thing called caprolactam, which is what's used to make nylon. And one of the main processes where we put the nitrogen into the carbon ring, we had five parallel trains that were built at different times. So they had slightly different process architectures, different technologies. And especially some of the older lines, we were constantly dealing with irregular disruptions. And it was difficult. Sometimes it would take hours to troubleshoot what's going on, what tripped the system, where did it happen. And so one of my first jobs as a brand-new chemical engineer was to understand the process well enough to document every safety interlock, every electrical, mechanical, pneumatic process that might not be on a P&ID because the P&IDs weren't kept up to date. And in the end, after documenting all of these things, we cut down the troubleshooting time and the ability to respond by like 80%, saved like a million dollars a year. And my next project was focused on a change in the catalyst, which resulted in looking downstream and changing the way that we connect, where we connect the process into the separations area. And again, this little change saved the company another million dollars a year in terms of energy reduction. And the point that I want to draw from this is that the more we understand about our processes, the more we understand about the information that moves and controls the process, the more we will be able to use these engineering and business requirements as a foundation for whatever it is that we're doing, whether it's doing engineering the process itself or whether it's securing the process in a way that continues to generate value for the company. And this is more especially important in industrial control systems as a foundation for security because these control systems operate as close to 24-7, 365 as possible. And there's money that's lost every hour that these processes are down. So to illustrate how this might maybe be unexpected, we tried to pick some kind of provocative examples from some of the papers that I've seen going around GE. So the first one is in the power production process, there are a bunch of very complicated processes that happen simultaneously. So for example, if you shove more cubic feet per minute through a turbine, then you obtain more efficiency in terms of the number of electrons you get out per cubic foot of fuel. But you also wear out the turbine much faster, and so you have to end up paying more for maintenance over the lifetime of the turbine. And so ideally what you would do is you would control the process where when the wholesale price of per kilowatt hour or per megawatt hour reaches a certain price, then you would ramp up the turbine. And when it doesn't exceed that price, then you would ramp it down and you would do it in a way to maximize the amount of money that you're making. And just to illustrate this in the paper, we describe a process where this can potentially make the difference of a million dollars a year in overall production. But it might end up requiring you to connect controllers in order to get information from outside of the enterprise. So in a second example, one of the main losses in petrochemical companies is these unexpected shutdowns. And in order to avoid some of these unexpected shutdowns, there were some processes basically to push operational data out to a data lake. And that data lake is then analyzed by a third party, and they can predict when maintenance needs to be done, pre-stage the equipment. And so this provisioning of third party access provides an extra $17 million in terms of savings from what would otherwise have been lost revenue from shutdowns. And then another example is a mining company in this remote location. Because they're so remote, when a piece of equipment breaks down, it takes sometimes weeks in order to get the part out there. And then they have to schedule very specialized skilled labor to come in, install these parts. And sometimes these outages last for two weeks at a time. But putting on some smart sensors and then collecting them out at a gateway and sending them allows for this predictive maintenance process to happen. And so these are just some examples that emphasize industrial control systems, the need for sometimes the need for remote connectivity. The value that comes from remote control, the value that comes from third party access. And we need to make sure that we're not jumping to security architectural conclusions just from what has worked in the IT security space. Instead, we need to focus on this initial first step is to understand the process itself, how those processes require information, how they move information, what information is required, what connectivity is necessary, and establish those business requirements as a clear foundation as we start to think about cybersecurity. So one of the ways that we've thought to think through this is we've kind of abstracted this scale of connectivity and remote control to illustrate some of these trends that we're seeing in increases in automation and machine centricity of how the automation works, connectivity between machines, even machines that are across enterprises. We're starting to see data lakes and data availability significantly improve the value and provide roles for users who might not be the maintenance or the controls engineers on staff. And so we wanted was a way to kind of describe how this this spectrum of connectivity. And so on the left are the traditional kind of hierarchical isolated systems where a control system is within a process. That process is potentially has some connectivity across the plant. There might be one or more plants on a site that site then has a separated system. The things that we would think about when we think ISA 95 or the kind of the traditional Purdue hierarchical model. And the evolution seems to be pushing things towards the right where we have increasing amounts of remote monitored remote controlled cloud based solutions that are appropriate in different kinds of conditions where the business or the engineering where there might be business or engineering requirements for that type of connectivity. Going to the next slide. The left hand slide. Is this what we call the Purdue model. Which a lot of people on this call have probably familiar with But for some of those systems that would be more towards the right hand side of that continuum. It doesn't provide a good model for what we call the Purdue model. The different operations where those devices are integrated But for some of those Systems that would be more towards the right hand side of that continuum. It doesn't provide a good model for Security architecture that it doesn't provide a model that describes what kind of Security solutions, we should be integrating and how we should be segmenting our system. So on the next slide. And in the white paper, we do kind of give an example of what something might look like on the right hand side. The, the automation area might be collapsed and might be much flatter. To allow the integrated connections between controllers and HMI and Area databases that might help to coordinate and do scheduling of operations. But in order to secure this What we're hopefully going to start seeing is Principles of zero trust networking and micro segmentation. So the segmentation might not be level one, level two, level three, but instead would be around Production concept one production concept to production concept three so that the those areas where they need to share a lot of information. Are then segmented from the other areas and there's careful movement when those systems are connected, we might see I've heard stories of engineers needing additional computational power. And so they install Algorithms on to the historian and then have the historian, which is in the enterprise area, then Send back information back to the controllers. So this would be that would be an example of a bad security practice. If that additional Those additional operations are needed, then that capability can be pulled on to An edge controller or something that sits inside of a more protected zone, but still allows for connectivity, but in a much more monitored and Process and a process that is more specific to controlling ports and protocols that would exist around that area. So this is, so this is kind of this this first and basic concept, the idea that If we're going to do good security in an industrial control context. The first step is to understand our process to understand the information that's required for that process to work and to understand the connectivity requirements. For that process, even if those connectivity requirements might deviate from the traditional Purdue recommendations. We need to start being able to as engineers communicate Engineer, so that we can start having this conversation based around what the business functions are So, Reed, do you want to add anything to that. As you transition to the next topic. No, I just think, you know, now that we've kind of looked at connectivity. It's good to keep the different levels of connectivity in mind as we talk about, you know, threats and what bad actors are actually doing against control systems. So I guess I'll just jump in and start talking about threats. So there have been a number of attacks over the last two years against control systems. Some have been pretty well publicized and some haven't. The first two on this list are are actually, you know, a lot of attacks against control systems. So there have been a number of attacks over the last two years against control systems. Some have been pretty well publicized and some haven't. The first two on this list are are actually just ransomware outbreaks that happened in two different manufacturing facilities. They cause a lot of downtime. For both of the manufacturers, which have like a real price tag associated with them. So those are, you know, some examples of attacks on industrial systems. And then we have the bottom two here a crash override and crisis attacks. Are more what I would call, you know, manual like manual threat operator people there are really trying to do something directly to the process control system. In the crash example, it might be a little bit hard to quantify a monetary loss to that. So the crash override attack was a Power outage that happened in Ukraine when a transmission operator was infected. The attackers wrote some custom payloads that would actually trip Circuit breakers inside of a transmission substation. But to that one, I mean, there is a slight loss of revenue there from, you know, transmission operator not pushing power anymore. But to me, the, you know, the actual loss there is sort of a loss in confidence in the utility and in the government. And then in the case of the Saudi Arabian oil and gas company that was affected by crisis. They had their plant trip offline by the safety controller malfunctioning, which was actually a piece of malware loaded into the safety controller. They can actually quantify the amount of loss there, which is $9 million per day. And I think that the two events that happened at that company, you know, not the plant offline for many, many days. So that money really starts to add up. And it's important when you start looking at those threats that we were just looking at right is to say You know, you're not going to patch your way out of those threats. I think it's a really common fallacy where folks say that, you know, hey, If I just apply all the security updates to all of my industrial control systems products. I'm going to, you know, get out of jail free, you know, WannaCry takes advantage of, you know, some unpatched Windows vulnerability. You know, and these other these other systems are these other pieces of malware may take advantage of actual vulnerabilities and software. But, you know, going down that kind of approach. I don't think it's good. And by the way, I'm a vulnerability analyst. Right. I love finding bugs and I love, you know, Making companies make patches for vulnerabilities. And even I say like that's not the right way to do it. You know, 10 years ago, if you would ask me, I would have said yes, that's great. Nowadays, not so much. But, you know, the if you actually blindly apply patches like we have in this comic, you can actually cause more problems than you solve. There were a couple of high profile vulnerabilities that have come out come out through the years. One that jumps to mind right away. Is this specter and meltdown processor bug that affected Intel. Right. So Microsoft came out with patches for Windows to help mitigate this processor bug. Well, it turns out, if you install that patch, you can break a really important part of a lot of process control systems called an OPC server. That will cause your HMI systems to no longer be able to communicate to all your field devices. So if you blindly apply that patch You're actually kind of doing the attackers job for them. Right. If you're actually causing your own downtime and costing yourself money. You definitely need some kind of development or test environment to to apply patches safely. But really, more importantly, if you should look at what the bad people are doing or what these You know, these threat actors are doing before deciding what patches to apply and deciding what you know mitigation steps to take So this is kind of a another sliding scale here we have, you know, a, a, a look at the various threats that that affect the industrial space. You know, these may range from script kitties or even automated tools that take advantage of sort of weak vulnerabilities, but they're probably not going to have such a high impact on you. All the way up to, you know, these kind of national state sponsored attacks that that go after you using like new vulnerabilities or new, new, new. Yes, bad, bad pieces of software. But a lot of these do have some commonality. There's a lot of techniques that attackers use to spread around your network, no matter what level of They're at, but this scale can kind of say, well, you know, as you progress through blocking these threats, you'll start by just Taking steps to to stop the lesser threats, you know, the vandals and and minor criminals and then as you step up your game, you'll be able to stop the more sophisticated sets. So in the threat analysis world. We use this thing called the diamond model to kind of define what our threats are right and the diamond model is this This idea that we have an adversary and we address. We never named the adversary like we don't believe in doing that kind of attribution. We think that's something that Really can only be done by governments, not by private companies. But, you know, it's taking a look at what infrastructure and and Tools and capabilities and adversary using and also looking at what victims, they're targeting Right. So some examples of activity groups that we see in the industrial space. This is one that we we call electron. These are the people that are responsible for the crash override back right We know that they are targeting electric utility companies in Ukraine. We know that in the example of that specific event where they caused the the Ukraine power outage. They were using a custom tool that's called crash override that, you know, manipulates and OPC server to trip circuit breakers. We also know that they were using a lot of SQL databases within the electric utility for actually spreading They were using a feature inside of Microsoft SQL database to, you know, remotely execute commands on other database servers and pivot around the network. And we also know a bit about their infrastructure, you know what IP addresses, they were using for command and control servers. So if you're looking at this as an engineer, you might say, well, I operate an electric utility. This is something that I'm interested in, right, because this is an activity group that's going after electric utilities. They wrote this payload that this electric utility specific So what on my network looks similar to what was on the Ukraine utilities network, right, do I use a lot of this infrastructure. So what on my network looks similar to what was on the Ukraine utilities network, right, do I use a lot of these SQL databases for pushing production data. Out of my, you know, energy management system or my my generation VCS do I do I use the same same Vaguely the same tools to get data out so that my, you know, chief financial officer can see how much power producing or shipping. But don't focus, especially on don't focus too much on the specific tool, but focus more on the, you know, what techniques. Am I using that are similar To the electric utility that could be used by the attacker. So maybe you're not using SQL databases to get data in and out of your network. Maybe you're using some Other historian system or something else. But, you know, that's the sort of system that you want to look at, though, is, you know, if that's how the attacker got into the network, they might use a similar technique to get into yours. So that's what you should be looking at. In the case of Genetime, they're another activity group that's active in this space. They are the people that are responsible for that Saudi Arabia gas plants outage. They wrote a custom payload called Trisys for infecting safety controllers. They wrote some, you know, custom credential harvesting tools. They also use a not open source, but at least a freely available Administrative tool. I believe it's called MedExec for spreading around the victim network. But this group was targeting oil and gas in the Middle East. So again, if you're an oil and gas provider, whether you're in the Middle East or elsewhere, it's still kind of an interesting group to look at You know, they had something really bad in mind by going after a safety control system. So even if you're not using the same kind of safety controller, that's something you might want to think about. Okay, there's an activity group targeting this sort of safety controller. We need to protect safety controllers on our network, you know, maybe make sure that we're keeping the key switches in the right position so that they can't be remotely programmed Maybe even isolating them from the rest of the DCS. Those sorts of decisions go into responding to that kind of threat. Yeah. Do you have anything to add to that, Ken? Yeah, so I think what's important is that the diamond model gives you a way to start to organize what threat information is relevant. So as we talked about, understanding the victimology of the adversaries, it allows us to start understanding whether or not we're exposed and going to be targeted in a similar way and understanding those tactics, techniques and capabilities then provides a foundation for what do we need to protect against and it provides us a way to defend Not just the networks, but defend our decisions of, you know, how we're prioritizing security across our networks to the plant manager to To put it to requirements for the vendors to Go up to the board or whoever is funding capital investment projects for a plant. So this third piece tries to tie those those pieces together. And so what we've done is the reason why we had those two kind of sliding scales was so that we could then juxtapose them together and kind of give you this sense for We're not trying to create like a maturity model. We're just trying to collect our thoughts so that we can What we wanted to do our primary target for this were engineers. So engineers that are designing building integrating Processes And we wanted them to be able to start having an intuition for some of the cyber security concepts so they can start engaging and Deeper more active more productive conversations with the OT security and IT security folks. That they might need to interact with at the plant. And so this is a way to kind of It's meant to kind of help tune the intuition of engineers. So the idea is that if you take your kind of Level of connectivity and remote control based on the requirements for moving and connecting across your network. And if you Place that on the same kind of plane against the increasing stealth organization and resources, the sophistication of adversaries. If you are, if you have similar Industry structure organization exposures as As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As And As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As As
