Transcript
senior advisor for grid security at Southern California Edison. How are you? I'm good, Mike. Thank you for having me. You bet. Thanks for joining me. Great session this morning. Thank you. A little grim at points, but I mean, like it was, it's necessary though. I mean, I think it's important. Yeah. And before we get a lot farther into this, let me just say that any opinions expressed are my own and not those of Southern California Edison's or their parent companies and so on and so forth. Disclaimer. Disclaimer. Yeah. I can't endorse any particular brands or companies or anything like that. So on and so forth. There you go. Perfect. Cool. All right. So, I mean, we should talk about just kind of the session from a higher level. It was basically how an attacker could compromise a hyper-connected grid. Yeah. Taking advantage of IOT devices that are centrally managed. Not really the status quo right now, right? I think that was a big point of your talk. Yeah. So it's really using smart meters, often referred to as AMI 2.0, that will have customer facing Wi-Fi. So they'll join the Wi-Fi that's already existing at a customer's place to be able to issue command and control to IOT. It doesn't exist today. I'm hopeful that we'll find a different way to do it and it won't ever exist. Because that level of centralized control of that many vulnerable devices keeps me up at night. Yeah. I mean, you're opening up so many exposures once everything is enrolled and managed centrally, etc. Yeah. I mean, it's when you look at the potential of a metropolitan area, having millions of devices enrolled into control, centralized control at one single point, that's then backed up by software that we know to be vulnerable with a whole bunch of ways into it. Sure. It starts to add up to something that's pretty concerning. Right. And yeah, I think that there are better ways that we can go about it. And I'm really hopeful that we'll, as an industry, kind of select those directions. Yeah. I mean, and from an attacker's point of view, whether it's a nation state or KD, I mean, the opportunity for an attack at scale at that point is really kind of concerning, obviously. Yeah. Well, even if you just look at something, so not burning a whole city down, but if you just look at something like, you know, creating a giant botnet, you have what would, in Los Angeles, we're looking at like five to six million meters, right? But if you look at countrywide and all of the different utilities that are rolling out AMI 2.0 at some point in the future, you're talking a hundred million meters. Right. And they're all connected to the internet. They have to last for 20 years. And there's no way we're going to be able to secure them that long. So at some point, someone has a hundred million advanced compute edge device botnet. That's best case scenario. I mean, obviously best case is that nothing happens, but I think we've learned that something always happens. Something always happens. Yeah. And I mean, you know, you mentioned burning down cities. We have this tragedy in LA that's really fresh on everyone's minds. It wasn't a cyber attack, but it kind of gives you an idea of just the potential in terms of, you know, structure fires and capability to put those out and defend. Yeah, that's what we really... I mean, it changed the tone of my talk, obviously. When I pitched it, Los Angeles was not on fire. Right. When I gave Dale the final presentation, Los Angeles was very much on fire. So yeah, it was eye-opening. It did actually change our mathematical models, which I mentioned in the presentation. We didn't realize how quickly fire departments could be overwhelmed. We knew it was bad, but I didn't realize it was that bad. And we based our math off of some publicly available research, but then we saw a real-world example. We changed it. Yeah. It's two to three engines if you have the staff to support them per fire, and that's all you can do. And that's assuming you have the engines to pump the water. And if you start opening all the fire hydrants, you run out of water pressure really quick. Right. When you're trying to fight fires across a large distributed area. And that's the scenario we're looking at. Right. Yeah, I mean, you could be talking thousands of fires, tens of thousands of structure fires, and very limited response capabilities, even in a large city with a lot of resources. Yeah, I mean, it's .005% of structures catch fire, and that overwhelms a fire department. The Fort Worth example, including Dallas was, what was it, like 45 fires that they could fight simultaneously, if I'm remembering my own presentation correctly. And our attack causes 3,000 plus. Yeah, I mean, it is concerning how quickly fire departments can be over capacity, but it's not the fire department. I mean, it would be impossible. There's not enough fire engines in existence to fight fires caused at this scale. So it's not their fault or anything like that. It's just the nature of the beast. But we really have to think about, how do we prevent this? Because we can't fight it. So take me through what an attack might look like, because you mentioned a lot of different stuff, the meters, smart devices within the home, everything's going to have a Wi-Fi connection eventually for a lot of different reasons. I mean, if you want to start there with kind of like the resource planning on the, you know, for an energy company or something like that. Why in the world are we doing this? Yeah, yeah, so I mean, in order to support the grid of the future, both from a demand side and from the distributed energy side, where we're changing the model, we really kind of have to move to a place where we can both forecast at a granular level and get response, right? So demand response, we need to change your thermostat by a few degrees, right? And obviously customers enroll in this, it's not forced on them or anything, but they get incentives, they enroll, and the utility has the ability to make small adjustments. And that exists today through aggregators, right? This is a more centralized, every device in theory is enrolled into direct command from the utility. The aggregator is no longer in the middle for that command. And that gives a single point of attack to every device, every enrolled device. And that's really the concerning part is that if an attacker were to get into those systems, which, as I talked about in the presentation, aren't as secure as they used to be, right? They're not isolated like they used to be. We have a lot of connections into them, but the systems are still legacy and still pretty vulnerable. If an attacker were to get in there, they could issue commands to every single device in the network, every enrolled device. And when we start to model that, it starts to get pretty concerning. Right. Yeah. And then the vulnerabilities are there. I mean, exploitable commodity stuff. We're not talking expensive zero days. There was no zero days in our research for this. Yeah. So setting a smart ovens heating element effectively to infinite is an existing vulnerability that is out there. It's not something that we created or dreamed up. It's research that we looked at other people. And we did do some examining of smart devices ourselves. We did a little bit of looking at the firmware on them and that kind of stuff and how easy it is to change to kind of validate the research that we saw because we frankly were getting a little worried about what our numbers were starting to show us, what our model started to look like. Sure. When we saw 20% saturation of vulnerable devices in a metropolitan area, well, that's a lot of structures. And even at the 5% success rate that I did in the presentation, which was probably pretty low based on the research that we saw, it's a catastrophic event. Yeah. Yeah. So an attacker who has compromised the command and control is it has the ability to send commands. What can they do? What does it look like? Sure. Yeah. So, I mean, it can be something as simple as they could command off every meter and cause a blackout, for example. It's a relatively minor issue because we could get it back. It's a black start event, but it's not something we couldn't recover from. Right. That's your base level kind of not physically damaging attack. But if they start issuing commands to, let's say every vulnerable space heater in the network, right? So that's probably gonna be several hundred or possibly a few thousand fires within a metropolitan area. Again, most fire departments are able to fight, you know, a few dozen. Right. So they get overwhelmed pretty quickly. If they're able to do that at the same time that there's high winds, that fire becomes uncontrollable. Mm-hmm. Yeah. And what about existing controls on the devices or on the networks? Why aren't they good enough? Why aren't they work? Why wouldn't they work? Sure. So when we look at like IOT devices, it's pretty well known how vulnerable they are. There's a lot of available information on it. Why they're made that way? I mean, I would love the manufacturers to put a little, why don't you need to authenticate before you load new firmware onto them, for example? Like that's just, that's crazy. You don't even have to have signed firmware. You can just push anything you want onto a lot of these devices. In fact, I think Dragos just published one on a space heater, like a week or two ago. Really interesting read. I would definitely recommend people go look at that. But yeah, it's, I've wandered from your question, but it's interesting to me that there is not better security controls. But on the flip side, there's not demand for better. I mean, people like me are screaming it from the rooftops, but I'm not your average consumer. Your average consumer just wants the thing to work. They want their smart kettle to turn on when they pull their phone out of their pocket and say, heat it to 185 degrees. And it's cool, it works. It's just terrifying when you look at the flip side of that, that that smart kettle could catch fire. Right. Yeah. And they're much less likely than ovens and toaster ovens and things with heating elements, right? That's where we tend to see fires, but it can still be pretty inconvenient when your smart kettle blows up on your counter. Did you mention during your talk that some of the physical or mechanical controls and safety mechanisms in ovens, for example, may have been removed or? Yeah, well, I think it's more that they, yes, yes is the short answer. The long answer is I think they weren't needed anymore in the view of the manufacturer, right? So when they moved to smart controls and there's now a control board with firmware on it that's operating a relay that turns the heating element on and off, they removed the need for, you know, resistor or thermistors, if I'm saying that one right, in there and other things that, you know, would have disconnected the heating element when it got too hot. Where we see that in mechanically controlled ovens, that's literally how they work. You are adjusting what the temperature is and it's just cycling the element on and off based on that, off that temperature. Where if you interrupt the control loop on the smart ovens, it doesn't care what the temperature is. Right. The element's just on until something happens. Yeah. And you're surrounded by cabinets and chairs and tables. Yes. And it becomes tinder suddenly. Yes, yeah, exactly. We surround our ovens with wood, which my oven is surrounded with very beautiful oak wood at my house. It looks really nice. I can't knock it. But at some point of the outside, the temperature of the oven reaches the auto-ignition temperature of the wood, bad things happen. Right. This whole scenario really illustrates as well that like, you just, eventually nothing is going to be isolated. We've already gone through this in OT where, you know, the air gaps are a myth. You can't isolate this stuff anymore. This really kind of puts a real world face on that whole idea. Yeah, well, the whole concept of isolation to begin with is always been a false sense of security. And I think we've seen that proven time and time again where isolated air quotes networks get compromised. Sure. I mean, we've seen some of the securest networks that we thought of on the planet get compromised. Those, you know, inside of our own government, for example. I don't think that anyone can rely on the idea of isolation or even security controls as simple as like a firewall, right? So if the meters are connected to the internet, they have a software-based firewall on them. That might provide some level of protection, but I don't think that that's something we can rely on. Right. Yeah. Particularly with millions of them out there all with the same firewall. So how far away are we from this kind of centralized command and control and all these devices getting enrolled, the whole scenario that you painted? So I'm not an expert on that. But what I'm being told is that probably within the next few years, we would begin the rollout of these new meters. So let's say, you know, maybe we're utilities. We move slowly. So when I say a few years, maybe it's closer to five, maybe a little bit more, but it's not terribly far off in the future, but it will take us a long time to roll out the meters. Again, Los Angeles, we're talking between five and 6 million. Sure. And you have to physically change them out on the sides of people's houses or businesses or whatever it might be. So there's a lot of work involved. So it's going to be a slow rollout probably over a decade or more. And then customers have to be incentivized to enroll in the program. So I think we're still pretty far away, which is I'm trying to get ahead of this now. Yeah. So that we don't design the next generation of meters with customer facing Wi-Fi. Or if we do, it's extremely protected and it's not used for stuff like command and control. It's used for like, we can provide information to the customer. Although again, I would argue that that could be done better through the cloud. Yeah. And I mean, you made a great point too during the talk about like, you know, we're putting advanced computing onto the meters and it's something that you're going to have to maintain for a long time. Yes. Yeah. So the lifespan of a meter is roughly 20 years. From an advanced compute perspective, that's unusual is a good word for it. We typically have a lifespan of three to five years for most more advanced compute environments. In the OT space, when we look at like the equipment that we have out in substations, for example, they last a pretty long time, but they're not connected to the world. They're not there. You know, they're serving a individual purpose. They generally don't have a whole lot of compute environment. Even the ones that do are pretty locked down on what they can be used for. We're looking at meters that, you know, are supposed to be future proof. So we can load whatever application onto that compute environment that we want, for example. And obviously, you know, manufacturers will build some level of protections in, but you just simply can't build security into a device you're making today for 20 years from now. I mean, think about 20 years in the past, all of those physical devices that, you know, if they're even out there, most of them are gone, are highly vulnerable. Breaking into an access point from 20 years ago is pretty trivial at this point. They don't support any of the modern security standards. They really can't. They don't have the compute resources to do it. And it is kind of interesting to think that we want to place an advanced compute environment out there for a 20-year lifespan and that we think we can maintain any semblance of security on them for that longer period. I mean, that was kind of my next question is like these devices, can they, with their form factor, can they support security like authentication? Can the encryption, whatever, you know, is viable? Yeah, and certainly like today, right? But if we look at encryption of 20 years ago, same challenge, right? It's not what we consider to be secure encryption today. Most of them have been compromised. Most of our encryption standards from 20 years ago. And same with authentication, which is closely related. But yeah, it's, we can put the best of today on them because they're new, right? When they're designed, we could put the appropriate compute resources in there, the ability to handle modern encryption. But I don't think we can even forecast what encryption will look like 20 years from now. Sure. And certainly we can't future-proof for it. Mm-hmm. Yeah. I mean, do we need Wi-Fi on eaters? Yeah, I would argue that we do not. You know, in the presentation there, I presented an alternative in which we could do the same level of control at the individual appliance level that meets that demand response need, but through the cloud, using APIs from the manufacturers or using hubs or in applications to issue commands to the devices, but not directly and not through infrastructure meant to last 20 years. Stuff that we can keep modern and we can have better security controls on. And that's what I'm advocating for today. And I'm definitely open if people come up with better, maybe from the talk, someone will come up with a better security option for this going forward than I've thought of. That would be great. You know, something that people can really buy onto and that we have a lot of good confidence in the security long-term for. Sure. But short of that, yeah, I think that a cloud option is a really good way. We're not trying to backhaul through our SCADA network and our meter network, a bunch of commands out to devices. It's all going out to cloud-based systems that we can have modern security on, that we can keep modern security on for forever because they're always, they're ever new. And yeah, and then that gives us that ability to meet the need, but without the risks associated with trying to backhaul it all through our relatively vulnerable legacy networks and then through hardware that ages. But if devil's advocate, if these devices are talking to the cloud or an API, there's still a connection there. Is it less exploitable? Yeah, so the IOT devices are already connected. So the meter wouldn't be connected at all anymore. We would get rid of the customer-facing Wi-Fi on the meter. It would have its wireless network on the backend for our control, but that's a locked down network that people shouldn't be able to get into, at least without breaching a lot of other security controls first. And that would allow us to keep that security modern. Now, the IOT devices are still going to be vulnerable. They're vulnerable today. They're going to be vulnerable in the future, but they're already connected. Our centralized control could still be compromised, right? Everything's compromisable. There's nothing preventing that. Well, there's a lot of things preventing that, but there's nothing foolproof that we can guarantee it won't be compromised. But if we're sending all of those commands, instead of backhauling through an old-school SCADA network, and then over a mesh network to the meters, and then out to the devices, if we're taking it out to the cloud, instead we can have modern security tools doing analysis of all those commands. Things like, oh, we just issued a command to turn all ovens on to infinite is going to be a little bit alarming. And while we might be able to see that on the backhaul as well, it's certainly a lot easier for us to build that security at the cloud level. And then if we take it another step where our cloud is communicating through APIs to the manufacturer's clouds that are then communicating to the devices, we don't have that direct control of the device anymore, and the manufacturers can build in whatever additional security on top of what we've built for their own products. I'm not saying they would, but they would have the opportunity. I mean, what would it take for that to happen? I mean, this is obviously a very reactive industry. Yeah. I think that it's the direction we're starting to head in. There's a bit of a challenge, I think, in just trying to get everyone on the same page. Obviously, no one wants cities to burn down, right? No one wants that. I think that should be self-apparent. Yeah. But I'm not sure everyone understands how real the risk is because they don't necessarily understand how vulnerable the IoT devices are, and they don't necessarily see the alternative paths as being as viable, right? Because they want granular control on the millisecond level. So if we're stepping through all these different clouds, are we achieving that, or is there too much latency? Right. And the need isn't there for that yet, but it's forecasted that that's the kind of level of control we will need at some point, or at least that's what they're telling me. So we just have to, I think, prove that the alternative solutions are not only more secure, they're likely better for the use cases in which the utilities need them for, for the demand response and the forecasting and all of that, and that it will give them an environment that has the response time they need, but also allows them to deploy new applications if they need a lot easier than they would be able to in the aging infrastructure of meters. Right, makes sense. Probably should ask this when we were talking about the attack, but like how targeted can you get? And is that the more likely scenario than a very noisy scale attack? Yeah, so as long as someone has enrolled devices, you could target any individual, right? Either through compromising their meter directly via the Wi-Fi, the connection that's on the meter, or through that centralized control. Now, if you compromise the centralized control, you're gonna also have to get ahold of a database to be able to determine, you know, which customer number is the customer that you're targeting. But once you have that, you'd be able to see the devices that are enrolled and issue commands to them. So yeah, targeted attacks could certainly happen. Yeah. So what's your plea to your peers, to the manufacturers, the vendors, et cetera? Let's find a better way. That's really what my focus is. Like I understand the need. At first I didn't, to be honest, when people told me that, hey, we need to be able to issue commands to all these IoT devices. I was like, that's madness. No, we're not doing that. And then I was informed that, well, yes, we are. And here's why. And it's legitimate, it makes sense. I understand the need. And I just want us to find a way that does it without me laying awake at night, wondering if we're going to burn a city to the ground. I hope you're not lying awake every night. Not every night, no. But it's a scary scenario. And that's the main thing is I want to prevent that. I think we have better ways to do it where we can still have that level of control, which the control in itself is terrifying, but let's build as much security around it as we absolutely can. Right, right. So just kind of as a last question, when I meet someone that's like so in the weeds in OT security, I always like to hear about your career path because I mean, I don't think anybody grows up wanting to be an OT cybersecurity lead, but it's definitely happening. Yeah, okay. So I started as a controls engineer, classically trained, got a couple of engineering degrees, went into controls, did a couple of different things, plywood manufacturing, became a consultant and did a lot of, a whole bunch of different kinds of stuff, food processing, pharmaceuticals, paper products, all sorts of things. And the need started to arise for security. And this was probably like 2012 or so, somewhere in there. You know, there was a movement towards securing the OT space around that time, very similar to how there was in IT, what, 20, 30 years ago now. And people were like, well, who can do this? And long before I got into engineering, I grew up as a hacker, basically. I was constantly pulling things apart. I was breaking into computer systems I probably shouldn't have. Long beyond the statute of limitations. Don't tell those folks. Yeah. I mean, it was all harmless kid stuff, but yeah. I was interested in the space. So when the demand came out, people like, we need someone to secure this stuff. And we were already in there. We were building a new control system for them. I'm like, well, let's, we're putting a new control system in. Let's make it secure from the start. Sure. Instead of trying to slap it on afterwards. Later, I found out that that's like a big deal and that's what everyone needs to be doing. But that's how I, that was the very infancy of me getting into OT security. And it really grew from there. What narrowed me down to the electric utility space is the realization that everything is dependent on power. We get home and we flip our light switch and our lights come on and we don't really think about it, but it's, if you don't even consider security, it's practically a miracle that that actually works. When you look at our aging infrastructure and everything that's there and how we get power from point A to point B, it's marvelous and it's delicate. And now that we've introduced a lot of security concerns, I became personally very worried about, well, will my power stay on? Right. And I wanted to do something about it. And that's been a passion that's driven me for over a decade now in the electric utility space to do my best to ensure that when people get home and they flip that light switch, they get their power. Yeah, I mean, that's the thing. People just want their stuff to work. Yeah. We have to figure out how to fix it and secure it. Yeah, ensure it keeps working. Yeah, and that's the goal. All right, great. Brian, great to meet you. Thanks for coming on. Yeah, good to meet you as well, Mike, thank you. All right, bye, bye.
