Transcript
Thank you for all that made it to this webinar. Here I am, Vivek Panada and Sandeep Lodha from Nozomi Networks. And here we are talking about endpoint security today. Sandeep, do you want to say hi as well? Absolutely. Thank you, Vivek. Good morning, good afternoon, and good evening, everybody around the world who's joined us today for our second episode of Security Sandbox. This one's going to be pretty exciting. We've got a good 30 minutes ahead of us talking about something very relevant, actually. And funnily enough, we couldn't have planned this any better. This was our scheduled topic coming out for the last month. And as you guys are aware, the topic for today is evolving endpoint security and OT. And just like that first word says, this is an evolution. We are seeing evolution happen right in front of us based on events that have happened in the last 7 to 10 days. So this, unfortunately, as tragic as the event that happened was for everyone in our space and all of our cyber responders, there is going to be a lot of good that's coming out of it. Evolution is good. Evolution helps us survive. So I'm pretty excited to talk today exactly about that evolving endpoint security and OT. Thanks, Vivek. Awesome. All right, so I think first things first, let's actually define what we mean by endpoints in OT, right? Because compared to IT, we have different technologies, different protocols, different equipment. So could you maybe chat about how we define endpoints in OT? Absolutely. Great question. And right away, this is where the differences start to occur. We're all very familiar with IT endpoints, computers, laptops, servers, printers, IP telephones, things of that nature. With these types of devices, generally, businesses can tolerate a little bit of an outage with them, right? They are critical for the function of the business. But let's be honest, if email goes down at a large company for a minute, the impact is almost negligible or immeasurable. Whereas if you're in a mine or a chemical factory, if some of those processes start to be disrupted for even a minute or even 10 seconds, there can be huge impact. So the different types of endpoints we typically run into in the OT world, first and foremost, PLCs, right? Programmable Logic Controllers. That's the heart of what operates industrial processes. Along with PLCs, we find in tandem a lot of HMIs, Human Machine Interfaces, that allow us to interact with the process. But beyond HMIs and PLCs, we run into things like RTACs, Real-Time Automation Controllers. We have IEDs. We've got a number of different devices that are very specialized. A lot of these devices work and operate on different operating systems than we're traditionally used to seeing in the IT world. And as such, we need different and new and evolving methods to protect these unique types of devices. Of course, we see servers and workstations, historians, engineering workstations, things of that nature. So there is some crossover between IT and OT. But for the most part, these are very specialized devices that communicate in their own specific languages that need to be dealt with in a very different manner than your traditional IT endpoint devices. I agree with you. Whenever there's a discussion about endpoints in OT, I typically tend to ask the question, what do you mean? Can you clarify or can you explain? Because endpoints in some conversations could be the HMIs and workstations that are typically just like what they're in IT, except that they have specialized engineering software. In other cases, it could be controllers, like you mentioned, PLCs, GCS, SCADA systems. So I think we just can't assume, especially in this evolution, like you mentioned, if someone is talking about endpoint security, I think it begs the question, can you clarify exactly what you mean? So that certainly makes sense. I guess digging in, you mentioned the word evolution. Could you maybe explain in your mind what led to the endpoint security solutions today? Because things are different today than they were a decade ago. So you've been in this journey for a while now. So what have you seen in this evolution that are key points to highlight? Yeah, great question. And evolution happens everywhere around us. So it's no surprise that it's now happening for OT endpoints. This is, again, another highlight of why you can't use IT tools to solve OT problems. Evolution is a good thing. Change is good. It fosters more efficiency. It allows for discovery of better technology. And it gives us better ways of solving problems. So, you know, 10 years ago when the company started, we very quickly became the leaders in network-based detection, which was fantastic. That's what we needed at the time. We didn't even have a tool that was able to provide visibility into network traffic for these kinds of devices. So as the industry has evolved, as technology has matured, as customers have evolved within their own environments, there now has been a need to understand assets and their behavior at a much lower level in the Purdue model. Right? Our network technology has been great at detecting things in 3.5, in 3, even in level 2. But as we start to get down, the blind spots are really around, you know, level 2 and below, level 1, level 0, in fact. Again, this goes back to the whole north-south and east-west mentality. North-south traffic paths, again, we've been the master at capturing that stuff for a number of years. But as we've become so dominant at understanding network-based traffic, it's also highlighted where we have some gaps that we need to fill in. And those gaps tend to be very down low, level 2, level 1, level 0. And a lot of times it simply is not feasible. We've reached that point in the evolution of network technology where it's not feasible to capture traffic that low down for a number of reasons. You know, just for starters, a lot of things like OEM vendor certifications, we're finding now more and more OEM vendors are not allowing things like SPAN ports from down low. So we've got to now look at another method to gather information. And we're getting great success with things like our ARC endpoint agent or smart polling. Point being, we've kind of exhausted the limits with which we can use passive technology to give us this information. So we had no choice but to evolve to continue to seek solutions to help our customers understand what they're operating at those low down levels. And of course, that's where smart polling came. And as an evolution of smart polling, we now have ARC, which you guys are all aware, ARC incorporates smart polling. So we really, when you talk about evolution, we went from crawl to walk to run in a very short period of time where ARC first released, we were able to gather information about the host it was installed on. Perfect. The second quick evolution for ARC was being able to passively discover neighboring devices within the same subnet. And now again, very quickly, we've evolved to giving ARC the capability to do active discovery. So as you can see, we're taking everything we've learned across the rest of the platform and portfolio and starting to incorporate it at the endpoint to exactly fill in those low level gaps that we've historically had challenges to visualize. Yeah, you make a good point that our company has been constantly on the innovation forefront, building stuff as the customers need and the problems evolve. It is fascinating to me. So as a PLC guy, I've always enjoyed these developments happening further down the Purdue model, right? So a few years ago, there were many people trying to do things at the level 3, 3.5 or the DMZ level. And now like at level two and level one, we have a range of solutions, right? So there are now firewalls that talk about blocking attacks at the Purdue level one PLC level, right? There are also other OEMs offering a cyber PLC, just like the concept of a safety PLC defaulting to a safe condition. There are now vendors offering a cyber PLC with known good firmware running some base level code so that your operation continues unabated, right? I guess that's the key. So maybe a good transition point is the operation and availability and reliability being the key metrics in our space. You know, what are your thoughts on how vendors need to adapt their solutions to what's important for operations, right? Very different from, you know, the CIA triad for information. Now we think about safety, reliability and availability in the OT world. So how do you think both vendors and end users need to think about OT security solutions as it pertains to the SRA triad in OT? Super question, Viv. And again, really relevant timing given recent market events. And again, we find ourselves revisiting the same topic. And that is you can't use IT tools to solve OT problems. This is a classic example of what happens with that. Specifically, from an OT endpoint perspective, there have to be some key differences between IT and OT tools. Number one, IT tools, particularly on the endpoint, tend to be heavyweight and they're disruptive, right? Two things that are not tolerated within OT environments. So, again, Nozomi Arc being able to be extremely lightweight provides kernel isolation. We don't touch the kernel, which is a big problem. You know, it has its pros and cons, but in an OT world where we want to err on the side of caution and safety, the more lightweight and the more non-disruptive, the better. Number two, IT tools are typically looking for the wrong threats, unfortunately. They're simply not tuned for the protocols, the languages, the behavior that we see in OT environments. And number third, very important, is OEM vendor certifications, right? There's a lot of things you can't do with your OT network because your OEM vendor simply won't allow that. It can cause problems and, in fact, go so far as to nullify warranty agreements. So these are just three off the top of my head that are very important that we need to look at when deciding how to approach OT endpoint security in OT. What about you? What are your thoughts on that, Viv? What are you seeing in your patch? When you mentioned the kernel versus user mode, I think that's very important in this world because as typical IT EDR tools tend to have device drivers, they have access to the kernel memory and, you know, whatever the reason might be, whether it's because of lack of validation, lack of testing, some other environmental situation, if the kernel needs to reboot, right? So that's a big problem in OT for availability. So I think one of the key aspects to really consider as an OT vendor and an OT security vendor is to ensure that you do not cause disruption, right? So whether it's focusing on user processes or validation and testing. So a lot of people underestimate the value of, you know, testing and validation in a representative environment, right? So in our traditional NERC SIP compliance for our power, there are standards set for how you validate in a, you know, fairly representative environment so that you test it first in a lab or a window where you evaluate if this patch or this code is working just as you expect. And then you expand to the other devices on your networking and production environment, right? That has become less of an opportunity in IT because speed is of the essence in IT, right? Where, you know, you're using a lot of behavioral analytics and trying to avoid a process from being run, trying to avoid from some code being executed because once it executes, you might not be able to stop it. I think in OT, we do have the advantage that we have other compensating controls. And so not everything, you know, whether it's log4j or any kind of latest and greatest updates don't necessarily need to be installed that moment, that instant right before something, you know, could happen. I think the validation and testing aspect is so critical for an end user, right? So it's one thing for the vendor to provide a patch, which obviously they've done some level of validation and testing on their environment, but it's for the end user to then figure out, you know, how applicable it is for their environment, right? And part of what we do as a company is to give insights to our customers as to what patch might be relevant for them. And that's true of our own products as well, right? Absolutely. We do have a question that came in from the audience, but before I hit you with that question, Viv, I'm going to just touch on something that you've heard me harp on for years. And this brings me back to my network engineering network architecture days. I'm always a fan of N-1, right? There's a reason why you don't adopt the latest and greatest code for whatever reason, no matter how great any vendor is. You still have to take caution. You've heard me for years harping about test environments, pre-prod environments, right? When we set up customers with a solution for a platform, it's extremely important. And this highlights some of those reasons why you need a pre-production environment to be able to test and to be able to stage things. There is no predictability. The only thing predictable is unpredictability in this space. How many times have people been burned by patch Tuesday, right? That's a very IT thing. We don't see that at all in our world. We're lucky if we get annual patching in some of our environments. So, again, this pre-testing, staging, pre-production thing becomes even more relevant, especially when we talk about critical infrastructure and the processes that run behind that. So let me quickly hit a question here for you, Viv. We have someone who's asked, what is the difference in architecture and process for endpoint signature updates? Example, air gap or data diode isolated endpoints versus traditional IT endpoints. For example, less often and hence less impactful updates. Interesting question. Do you want to take that? Yeah, there are many layers to this question, right? So, obviously, this person has a significant OT background, and they're saying what's the difference in approaching OT solutions that could be air gap potentially or might have isolation using data diodes? How often or what kind of approach would you use for an update there versus IT? So the bottom line is in IT, let's say in level 4, level 5, if you have a device that's 24-7 connected to the Internet, of course you want that to be updated as quickly as possible, right? So N-1 might not be an option there because the risk of a connected device getting impacted and then becoming a host for something else might be very, very high. But to Sandeep's point, N-1 or even N-2 is a very OT-centric approach that works really well, right? A lot of OT systems are not patched, right? Not regularly anyway. So the best case might be once a month. The usual case might be once every six months or annually if at all possible. So if you're not patching it, there are so many vulnerabilities already associated with it. Just updating signatures on an endpoint detection solution might not be cutting it in the first place, right, because there are vulnerabilities inherent in those systems that are not patched. And the engineering software, engineering version control might also have other vulnerabilities. And as we know, OT protocols are not typically secure. So even on a fully patched endpoint system, you could run code on the workstation software that could impact your process. And this has nothing to do with the vulnerabilities or the EDR itself, right? So the difference in architecture is evident, right? And what's applicable for your environment is dependent on a lot of factors. So not every critical process needs to be isolated or air-gapped or even data-dyed because, you know, there's cost impact, right? So there's management, there's overall cost and overall technical debt that needs to be accounted for when you figure out what the architecture is. And the management of that going forward also depends on how many resources you have and what your tolerance is. So in my mind, there is no easy answer because it depends on all these factors, right? But once you have a process, the key is to follow that process. So if you ever bypass something just because that was the easiest thing to do, right, so you had the N minus one schedule because you had some validation requirements, but then you bypassed it because you figured this was such an important update that you had to do today, right? Or because this was your only opportunity to download and you did it. The next thing you know, you're in a bigger trouble next week. So I think the key is to evaluate all these different considerations up front for the architecture, but going forward ensuring that it's applied for consistently. Your thoughts? Absolutely, absolutely. No, great point. And super question. Again, multiple levels on this question. First and foremost, the thing I want to point out is that Nozomi Arc is not like a traditional EDR. We're not an EDR. We don't need signature updates. We don't rely on threat signatures. That's not what we're looking for. Again, we don't have or want kernel level access. We're looking for threats in a very different manner than some of the other EDR organizations out there, right? Keep in mind, Nozomi Arc, our primary use cases are number one, gathering asset inventory of the host it's installed on, right? People are coming to us saying, hey, we can't tell what's on our network. We're not able to get it passively. So I need something that's going to help me with asset inventory, right? At that point, that has nothing to do with threat detection yet. They just want to know what's out there on the network. So Arc is bringing asset inventory, asset visibility. Secondly, we talked about this passive discovery of devices around it, right? I've got an isolated subnet. I've got an isolated situation where it's not cost effective to put a guardian or it's not technologically feasible to put a guardian sensor. So we turn to Arc again to do that neighboring discovery. When that passive discovery becomes exhausted, we can turn on active discovery from Arc. So, again, these have nothing to do with signatures still. This is all discovery. When it comes to threat detection and prevention, we are looking at things like USB activity. We have the ability to monitor USB ports in real time and provide alerts upstream so you know when things are happening at the USB level. On top of that, we are able to spot if malware is transferred from a USB. So, again, we're operating outside the bounds of the kernel. We're still looking at peripheral devices, right? So that's huge, being able to understand USB activity. That's another big driver for this. At the end of the day, Arc and OT endpoints, they're not a leading solution. You're not going to start your OT security journey with an endpoint agent. It's generally the last step down to fill in the last bits of gaps that you've got in the network. Again, for those that are familiar with our platform, deploying a guardian sensor in passive mode right away is going to give us between 70% and 80% visibility to those assets. I identify where my gaps are, and we can augment them with things like a remote collector. That will give us another 5% or 10%. If I've still got remaining devices that are not visualized, then I can turn on things like smart polling and move into targeted active discovery. That's going to fill in another 5% or so for me. So now at this point, we're at 95% visibility. So you can see the available footprint left for something to discover that remaining 5%. That is where we would turn to Arc. Again, super important to understand that it is not an EDR. It does not require daily, weekly, monthly updates even, right? Of course, we evolved the product, and we put out updates to the software, but those are not signature updates. Those are generally feature enhancements or new things that we're bringing into the product from a customer-driven request perspective. So we can continue to operate just like many customers of Nozomi do today in air gap environments where they don't need to update the firmware running on the box, right? It's doing exactly what they need it to do, and if it ain't broke, don't touch it. Don't fix it. There's no need. Rule number one in IT, and we all like to tinker. That's our problem. We love getting carried away and tinkering and breaking things and fixing them again. That's a very IT methodology to doing things. So OT is very, very differently. It's the ultimate soft touch environment. So again, I hope that helps a little bit. Irrespective if we're sitting behind data diodes or air gap environments, it doesn't matter. We're still going to provide that asset visibility, asset identification, USB notification, things like user activity correlation, which is huge. For a decade, we've been the master at telling you that, hey, 10.1.1.1 is uploading a program file to PLC 10.2.2.2. Now, with Arc, we can layer that in and say, hey, that's Vivek, who's logged in to this engineering workstation, who just uploaded a program file to this PLC. So bringing in that user awareness. And, of course, last but not least, very important, is our Sigma rule engine. So we are able to look through the log files on Windows machines. The security log, the application log, the system log, and we are able to spot abnormalities or changes before a packet even hits the wire. Keep in mind, Guardian needs to see a packet on the wire to determine if something has changed or not. With Arc, we're able to look at those log files, and we can recognize changes happening to those machines before they even drop a packet on the network. So it's a very different approach to understanding endpoint security. Not at all to be confused with EDR. Very complementary products. They do very different things than what we're trying to achieve with Xomi Arc. Yeah, I don't think I can stress enough about the differences, right, because what you're describing is how an OT-specific solution is built in the first place, right? So instead of saying, hey, I got an EDR solution, how can I expand into OT, we think about what specifically are the OT users trying to secure against? Like, what are their problems? What are they trying to solve, right? And if the question is they have issues with USB keys being used and they want to be able to track, okay, that's a use case. Let's build a solution around that, right? Second is they have access issues, can't get network traffic out of there, but they still need to discover access in the area. Let's build a solution around that, right? Similarly, somebody might be maliciously manipulating their log files or file integrity is a key issue, so let's build a solution around that. I think that's maybe the question that our audience and, in general, most end users need to ask in that, what problems are you trying to solve? As opposed to, yeah, we understand how EDR works, we understand what it does in the IT environment, and as our friends at Mandy and say, you know, Theory of 99, the vast majority of the attacks come from the IT world, so your IT endpoints need to be secured. I don't know of a single security practitioner that would recommend against a traditional EDR in IT environments. However, is that applicable in OT? Most likely not, right? I have no problem saying depending on your use cases, you might not want an EDR solution that especially has liability issues, right? So you want to have a solution that solves your key aspects, right, whether it's securing your endpoint from USB attacks or maybe some other aspects that we already discussed, but the key is to ask, what problems are we solving by installing this piece of code in either the workstation or in the PLC? Now, talking about PLC, that's the most exciting new announcement recently. We have Arc embedded. That's good for level one, and that's pretty unique in this space, right? We haven't seen anyone else consider this in the past. We have seen some folks trying to manipulate the PLC code itself, you know, create some kind of monitoring secure solutions within the PLC environment. But ours is very light touch, like you were describing before. It's installed in the Linux environment, and it can detect changes in the PLC. So do you want to maybe talk a little bit more about how Arc embedded is going to likely revolutionize the aspect of level one security and monitoring? For sure. You took the words right out of my mouth. I was going to say with a couple of minutes left, let's talk about the elephant in the room. And first off, a huge thank you to the folks at Mitsubishi Electric who approached us to develop a security solution to sit inside their PLCs. It wasn't us approaching them. They actually came to us with a very unique offering and asked for our help to collaborate and build out a security solution that could embed on a PLC. And like you said, this is another industry first for Nozomi, yet another. And it represents the need for the industry wanting to get closer to level one, level zero. Again, Arc is fantastic, but it's limited to Windows, Mac, and Linux. So I can get down into level two. There's HMIs there. There's occasionally some servers, some Windows boxes. We have the ability to install there. But again, we're still left with level one, level zero. So Arc embedded, as we're calling it, and this first generation supports Mitsubishi Mailsoft devices. By being able to install on them, we're actually injecting ourselves right into the middle of that east-west traffic flow. So now, irrespective of what the network tools, if there are any, are pulling off the network and analyzing, we can analyze that right at the host level. We can see traffic. We can see communications. We can understand flows. We can understand protocols. Of course, we're going to be able to spot PLC-level changes, code changes, register changes, function codes, program uploads and downloads, things of that nature. We're going to be able to spot without having to have a dependence on the network now. So this is critical. This is what we've been searching for. We've been trying to get lower down the stack. And again, it started great with network. Our network tools got us to around level two, 2.5. Of course, adding things like smart polling and ARC allow us to get a little bit more. But to get into that level one, we've never been able to penetrate that. The rules don't exist. You're not allowed to start smart polling things down from zones. Cost-wise, it's not efficient to start dropping appliances into every zone. So our customers turned to us and asked us to help them develop a solution to provide the asset information that they're requiring from such a low down spot in the Purdue model. And of course, embedded ARC fits exactly that need. And again, just like everything else in the Nozomi family, I expect rapid evolution of the platform. If we look back at Vantage just a short time ago, the evolution that Vantage took from when we released it to where it is now is multiple generations of features being built in at an accelerated pace. The same thing happened with ARC. Started off day one. We could get host-level activities. Then we expanded into passive detection. Now we've expanded into active detection. So now with ARC embedded, this represents a whole new journey within this evolution for us to get even lower down to provide more accurate visibility to help protect these critical networks even further. Yeah. And I'm super excited. I can imagine a current customer leveraging ARC embedded at the level one, ARC at the workstation HMI level, maybe a container in their Cisco or RuggedCom infrastructure and connect to Vantage. Overall, you know, with no footprint of physical appliances anywhere, right, this is an amazing end-to-end solution from level one all the way connected to the cloud as needed for signature updates and functionality. So I think it's an amazing journey to be on. It's kind of fun, and it's really interesting that now that our solution is available all the way at Purdue level one. So exciting to talk about this stuff and sharing with you all, folks. I really appreciate everyone that was able to join us. Any closing thoughts? Yeah, thank you for your time again, guys. I love these little 30-minute bites we do once a month. The last thing I'm going to do, if it hasn't already been done, is I'm putting a post into the chat here. This is a blog post that our amazing technical writer, Michelle, put out just yesterday, and it talks a lot about what Vivek and I just reviewed. So if anyone is interested to learn a little bit more or just get a review of what we just talked about, please make sure you click on that link and feed yourself all the information you need. But thank you again, folks, from all around the world. It's been a pleasure, and Vivek and I are looking forward to seeing you next month. Thank you.
