Transcript
how long would it take for us to recover in a cyber event? I don't know. It's going to take us a day to figure out or two to figure out what was even impacted necessarily, right? Like we'll have the initial things that we know for sure that are down, but what else is there, right? What other things are impacted? It's not as what we typically, you know, thought of in a DR type event. It's just that fog of war understanding. Welcome to Building Cyber Resilience in Healthcare. The attackers are learning from each other and we should be too before everything's on the line. Here, healthcare leaders share the stories, insights and lessons that get hospitals operational again faster and patient care restored. I'm Josh, your host. And with that, let's get started. Hi, welcome back to Building Cyber Resilience in Healthcare. I'm Josh Howell, Rubrics Healthcare CTO. And I'm really excited about today's guests. Christian Lindmark is the Vice President and Chief Technology Officer at Stanford Healthcare and School of Medicine. And today we're gonna get into an interesting discussion about what an organization's contingency plan should be, which use cases deserve the highest consideration and how organizations can build an isolated recovery environment within the budgetary footprint that they're already spending. Let's jump in. Christian, thank you so much for joining me this morning. I've actually been really looking forward to this discussion because of how it came about. So for those of you who are joining us, Christian and I met at a conference. I had had this apocryphal idea that everybody at Rubric was like, eh, I don't know. Like, that's a little weird. But I threw something out and you were the first other person I've ever heard talk about it. But before we jump into that, maybe you could just give us a little bit of your background, your history, and then I wanna hear about First T. Would that work? Yeah, great, okay. Yeah, thanks, Josh. Yeah, so my name, Christian Lindmark. I'm the Chief Technology Officer of Stanford Healthcare and the Stanford School of Medicine. Been here a little over 10 years now. Spent 25 plus years in healthcare IT though on the consulting side and then working at NYU Langone before I was at Stanford. So really loved the mission of healthcare and what we do and how we serve our communities. And Stanford's just been a wonderful organization to work for. In terms of First T, so I love to golf. And so I found an opportunity to be able to give back in that community through the First T. So we have a local chapter here in the South Sound. I actually live up in the Seattle area, so South Puget Sound. And I'm on the board for that. And our goal is to raise money and to give back to the community and teach to children the skills of golf and how that corresponds to life. And as we all know, golf can be very frustrating and tedious but it provides a great opportunity for kids in situations to learn some great things about life. And it's been just really rewarding to be part of that, to give back to the community and see the impact that we've had throughout the South Sound. And it is really cool that it's a national organization too. And so I've had a chance to meet numerous people around the country as I've traveled from other chapters and organizations. And it's a really tight-knit community. So it's been a lot of fun. So if I had learned as a child not to lose my temper and throw my golf clubs or whatever it is when you get frustrated, maybe I would be more measured as an adult. Yeah, that's some of the hope and the goal out of it. Absolutely. But it's just another avenue for kids. A lot of them come from difficult environments and this provides an outlet for them. They get mentors, they get coaches and just provides a place for them to maybe get out of the difficult situations that they're involved in in life and teach some valuable life skills along the way. Love it. The more we can invest in helping kids have life skills early, I think it just makes everything else. I actually have people here painting my house today because we're getting ready to sell it. And there's a spot where I gave my daughters hammers when they were like seven, eight and I was like, I shall hole in the wall. And they lit up and they're like, really? I was like, yeah, today you're learning how to repair drywall, right? And I was going over it this morning, getting ready for the painters and having a chuckle over that moment. I still have pictures of them gleefully smashing a hole in the wall and then me showing them like, this is what it takes to repair drywall. So, be gentle. That's awesome. That's my own life skills. So you and I first met in person at this conference last year and I have three to four meetings a week with different health systems who are trying to figure out how to build cyber resilience and approach it in a practical way. And one of the things that I've realized is just the intense competition for every dollar, every hour, every organization has scarce calories the way I think of it. And so we have to be really strategic where we allocate those calories to. And going and getting a budget for building some sort of isolated recovery environment or IRE on-premises means yet another trip to the board for a large amount of capital. And so I had had this really weird idea and I actually talked about it internally and the team that I'm on, everybody was kind of like, that's heresy. Like we've spent the last 20 years building DR. Maybe you shouldn't say that publicly. People will think less of you. So we're at this conference and you were the first person and you actually formed it in the way of a question. You're like, am I insane? So maybe if you would like talk us through what your original idea was that you were working with your teams on because I think there's a tremendous amount of value there. I'll be honest, Josh, it was really kind of off the cuff. Like we were talking, I think you guys were talking about cyber and sharing some of the things that you're doing. And I was just sitting there and thinking, this is all great, but we don't have the money, we don't have the resources and it kind of just hit me there in the moment. And that's why I kind of framed it as like, am I insane here? Because I really hadn't thought this out, but I thought to myself, we spend so much time focused on a good DR environment. And we run tabletops, we fail over to DR all the time and for good reason. But a lot of that goes back to 20, 30 years ago when we didn't have good data centers, our server rooms, right? You had water adjacent to them or water in them. And you were really concerned about potentially losing your main server room or data center. That's obviously changed, that's progressed. We work with colos and these are some of the most robust facilities that you've ever seen before. And knock on wood, but I'm not concerned about losing a data center anymore and having to fail over to DR. But I thought to myself, if I ever needed to, we've also started to invest in gold copies of data, vaulted copies, air gap copies of data. And so I have this data there. It takes longer to get to, it takes longer to bring back than a wood in a normal DR failover. But what if we spent our time and money investing in that technology and saying, hey, I can use that vaulted air gapped copy as both a cyber resilience copy and or a DR copy if I lost my production environment. And I kind of just threw that out there because we don't spend enough time focused on the tabletop exercises and the real life failover of cyber incidents. They're really impactful, right? To have to do failovers, even on the DR side that takes time, it takes downtime, it takes planning, you have to plan that. But on the cyber side, if you're going in and actually doing a true failover, it's really unrealistic to think you can do that because it can take you days to fully bring that environment back up. But if you do it in a test environment, that would be great. But we often don't have the money, like you said, tons of capital if you're doing it on-prem. If you do it in the cloud, it's obviously still money in OpEx to do it in the cloud. Plus you've running these other DR environments already. So again, I just kind of threw that out there and I was kind of curious what other people thought. And I don't know, I feel like it was a little bit of a, I don't know, lukewarm reaction, maybe in the room. But I remember you came up to me afterwards and you said, that might've been the most valuable thing I heard out of the conference. And then you said, I said the same thing to my team and let's keep talking about this, right? And I think, you know, so I think that's, you know, why we're having this discussion today. I continue to bring it. I was just at an event this last weekend talking to other CTOs and I shared the same idea. I said, we need to push this industry forward. And I think the reaction is becoming more and more receptive now to people saying, yeah, we don't need the same DR environments we've had before. We should have this joint DR isolated recovery environment. So. I completely agree with you because I think, and you actually articulated that really well. It hadn't occurred to me that there was also this maturing of the facilities that we have hosting our production stuff where, and when you said there's like water in them, I was thinking specifically of a data center on-premises data center that I used to walk through a lot on my way to meetings. And they actually have footage of, you know, 10,000 gallons of water coming down through the light fixture. It was right underneath the kitchen at this sporting facility. Well, you know, big organizations, usually that's not the case anymore, right? We've all realized how expensive that was to run. We've taken them to Colos. So the physical disaster has been matured, mitigated out to some degree. And it's congruent with my career where there's always the opportunity for an oopsie, right? I once had to call my CIO late at night and tell him what I'd done. I'd made a dumb mistake. I got put in charge of change review as a punishment for the next couple of years. I owned the change management program as penance for my sins. But, you know, outside of those oopsies, like tornadoes, yes, they do damage, but most of us have moved production out of that region, or, you know, even with a hurricane, you've got three, four or five days of warning usually if you're gonna cut over. So how often are we actually declaring physical disasters? Not that often. But by contrast, we had 130 health systems hit in a 90-day period last year with cyber attacks. So it's almost just a, let's make cyber the first case or the first class use case, and DR, like not everything at that site needs to be architected for fast failover. I think that's exactly right. I mean, cyber is our biggest concern, right? And that's the thing we're most worried about. It's not a matter of if, it's a matter of when. So why aren't we focusing our resources and our time planning on that event? You're right. The other physical events could happen where you need to fail over. But if we do this the right way, we can use that same environment for that recovery as well. Again, it might take a little longer initially, right? And this is why we need to work with our partners and our vendors to say, hey, how can we speed that up? How can we get that environment back online faster in a situation where we don't think it's cyber-related, right? Obviously when it's cyber-related, there's a little more thoroughness you have to go through to make sure you're bringing up the system and environment where you're not gonna get corrupted again. But when you know it's not a cyber type event, maybe there's a way we could speed that recovery up and we get it back online faster. I also think AI is gonna help us a lot in terms of automating a lot of these processes in terms of how we bring systems back up online, how we script them, how we run them, how we look for things. So I'm hopeful that there'll be a good merge of this idea and technology to help make it a reality. Because I really think that we all plan and are talking about the cyber, but I'm not sure that we're all focused in spending the right amount of time and energy there compared to what we're doing, still thinking about on the DR front. I'm gonna share my screen real quick and let's see if we can actually like walk through talking like what this looks like. So hopefully this is visible and they can make this look seamless when it gets edited. But these are kind of the options that I'm aware of when I talk with different health systems. And there's the traditional like building on-premises, that's insanely expensive. And most organizations are not gonna go get 15% more CapEx or whatever from the board. So some are looking at cloud, there's pros and cons, not necessarily the wrong answer. It's just, it's not free either. And one thing that I've just noticed is, there's a different skillset, there's a lot of complexities, getting both of those working together. And then, you backup something on-premises, whether it's VMware or some other platform. And then when you restore it, you need something compatible. One thing I've tried talking folks through is like you don't wanna be doing a cloud migration at the same time you're trying to run a cyber recovery, the two hardest known in IT, right? So pros and cons for sure. Some vendors have gotten into offering IREs and it's a nice blend of like, you put your backup equipment in their racks, you pay colo style pricing, but then when you declare a disaster, they give you a composed environment on demand out of their private cloud. So it has that elasticity and that like consumption-based pricing. And then there's like application specific. So like your EMR vendor might offer you an IRE, but it doesn't cover like the other 20, 30, 40 applications that you know you need back in a hurry, that's just for your EMR. And so that brought us to like this idea, which is what if, you know, you took your existing DR environment and you had all of the compute, networking, storage, et cetera, that you've provisioned for a DR environment, what if we split it into two personalities? So I still see the value of having the DR environment. It's where we land that active-active replication, you know, your storage replication, all of that real-time stuff still needs a landing zone because we have that for good reason. But then can we just take a little bit of it, like the bare minimum, you know, three node cluster, a little bit of storage. And one of the things that I keep coming back to is this idea of like slow thinking versus fast action. Like if you need to design something, it takes a little while to figure that out. But if you've already done the slow design work, which doesn't require a lot of physical assets, and then you get just that pilot light environment, now, later when you go to add capacity, usually adding capacity to something happens really quickly because we've done the slow thinking. And so in this, with this in mind, if you had a DR event and you needed that extra capacity that you borrowed, you know, in theory, we just take these nodes and compute and storage and roll them back over. We would accept a slightly slower recovery or failover because we need that, those assets. But on the other hand, if it's a cyber attack, then we have already done the slow thinking. We've got our VLANs, our access methods. We've got that pod of infrastructure. And now we're looking at how do we rapidly add capacity to that? So the way I think of it is like, your incident response vendor is going to get involved. It's going to take them some time to start figuring out exactly what was the blast radius? What was affected? Can we recover trust in production? Most times, if it's been, you know, the type of attack where they got domain admin, it's deep, probably your DR environment isn't going to get, be trusted anyway, right? Because the show shared IP space, if they compromise production, odds are they also had access to DR. So in that case, we've got maybe a bit of time in that first 12, 24 hours for some infrastructure folks to start setting this stuff back to factory defaults and moving it in and adding capacity to this environment. And I think it's worth remembering, like we don't need a full DR site, right? The goal is going to be that minimum set of applications, that 20, 30, 40 applications that we need back. And like, yes, that's easier said than done, but it's not like the full fleet, right? So am I thinking about this the right way? Is that kind of like how you envisioned it? No, I think that's exactly right, Josh. Yeah, a hundred percent. This is exactly what I was thinking of, you know, and just to go on and carry on a little more. So as you think about the different types of data you have, I really think this is applicable to a lot of our block storage, right? So today, you know, in our environment, we've got all of our block storage and we have immutable backups, right, of all of our block storage. And then we have a DR environment that is replicating in real time. And then we have backups of the backups in the DR environment too, right? And then we have a AirGap copy of the production environment that's got that clean data. How many copies of this data, like, do we need? I mean, we have them for good reason, but it's been based on this historical mindset that, you know, you need to plan for the physical disaster. And now in modern times, you've got to plan for the cyber disaster. So we will still have our backups, right? I still got to remind people like this. You'll still have your backups for your block storage. So even if you had an issue with your production data, you can go and get backups to come back. You don't have to fail over this other environment if it's just an issue with the data itself. The only time you would need this is if you had a true physical event or a cyber event that you would go to this other environment to stand it up. It does get a little more tricky as you start to think about your file storage, which is namely in healthcare, it's our imaging data, right? We don't have backups of imaging data typically because the files are so large. So we really think about DR as our true backup. It's not just another environment, it's actually the backups of that data, the raw data. And so having a vaulted copy though is very expensive because again, how costly it is for this file storage. But again, it's a business decision for the organization. This is the way that I want to present this. It's like we can continue to operate the way we're operating it and here's the cost to continue to operate this way. Or it becomes a business decision on what the potential risk is and what does it cost and how quickly we can recover from this other environment. And would I make the decision today to say we're doing it? The answer is no, I wouldn't say we're ready today. I do think though that we have the ability with minds like yourself and others to push this industry forward. And I would like to think within the next two to three years this could become a reality that we could start to see in play and it could actually make a huge difference for us in healthcare because we're not getting any additional money. Our reimbursements aren't going up. We need to find the ways to be more efficient with the capital and operating dollars we have today. And I'd like to think something like this will allow for that. Yeah, I see two approaches to building cyber resilience and I guess I would characterize them. I tend to think in terms of like Gantt charts and waterfall style projects because I grew up on infrastructure teams where that was kind of how progress was envisioned. And I see some organizations leaning towards this like consulting first, you know, it's business impact analysis, it's building out the perfect list of every application that's needed, you know, issuing RFPs for external consultants to come in and build this like amazing God level IRE for everything versus I would almost characterize it as like inverting everything and starting with what you can do now with the skills and the budget that you have. So when I think of like what one of the things I try and help people explain to folks is we built DR environments at geographic remove because the threat that we were negating was physical damage. So you needed physical. In this case, you don't need another site. This style of an attack, everything physically is fine. It's just that your trust in your environment and your security context has been destroyed. So what you need is security distance and that's that isolation component, right? So this equipment could be sitting in racks right next to production or in racks at your DR site and it's fine, right? 100%, that's correct. Then starting with what are the predictable things I know I will have to do in the immediate aftermath? If I need an IRE, the very first task is gonna be recovering identity, right? So once you've like outlined in this like nucleus of an environment, here's how I access it from privileged access workstations. Here's how it's isolated. Here's how I get things in and out of it. Here's how it's exposed to my backup data. Can I then start running tests of how long does it take for me to recover active directory back to a previous point in time on new equipment and get that up and running DNS, DHCP, global catalog, all of those things around it, rotate the Kerberos ticket granting tokens, roll passwords, do all of that work. And let's say we can get that down to four or five hours, right? We'd like to think two or three, but whatever. You get that down to a reasonable amount of time and then blow it away, right? Reset back to factory defaults. And there's gotta be that infrastructure team who's talented with their Terraform and their scripts and their Ansible, whatever it is that we can kind of take it back to known good clean state with nothing in it, run the test again. And then I deal with this constantly where health systems are like, we need to do this business impact analysis. I'm like, we could, or I'll bet you if we sat down, your team knows off the top of their heads, five to 10 applications that are unquestionably on the list. Why not start? And so it's this idea of like inverting and like starting with kind of, I hate, I feel like minimum viable has been beat to death, but the things that we know we will have to do no matter what and getting really good at those and resetting and running it again and getting it to the point where like that's scheduled and it just runs every Thursday. And the beauty of it is it's isolated. Like we can do whatever we want. There are no name conflicts, IP conflicts, nothing. And every time we get really good at it, we're like, okay, let's add in the next three or four applications. And you end with perhaps that consulting approach to like, what are all of the applications? Do we have them all? What are we overlooking, et cetera. But at every stage then you're getting meaningful, actual risk reduction at the lowest dollars spent to develop the skill sets and competencies to where you can legitimately look the board in the face and say, we will have these 30 applications back online 36 hours after the event or whatever that timeline is, right? I think that's exactly right. Yeah, and I love that approach. You know, we've been talking internally kind of similar thing too, is like, what are those tier zero applications that in an event we've got to bring up right away? Because without them, nothing else will run anyway, right? So you've got to bring up that tier zero. And I would say, you know, most organizations these days, to your point about the BIAs, you know, as I talked to my peers around the country, I think we've gotten a lot better in the last five to seven years. Most organizations have their applications at a certain tier level today. And a lot of that's just based on like what level of storage they're buying in DR and the RTO and RPO as they're going to bring those back online from a DR perspective. So I think we have a lot of that data already, which I think is great. Doesn't necessarily mean we have to go through the full BIAs on everything again, because I think it already exists. But I like the idea of starting small, not trying to boil the ocean with this, but let's start with that small environment and then continue to add to it as we refine it. And as we get it down, the process is down. I think that's the area within healthcare that we all could spend more time on. I think today for a lot of us in healthcare, we have backup cyber plans. They're documented. We don't spend the time testing and running them that we like we do for the DR events. And that's the part I think we need to change because cyber is much more of a concern than DR. Yeah, I try and tell everybody I work with how tough it is being in IT, where you've got a hundred things to get done and time for 19 on any given day. And so we can't blame people that they're not out running drills and testing all the time. But on the other hand, if we can find the time or get to the point where we're kind of good at infrastructure automation and get these scheduled, if you can schedule the recovery of that application and see does it work or not. And then the other thing that I've seen is we can have a plan, we can have this tier level of applications, but until we've tried recovering those first three or four and see, do they run in the absence of everything else? I always try and help people think about like there's something in that interface that when you click it, if that API call or that integration or that other application isn't online, what happens? Like does it gracefully and give you an error message or does the thing crash? And what is that sequencing of like, you may have an application that's well down the list, but it's unfortunately a prerequisite for something else that is on the list. And so we only learn by doing, like we can have all the best plans in the world, but until we've tried recovering those in an order that brings everything back online, that's not going to work quite well unless we've drilled it. Well, and we already know that because every time you do a DR event today and you do one for whatever reason, you always learn something new, right? Things usually don't always work exactly as you're supposed to. You're like, oh, I got to go back in and fix this. Okay, update the process and policies. So yes, you're right. Like I think every organization who's been hit so far with the cyber event, like they're like, oh yeah, we weren't expecting that. Well, of course not, because you never ran the test in the first place to see it. And so you're seeing it in real time for the first time. And I think that's what the whole goal of this would be, let's avoid that. Like we don't want that initial cyber event when they occur to be the first time we've actually ever seen this as we're running this through our environment, right? We want to test it. So I don't know, Josh, I'm excited about where this could go and I'm excited we've got vendors like yourself who are thinking about this because it's going to take a partnership to get there. Yeah, and I love the fact that unlike the DR tests that we grew up running earlier in our career where you were giving up more time in the nights and weekends that you could have spent with your family because all of us were patching things in off hours. Now, finally, it's in an isolated environment where we can run this in the middle of the day and like nobody cares. So now it's not drilling this, especially if we invest in automation, which most all of us have those tools out there. We can script something and have it run on an automated basis. Now, it's not more time away from family to prove out that test process. So I'm excited. Well, and not only that, I think one of the things about cyber too, I think that the concern to us all is that you can be hit and attacked in so many different ways. It's always hard when someone says, how long would it take for us to recover in a cyber event? I don't know. It's going to take us a day to figure out or two to figure out what was even impacted necessarily. We'll have the initial things that we know for sure that are down, but what else is there? What other things are impacted? So it's not what we typically thought of in a DR type event. It's just that fog of war understanding. And then once you figure out what's down, now you've got to figure out how to bring it back. And so having the IRE environment, I mean, you can sit there and run Monte Carlos now because of the automation. You can sit there and look at various scenarios and say, hey, what if we were hit this way? What if we were hit this way? Okay, let's bring that back up. And to your point, you can go through and say, okay, what fails? What are the dependencies? Okay, if we're hit this way, then hey, we've got to be thinking about this. So I think there's a lot of opportunity and it gives you a lot more flexibility to think about how you would respond by having that isolated recovery environment stood up and ready to go. Yeah, 100% agree. So Christian, thank you. I appreciate the amount of time you've given me. I know you've got a lot on your plate, but I think there's some goodness here that everybody can learn from. So experimenting from the bottom up, trying to build out those like individual building blocks, reusing what we already have. I think there's a lot that organizations can do to get started within the budget footprint they've already got. I hope so. And Josh, thanks for reaching out. And yeah, it's good talking to you today. And yeah, hopefully we can drive this, continue to drive this forward a little bit. All right, have a great day. All right, take care. To help build cyber resilience in healthcare, we've been gathering all the lessons learned from past ransomware attacks we've come across, and we're making them available to help listeners prepare effectively. To download this content, please visit the link in the comments. We also want to hear from you. If you know somebody with insights and lessons learned who'd be willing to share, reach out to me on LinkedIn and we'll get them on the show.
