Transcript
Essentially, we made things up as we went. Also, it was the first time for me to really witness the sheer chaos of a client whose whole livelihood and life's work and passion was stolen from them and there wasn't anything that they could do about it. Hello and welcome. Please come in. Join me. I'm Jonathan Crowe, Director of Community at Ninja One. And this, this is IT Horror Stories. Brought to you by Ninja One, the leader in automated endpoint management. Hello, everyone. Welcome back to another episode of IT Horror Stories. I'm your host, Jonathan Crowe, Director of Community at Ninja One. And I'm very excited to have a special guest with us today. We have Shelley Ma, Senior Incident Response Lead at Coalition. Shelley, thank you for joining us. Happy to be here. So everyone, we have folks who are brave enough to come on the show and share their horror stories. We have a guest who spends nearly every day in an IT horror story, not of her own making, but coming in and being a hero in that story. So Shelley, tell us a little bit about incident response and what your role is at Coalition. Yeah, absolutely. So I've been in the industry for 11 years. And as far as my day-to-day goes, I spend my days assisting companies of all sizes and functions who've experienced some type of cyber breach or cyber incident or digital compromise, i.e. they've been hacked. I had my start as a digital forensic analyst and then slowly made my way towards becoming an incident response lead. And day-to-day, I guess I work with the claims team at Coalition, responding to policyholders. We see a lot of business email compromises. We see a lot of fraudulent direction of funds, and of course, a lot of ransomwares, which is where I fortunately or unfortunately cut my teeth. Absolutely. And what a unique perspective that you're bringing. We have those folks who, these incidents, when they happen to them, it's a singular instance that really stands out. And you're coming from the perspective of this is your day-to-day. You're brought in to essentially I think it's fair to say, you're kind of like the Ghostbusters, right? You're in the team at Coalition. You're brought in, but you're so much more than that. That's a much cooler title, Jonathan. I think I'll adopt that. So someone who's having a bad time, the horror is happening. You're brought in to help in that situation. And it's much more than that. We were talking yesterday and your colleague kind of was describing your role as also being, sure, Ghostbuster slash therapist slash several other things, because you're really coming into working with people in a scenario where there's high tension, there's high emotions. Talk a little bit about that. Absolutely. So I would say that folks who work in incident response are prone to burnout. We're operating on the times and cadences of our clients, and we're also operating at the time zones of our threat actors. It can be really high stress. It can be really emotional, and it does involve a lot of difficult conversations. But I would say a common theme of motivation for anybody that works in this industry is the perpetual learning and growth. Of course, also like the innovation and the technology, but primarily what keeps us going is that very rewarding payout of seeing positive outcomes as a direct result of your impact. And that is unmatched, and I'm sure anybody that works in similar industries can relate to that. Talk to us a little bit before we get into some specific horror stories. Talk to us a little bit about how you got into this field, because, again, we're talking about these situations that most companies and most people are trying their hardest to avoid. Have you always wanted to seek out ways that you can help, or what are some of the other origins of how you got into this path to begin with? I would say that quite a lot of folks that are established in this industry, we just fell into it. A lot of us come from different backgrounds, different technical backgrounds, and then somehow just weaved our way into this field. So as for myself, I used to be a science major. That's what I went to university for. Believe it or not, I wanted to be a doctor, so I was in pre-med in South Africa, which is where I grew up and went to university. It takes six years to earn a medical degree. I wasn't patient enough. It was taking too long, so I decided to shortcut and go into the field of forensic science, like the traditional kind of forensics. I overcame my fear of dead bodies reasonably quickly. You had to. Also around that time, I applied for a scholarship to go to the United States to further my forensic career. I was lucky enough to get that scholarship. Enrolled in George Washington University in DC. So that day, it was the first day of orientation. They had all of these new graduate students packed into a room. The idea was that we were going to be introduced to the various forensic streams and the various faculty members. I just remember sitting there, I was sitting in the front row, and then you had all of these folks go up talking about forensic science, forensic psychology, forensic toxicology, chemistry, etc. I wasn't being very motivated. I didn't feel like it was capturing my attention up until the very last person who went up to speak. She was the head of the program of digital forensics. At that point, I had never heard of digital forensics, but it completely captivated me. I've always had an affinity for technology. I've always loved tech. At the same time, I was also very interested in science. I was very interested in forensics. Here is this industry, this career that married up all of these different components that I was super interested in. Right after her talk, I went up to her and basically strongly requested if she could accept me into the program. She did take a chance on me because I didn't come from any type of technical background, but she did take me on. That's essentially how I made my pivot into the industry. That's amazing. In just the field of forensics, obviously, in a lot of ways, some very obvious big jumps from that area of forensics into digital forensics, right? Do you feel like there is kind of a necessary ability for you to be able to step back and see things a little bit more objectively, not to be kind of caught up in that moment? I would say yes, for the most part. The reason is because it's very easy to be captured in the weeds of tech. We can get very engrossed in the nuances of the scenario that's happening in front of us. I think, coming from a science background, what that's taught me is to always have that curious mindset and to also zoom out and think big picture. What is happening? What is a story that can be told here? At the end of the day, when we're applying scientific principles, whether that's dead body forensics or whether that's digital forensics, the protocol and the workflow has a lot of similarities. There's always that common denominator of finding out the truth. So yes, I think having the scientific background and pivoting that into what I do today, it's definitely had a lot of advantages. Also, one of the things that science always teaches you, especially in those professions like medicine or any type of theologies like microbiology and physiology, is it teaches you to be a critical thinker. Of course, that's a skill set that is very relevant and very important in what I do today as well. It strikes me that this is also just one element of your role, that you also have the really messy components of dealing with people. How much of your work is also dealing with clients who are, in addition to, you're brought on board to see what's the story here? What exactly happened? Piecing together those pieces of the puzzle, following the clues. Then in the meantime, you also are working with people who have business disruption, who may be facing a lot of pressures themselves to resolve things quickly. Are you involved a lot with the client-facing aspect of things too? That's my everyday. Yes. I pretty much deal with people in all facets of what I do. Of course, there's the client aspect. We work very closely with lawyers and counsel in general. Even just going back to the clients that I'm dealing with, you're dealing with personalities of all type across the entire breadth of an organization. That does start with C-suite and executives, but it also involves the technical folks, the HR folks, maybe the person that clicked on that initial phishing email that is forced to be on the call even if they don't want to. Yes, you're always dealing with people. Then, of course, there's the internal team. It's a lot of collaboration. We have to work in a team dynamic because there's so many different components that are happening at the same time simultaneously. We have the folks that are working on recovery and remediation, the people that are doing the forensics and trying to figure out what's going on. Then you have the people that are actively monitoring the environment to make sure that nothing additional is happening, that the threat actor is not coming back in. Then all of that has to be threaded together so it can be relayed to the client and they can see the progress. Oh, and let's not forget, it's also the hackers. They're also people at the end of that. On the other side of the keyboard is also a person that you're dealing with. It's mostly people and a few garnishes of technology. Well, I'm so glad you brought that last part up because that's absolutely what I want to dive into next. Here you are. You're in the whodunit. You're the detective putting things together, but it's not often, I think, that a lot of those forensic experts are then expected to go, maybe not necessarily literally face-to-face, but engage with the perpetrator of the crime. That also is a key aspect of what you're doing, right? Absolutely, yes. A huge part of what I do is engaging with threat actors and trying to negotiate ransoms. And so that aspect of it, when we're talking about IT horror stories, ransomware has come up a few different times, as you would expect, right? It's one of the larger, more certainly prolific and headline-grabbing cyber crimes. When people think of IT disasters, that's often one of the first things they think about. The perpetrators of those crimes, the fear of what the downtime can cause, the pressure that people face in there. There's also a very different, very real layer to this where you're talking with criminals. I know you wanted to talk about one of your first investigations and windows into this world. Yeah, happy to. Right after I graduated from the digital forensics program, my first job was at a boutique digital forensics and incident response company. It was a relatively small company, probably only 14, 15 people. At that time, we saw a lot of remote intrusions. There were a lot of tax fraud matters. I worked on a lot of website compromises, and there were a number of civil matters. But it wasn't until April of 2016 where the cyber landscape made a huge shift, because that was the start of the rise of ransomware. I didn't realize at that time how much that would dictate the direction of my career, like how much it would impact literally everything I do from that point forward. When you're new in your career, you feel like you have a lot to prove, right? Especially in consulting, when you're engaging directly with clients, especially in a niched industry like incident response, people come to you in moments of crisis, and they expect you to be the expert. Let's say you have this weird growth, and you want to go to the doctor, and the doctor says, oh, this growth, I know exactly what it is. It's no big deal. I've seen it 100 times, and here's exactly how you treat it. That feels good versus, oh, yeah, I don't know what that is. Hold on. Let me Google this. Totally different experience. As a fresh consultant, you're already anxious. You're fearful. There's a lot of imposter syndrome going on. Also, I didn't come from that heavy technical background. Suffice to say, those early days needed a lot of mental gymnastics on my part. The very first ransomware case came in for me mid-April of 2016. That was also the first one our companies ever had, which, if I think about it in retrospect, in comparison to what we see today, it was a relatively low-grade attack, but at that time, it was new territory, and I was completely shook. I was like, what is this? Part of the response involves threat actor reach out, communicating with threat actors. No one told me prior to that, not in school, not in internship, that it would be part of my job to actually speak to cybercriminals in Eastern Europe. So there was no playbook on how to deal with ransomware. There was no protocol, no SLA. We didn't know anything about the attackers behind them. Essentially, we made things up as we went. Also, it was the first time for me to really witness the sheer chaos of a client whose whole livelihood and life's work and passion was stolen from them, and there wasn't anything that they could do about it. So the panic and the fear and the uncertainty was really shocking, and that was also one of the first times that I realized the real impact of cybercrime at a fundamental human level. So that first case, as I said, we made things up as we went. Threat actor asked us to reach out to them over email, so we set up an anonymous email account. We had a meeting to discuss what our first message to the threat actor should be, and then we decided on hello. After a few rounds of back and forth in very broken English, remember this is before the days of ChatGPT and Google Translate was like five out of ten at best, we eventually agreed on the hefty ransom price of one Bitcoin, which at that point, Jonathan, it was $300. Don't remind me, Shelley. I know. I did not invest either. It's a sore spot. We had no idea how to buy Bitcoins. We didn't know anything about crypto, so we ended up going on this dodgy exchange site, I would say akin to Craigslist, but not Craigslist, and then we found a random guy who agreed to meet us on a street corner downtown, and then he asked for the exact amount in cash, and he said that once we paid him, he would transfer the Bitcoins to us. So we did exactly that. We put the bills in a brown paper bag, and then we met up with a guy downtown, and he was carrying an iPad. We went into a bank. He laid out the bills. He counted them. The bank security was eyeing us the whole time. What are these people doing? He was happy with the payment. He swiped on his iPad, and we got our Bitcoins. Eventually, we paid that ransom. We got the decryption tool. Now, this decryption tool is a very janky tool. It's built in someone's garage, right? We're not talking about an enterprise-grade decryption software. It was like trying to use PowerPoint in 1998. I don't know if you still remember that, but it was very labor-intensive to get the thing to work. At the end, we managed to decrypt the files. We managed to decrypt 70% of the files. It wasn't perfect, but we still salvaged the vast majority of what the client needed to be back up and operational and start rebuilding some of those permanently broken items. We did give the threat actor that feedback. We let him know that, hey, only 70% of the files decrypted. They were very generous, and they offered us 30% of the payment back. They were like, here's a 30% discount refund. However, TLDR, we ended up getting nothing back because they eventually said to us that they spent the money on vodka and Tommy Hilfiger. As if it could get more cliche than that. What an amazing actual cloak and dagger, the money in a brown paper bag. Incredible. Going back to the mindset of the client in that case, this is something, as you mentioned, it really hit you, the real-world impact, how you saw them reacting. Then looking to you, did you feel a need to... You mentioned you had to be the expert. You had to play that role, even though this isn't something that no one has done before. You're doing it for the first time. You're filling it out. There's also an element where, did you feel like you're maybe putting yourself at risk? Not necessarily physically, but so much as digitally, you could now be a target. You're interacting with these people who clearly have no hesitation to commit digital crimes. We have to be very careful in the way that we engage with threat actors. We make sure that we have all the barriers in place to ensure our safety and our anonymity. When we communicate with threat actors, we do not reveal our identities. We engage with them on behalf of the clients. In those days, interesting enough, the ransomware actors often did not know the identity of who they attacked. It was very opportunistic. They only cared about the technology and its exposure, then they would target them. There wasn't the level of reconnaissance that we see today, where it's super targeted and they know exactly who you are. They know about your finances. They know about your employees. We didn't see that then. Suffice to say, it didn't feel as personal as it would today. Then at the same time, after you've had back and forth, this interaction, you realize that you're dealing with someone who spent their money on vodka and Tommy Hilfiger. Did it take the mystique away a little bit? Yes. Obviously, that was your first engagement. Now you've had however many. Do you end up developing an awareness? The monster in the shadows loses its power because you get to see a little bit more of it. 100%. Whenever I am put in a position where I have to communicate with threat actors, I always remind myself that it's a human on the other end. It's a person. They think like a human. They are prone to the same psychological impacts as we are. When I used to think about hackers, I thought about them in that very stereotypical hoodie in the dark basement, glaring over the screen. Do you know exactly what I'm talking about? The screen shining on their face? I don't see that anymore. It's just a regular person who is trying to make a living in not a very nice way, but yes, it definitely humanizes them and it removes the mystique. In many ways, that's very helpful when we're thinking of techniques and tactics on how to communicate and negotiate with them. A lot of my clients tend to ask the question of, we're dealing with criminals, we're dealing with terrorists, we're dealing with perpetrators. How do we know that they will stick to their end of the bargain? I always have the same answer because it is true and it always tends to surprise people. That's that threat actors, they operate their businesses like it's an actual legitimate business. They are very reputationally conscious and they do care about how they show up. If they start to garner the reputation that, oh, we have somebody that paid us and we didn't stick to our end of the bargain, ultimately that could come back to bite them and they might lose out on future payouts. Yes, it's a very human facet to this job. The evolution into this is run like a business. Due to that spike in incidents and attacks and all the funds and everything, these have become major, major operations. I know that that's a whole nother element. You've seen things from now almost 10 years. How have things changed? That's a great question. I will say that I'm very glad that I was working in the industry when ransomware was still in its infancy. I was able to observe its evolution and was able to develop my skillset along with it. You didn't know if you were on the right track because the track didn't even exist. If I were an incident responder entering this world for the first time today, I imagine I'd have a lot more anxiety. Today's landscape is so far beyond my wildest imagination from the brown paper bag days. The magnitude of damage that I see today is massive. It's magnanimous due to the interdependency and the enmeshment between our real lives and our digital assets. I've dealt with massive corporations losing millions of dollars on a daily basis due to a full ransomware lockdown. I've had clients in healthcare that weren't able to treat their patients or deploy ambulatory services. The ones that tend to affect me the most are the smaller family-owned businesses, the mom-and-pop shops that have had their businesses completely crippled. I've seen threat actors actively harass customers' clients. I've also seen threat actors threaten the public safety of the family members of my clients. I've also had attackers ask me if they could submit their resume to me for a job. It just goes on and on and on. Cyber threats, they really should not be on the back burner, not in present day. I never cease to be surprised every year, every couple of months by something new that threat actors are doing, whether that's tooling sophistication or stealthier intrusion methods, anti-forensics, increased aggression, or the integration of new technologies into their own protocols, like artificial intelligence. They will keep getting more and more sophisticated as our technology and our dependency on technology becomes more sophisticated, and we will always be playing catch-up. We're always 10 steps behind. You'll see the waves of things whenever there finally is a big crackdown, and they're able to uncover a big organization or infrastructure and take it down. Then, of course, the torch gets carried along. Things pop back up a little while later. A big noticeable shift is as backup has become more prevalent, as people are able to... I guess that threat of encryption has been able to be, at least in some cases, be addressed. There's a shift to exploitation, into stealing data, posting it publicly, shaming people. In terms of the horror stories, you're coming in, and the main focus is to help the client. It must be so rewarding to be able to come in and help people in dire moments. At the same time, there must be a cost to it. It must be having that continuous exposure and everything. I'm sure there must be some cost to it as well. A lot of us that work in cybersecurity or incident response or any type of tangential security industry, we become very paranoid individuals in general. There is general consensus that all of our data is already out there anyway. What you were saying earlier about the sophistication and the evolution in their techniques in response to the wide adoption of backups, that's what I call the irony of better security. When something happens, like something in the industry is happening, like attackers keep getting in through this one vulnerability, then we tell everybody, don't use this tool anymore, patch this vulnerability. What a threat actor is going to do, of course, they're not going to just stop, sit back and say, oh, well, too bad, let's go and do something else. They're going to find another way in, and they're going to find another way in. The next time they come in, it's going to be more stealthy. It's going to be more sophisticated. They're going to use better tooling, and we have to then respond with more effort and stronger tools and whatever. That cycle just keeps repeating itself over and over. That's why we're in a loop. That's why we're always behind them and constantly playing catch up. Security can't be on the back burner. It needs to be a priority. What about in terms of your work as an incident response provider? When companies do, unfortunately, get into situations where now they have to be in incident response mode, what are the things that when you go into a new situation, a new deployment, that you're saying, okay, I'm so glad this company had X, Y, or Z in place? What are the things where you're like, oh, no, this is a bad situation. This could have been so much better. That's such a great question. I don't want to be a walking cliche, but there is truth in the statement that it's not a matter of if, but a matter of when something happens. Often, a lot of focus by IT teams is to place a lot of attention on that perimeter. How do we prevent threat actors from getting in? How do we drill it into employees to never click on phishing emails? Employees will always click on phishing emails, period. That's always going to happen. What I like to encourage is for organizations to think about safety nets that they can put in place to minimize or completely extinguish the threat after that phishing email has been clicked on. After the threat actor bypasses your barrier and your perimeter, then what? Did you ever watch Game of Thrones, Jonathan? Of course. You know how they put all that focus season after season on that giant wall? You need to build the wall. Let's protect the wall. Build that wall as high as possible. The White Walker still managed to penetrate that wall, and then the inside was all soft and mushy. A lot of companies have networks that are exactly like that. They have these great perimeters, but the inside of the network is defenseless. Once something gets in, something like malware or ransomware or an encryption algorithm, it gets to absolutely everything. It cripples everything, all departments, all facets of that entire digital network, and then it becomes catastrophic. It's important to think about how you can neutralize the threat. Consider things like endpoint security solutions. There's a lot of talks about EDR these days. They are basically antiviruses on steroids. They are protective. They are great. Consider them. Segregate the networks. Isolate your critical and sensitive data. If something like ransomware gets in, they only affect a small portion of your network and not necessarily absolutely everything. Only permit access for people who absolutely need access to certain file shares or certain applications. Not every user needs to be an administrator on their endpoints. Not everyone needs access to absolutely everything. Why that's important is because when threat actors come in, one of the first things they do is they try and escalate their privileges. How do they do that? Let's say they come in on a random service account or a printer account like Canon service account. Canon should not have access to HR folders or anything like that if it's configured properly. But let's say they try and obtain the credentials of the next level account. And it's a random employee. But if that employee's accesses are not configured to only allow what's necessary, and let's say they have access to a lot of different facets of that network, then the threat actor would also have access to all of those different facets of the network. The one thing that you don't want is for a threat actor to gain access to the highest privileged accounts, which would be like a domain administrator account. And that's essentially like having keys to the kingdom. So it's very important to audit your networks. Don't have dormant accounts that are domain administrators sitting around and hasn't been used for 20 years. That's not necessary. And those are the things that threat actors love to target and love to exploit and love to use to carry out the rest of their attacks. I always encourage people to just clean house, you know, every couple of months, take a look at your network, what's old, what can be gotten rid of, what's end of life, things like that. It's also very important to have an incident response plan. When you're going through the exercise of creating an exercise, of creating an incident response plan, it like forces you to think about the components that are absolutely necessary to get your business back up and running in all sorts of different scenarios. And it also highlights what's missing. Like I heavily encourage fire drills, which are called tabletop exercises. And those are like essentially role play exercises that stress test your response in a safe and contained manner, so that when a real incident happens, you know exactly what to do. The, from a response perspective, those clients that have an incident response plan, that have a robust protocol on exactly what to do when an incident happens, the efficiency in which we can get them back up and running and business continuity, to get business continuity back, it's night and day in comparison to those that are truly experiencing it for the very first time. Also, the other thing that I always tend to advise is when you get an email request from even a known vendor or known contact, and they ask you to change the payment method, just pick up the phone and call the last known number before you go ahead and do it. Because we see fraudulent transfer of funds way too often. That doesn't need to happen if you just like spoke to the actual person and confirmed with them like, hey, you know, did you really request this change of funds or did you not? And that can save a lot of headaches later on. And also I like that you touched on the topic of backups, Jonathan, because yes, it's one thing to have backups. Most of my clients these days have backups, but we still run into a lot of issues with them. We don't make sure that our backups are up to date. Sometimes people don't know how to restore from backups at all. Like we know they exist, but we don't know how to restore from them. So you just always know that, always ensure that they're valid, ensure that your team knows how to get them, get access to them and know how to restore them in a timely manner. And also always make sure that they're disconnected from your primary network, because again, you know, if everything is connected, once something gets in, it's going to impact everything. And that includes the backups. I love that so many of your suggestions there were not just focused on tooling, on software. Of course, it plays a critical role in things, but having the people, the processes, the policies in place and actually reviewing those, actually putting them to use. I'm a huge fan of tabletops, so I'm glad you brought that up. Being able to actually go through those exercises and then cover, oh, we actually have a gap here. We don't know the answer to the question. Exactly. What if this person was out? Do we really know how are these things documented? I think it's a great use of time. And of course, time is the most valuable commodity we all have. And so I think it gets put aside a lot, but having you folks like yourself, really reinforcing that is a great thing for people to hear. Thank you. Shelly, thank you so much for joining us, really talking about your role here, which is a very, very interesting and important one. Thank you so much for everything that you do. And thank you for coming on and being a part of IT Horror Stories. Absolutely. My pleasure, Jonathan. And thank you very much for the opportunity. It was such a pleasure to speak to you. That's all for this week. We'll be back with more soon. Thanks for listening.
