Transcript
Hello, everyone. I'm Jonathan Crowe, Director of Content and Community at NinjaOne. I almost said NinjaRMN. And I am here to welcome you to IT Horror Fest. And I've got some sound effects I'll have to play here in a second. Let's see. Well, anyway, we'll get back. I've got some great guests for you. I'm so excited to see everybody here. We've got a great hour planned for you guys, where we're going to be sharing IT horror stories. We're going to be giving away some really cool prizes. And we're going to be subjecting ourselves to a really ridiculous challenge that I can't believe all of us agreed to do, especially in the middle of a workday. Jason's already saying that he's got a client meeting right after this. So we're going to see how he holds up. We're going to be eating the spiciest lollipop known to man. It's called the Toe of Satan. Strap in. We're going to see how all this goes. We're going to save that portion for the end. But for those of you, this is your first time with an event that we've done here at NinjaOne. First time on Crowdcast, maybe. Welcome. Thank you so much. You're already using the chat. This is awesome to see you guys. We want this to be very interactive. So feel free to go in the chat there. And in terms of sharing your own stories, that's something that we want to hear, too. These guys I have that I'm going to be introducing in just a second here, they've been around for a while. They've seen some things they can't unsee. And they've got some stories to share. And we also want to hear from you guys, too. So as they're sharing theirs, feel free to jump in the chat. And if you guys have a really good story, I'm sure Gavin does. I know he does. We maybe will be able to bring you on and give you some prizes in exchange for that. So let's get this thing off here. First introductions. I already introduced myself there. I'm Jonathan Malicious Matt Crowe, Director of Content Community at NinjaOne. We also have with us Matt Fox, the Kerninator. Matt, with the nicknames there, this is like a typography reference. Maybe also some Trogdor, the Berninator reference going on. I thought my name was Matty. I don't know. You've got your cat. Well, and actually, you've got another guest there with you. What's your cat's name? Oh, this is Big Pete. So Matt and Big Pete. Great to have you here. We also have Tom. I know you installed last summer Lawrence of Lawrence Technology Services. If you've been on YouTube, you've obviously seen Tom's videos. Tom does so much for the channel. So many good reviews, but also great just thought leadership and sharing what's working with his business. Tom, thanks so much for being here. I'm excited. And you, I think I may have something to do with the Toa of Satan being here, too. You absolutely do. Yeah, this fire is appropriate, I believe, because you're kind of a ringer with this challenge. You have done all these things. I asked Tom if I thought no one would have heard of something called the Toa of Satan. Tom's like, yeah, we tried that a couple of years ago. So Tom, this is not your first ride. You're coming back to the Toa of Satan. I'm excited to see how you do. Oh, yeah. Oh, this is good. Your last name was the best. It was the easiest one, right? Jason Slagle from CNWR IT Consultants. Jason, thanks so much for being here. He's updating his LinkedIn right now to change it to this. Yeah, that's, I mean, that seems like a thing I would do, doesn't it? And then, oh, yes, the MVP himself, Kelvin Invoke Terror, Techlar, CTO of Lime Networks. That's the scariest photo. That's the scariest photo of them all. Look at the calm eyes. You don't know what's going on behind them. I didn't actually have to modify this one at all. This was just a photo that Kelvin had. This is, Kelvin is actually hundreds of years old. That's how he's been gotten so good at scripting. Also, in addition to his his day job, he's also a blogger. He's created CyberDrain.com. Amazing resource for anyone who doesn't know about it. Go check it out. Kelvin blogs regularly, releases scripts. You've also been busy, my friend, with some other bigger projects. And I think you'll have some a chance to talk about that in more detail later. But you want to give a quick shout out for something you're working on? Yeah, sure. So I've been working on this, this CyberDrain Improved Market Portal, and it's a Microsoft 365 multi-tenant management tool. That's just the best way to describe it. And it's much better than all the Lighthouse and all the other stuff that's on the market right now, because it's more oriented at MSP workflow. So I guess that's the best description. I love it. I dropped a link to it. People should check it out. Thanks. All right. And then I'm going to switch out. Let's see here. I'm going to let Tom introduce himself. But to do that, I'm going to have to switch a little thing right here to get Tom back on the screen. So let me pull up Tom here. And I will speak for Tom a little bit. Tom is our channel chief advisor. He has been with Ninja. We started our MSP live chats back around a year ago. And Tom is a former MSP owner. And I can't wait for you to see his costume when he comes on screen. Here he comes. Look at this. Tom, you got to show off the claws, too, my friend. Look at that. That's commitment. I love the sound of those, too, because you can tell those aren't like plastic claws. They're 440 stainless. They're dangerous. Oh, my God, guys. We have a great panel, as you can see. Let's go over our agenda a little bit. We've got our Wicked Welcome. We're going to be recording this so you can relive all of this in all of its horror. But the first thing we've got going on here, I'm going to direct your attention to the polls down below. And Tom, thank you so much. Tom is our content manager on the back end here. He's launched a couple of polls there. One of them you guys are already voting on, and that's the six-word horror stories. We're But first, we've got, what is the worst IT nightmare this year? And we've got some options for you guys. And for those who are picking other, drop what you're thinking of in the chat. I want to know what you guys are thinking. I love people are impressed with your Wolverine, Tom. Truly impressive. So while you guys are voting on that, another quick look ahead here. We're also going to be talking about, obviously, we're sharing horror stories, right? Then we're going to get into how to survive an IT horror movie. Those are our rules, kind of like a la Scream. These are things you can and can't do if you actually want to survive an IT horror movie. And then we're going to get into our prizes. And we're going to do our Help Desk Hellfire Challenge. Guys, we were talking about this and leading up to this, and I wanted to do this at the very beginning. And Tom, having the experience, was like, no. No, do this at the end. Just in time for me to go to a client meeting. Just in time for Jason to show up and put his best foot forward. Okay, yes, exactly. Prizes. Everybody wants to know, right? And so what are we giving away? We're going to give away five IT survival packs. And what those are, we've got a Ninja Timbuktu backpack. And it's going to be stuffed inside with some Death Wish coffee, because I mean survival, right? Bacon on the go. This is individually wrapped bacon that's fully cooked and flavored that you can just pop open and leave. And then we got this multi-tool here, the Geek Key. And then not pictured here, you're also going to be getting a Ninja water bottle. So you'll be able to hydrate, caffeinate. Those have to go in them. I think we have to put some kind of disclaimer on there for sending out the toe of Satan. I don't know. We may have to talk to legal about that. We'll see. Okay, so let's get into this. And Matt, I'm going to bring you back on the screen in just one second. Let's look at the poll, guys. And let's leave this up for a second, just so people can see the options here. And I'm going to bring Matt on and let you guys take it away. What was the worst IT nightmare this year? We were talking in advance of this, too, that man, every year is rough in IT, right? But this one, I don't know what it is. This one just has felt especially rough with and for a large part due to these options that we have on screen. So interesting to see what people are saying in the polls. It looks like the Microsoft Exchange vulnerabilities are actually coming out on top. They're edging out the print nightmare option. I want to see who else is saying what are their options here in the chat. But let's just pull this down, get Matt back on the screen. And guys, why don't you answer this, too? Jason, I know you have some experience. I mean, you had to deal with all this stuff, obviously. What was the big one for you? Yeah, I mean, so we don't let our clients run on-site exchange. So I only have one co-managed customer that I deal with. So that was kind of a non-issue for us. But print nightmare, man, it's just like the flipping gift that keeps on giving, right? Like it's a thing, you can get popped by it. And then like literally every week, there's some sort of update or thing. And I think we've completely broken printing for clients at least five or six times throughout the year. And in the most recent time, we don't even have to do anything, just patches installed. And now none of their print drivers work. So it is definitely, definitely been over the past two months, probably 30 to 40 percent of our service time has been printer related issues due to print nightmare. So it's been pretty terrible. Wow. And I mean, and you were saying, too, that you actually took the hard line. You're like, yeah, well, no one's going to print. Yeah. When it first came out that first weekend, we're like, oh, they'll have a patch Monday. So we're like, we're just going to turn printing off over the weekend. Needless to say, they didn't have a patch Monday or the following Monday or I think like several Mondays following that. So, you know, then we're in like crazy workaround hell and it just wasn't fun. No, I mean, dealing printers, never fun, right? But I mean, this is this is nuts. I see Tom Lawrence, you're nodding your head there, too. How's it been for you? You know, I think we now get anxiety whenever there's a ticket open with printer and it's just this year's been really rough. It was always bad before. No one wants to deal with the printer problem. But now we know that we may have been partially causing it by patching and updating. So now it's like, which printer problem is it? The one that the printer companies created with stupid drivers or is the one Microsoft created? Either way, it's my problem. It's not going to be fun to fix. Yeah. Kelvin, how about yourself? I mean, would you rank that one top for you or is it something else? For us this year, actually, the printer nightmare wasn't that much of an issue. A lot of our clients are completely cloud only. So we were lucky enough to use external tooling, no print servers, all that kind of stuff. Printers weren't really an issue, but still some things broke with patches. I mean, it was unpreventable. Some things start breaking down. So printers are somewhere at the top of the list. But for the entire industry, I'd say the Casio ransomware industry was a bit like, holy crap, this just happened. The largest ransomware attack in history was performed. Well, thanks to MSPs or at least to MSP clients. So it was, of course, a problem with Casio software. But generally speaking, MSP clients got hit. So I think that's the thing that had the biggest impact this year. But regarding ticketing and actually bothering the workflow, definitely printers, definitely. Yeah, I mean, the Casio thing, we of course, Ninja being primarily an RMM there, too. I mean, we had a lot of questions that were answering from partners and had some webinars. I mean, it sent everyone for, you know, security is always top of mind, but it's just one of those reminders of like, this is the stuff that keeps you up at night, literally. And so it really rocked our world for sure, too. You know, looking in the comments here, I see a couple of things that came up, supply issues. So even going back throughout the pandemic, I don't know if you guys have been seeing that kind of throughout, but getting your hands on the right hardware. Any hardware, yeah, I mean, it doesn't have to be the right hardware. Yeah, exactly. There's devices we still can get and they're saying like, oh, yeah, the chip shortage has been slowly being resolved. So you should be able to order new machines. Yeah, you try ordering good, decent hardware right now. Yeah, and then the other thing I see, too, is a comment that Chris made, insurance, which I mean, that touches everything, right? Like, I mean, this is something that comes up with all our partners all the time is that the insurance companies realize, well, the cyber insurance stuff, we're just getting hammered. We've got to up our requirements. And so he's saying some changes on that. We're not qualifying for insurance, too. I mean, I've had a lot of people I've been hearing about that. They're not able to get it. And that's a huge business problem. Yeah, absolutely. Three or four underwriters left that are willing to write an MSP at this point. Wow. Yeah. So just I mean, I've heard applications going from, you know, a couple of years ago being these one, two page things. Now they're five pages and getting very, very intensive and people struggling with with race getting erased and all that good stuff. My business manager formerly came from the insurance industry of 26 years. And so he's got a lot of context there. Basically, the insurance industry said there's money in them. Computer guys, we can we can make a bunch of money insuring. Oh, we didn't really mitigate the risk. We didn't think about the entire scope of what this actually means. And of course, it's that kind of feeding it, because once there was more money from insurance to pay out, the ransomware actors go, hey, look at that. These guys have insurance now they can pay. My favorite nightmare one was if you remember that incident in Florida where they actually called, they hacked the city and then called the insurer. They didn't even tell the city people. They said, I'll be sure we already know this. We're saving a step. Where are you? We already hacked your clients. Pay us. Oh, that's like that's efficiency. When you contact the insurance provider. Yeah. You know, you know, the ransom people, you know, the limits, too. So you can ask for like exactly as much as exactly exactly. I didn't pull the policy off a computer. Well, this is actually fun, fun twist here. We dug into this a little bit reading about it. And we pretty sure we don't know for sure from we didn't talk to like the FBI who investigated this. we didn't talk to like the FBI who investigated this, but we deferred based on our own digging through some open source intelligence is, you know, what's in city council meetings, your insurance coverage, it's it's voted on. And so you can just go look around some websites, read some city council meetings, you get some information. And you're like, Oh, they're insured for this much. I mean, think about that for a second. And then other cities have council meetings, you can search for this do something loaded PDF, it was kind of like, wow, I bet this is how they did it. Like to get that knowledge. Yep, yep, absolutely. Well, now I want to give you a chance to jump in here too. I mean, so you are with an MSP buying technology of New York out of Long Island. And yeah, I mean, did you just get my personal address location? That's it. I just talked to you. Yeah, this for a second, because I've been speechless. I've been very quiet so far, because I can't get my eyes off of Tom. He is so in character. I'm intimidated. I'm like nervous right now. I'm going to say the wrong thing. You're not gonna see the whole thing. I got the belt buckle guys, everything. Oh, we don't need to see the belt buckle. Yeah, all right. So I work for an MSP based in New York City. I'm physically located on Long Island, but actually in Amityville, where the whorehouse is. We'll theme there. But you know, and I've been removed from hands on it for five or six years now. So I think that the way I see things is going to be a little bit different, because you know, I'm not getting really into the work. But and everything that went on with exchange in the past year has just given me flashbacks of when I did run IT departments. And it's it's nauseating, it's anxiety inducing. And, you know, to say have a conversation with Calvin over something in the evening that's going on, and we will chat. And I, I have not had that sense of paranoia or anxiety since getting out of it. I think that's natural for that that role. And I mean, between that and just Kaseya, I mean, that was a very large, publicly visible incident. I think that's the big part of it. It raised awareness to a lot of companies when the MSP was in both good and bad ways. And I think that there will be benefits that come from it that help people. But yeah, I mean, those were big things. 2021 just beat people, you know, on multiple fronts. It's not over yet, my friend, it's still coming. The countdown is still going. Oh, the fuse is almost at the bottom. It's gonna explode soon. But yeah, I think that it's a tie between Kaseya and exchange for me. The printing stuff. Yeah, if I was still physically actually working and dealing with big production printers still, I would have probably lost my mind. Literally just been done. But yeah, let's get into some horror stories, man. I got some good stuff. Yeah, yeah. Well, okay. Well, then I think you should probably take this on. You can kick us off here. I mean, I will give us a little bit of a preamble here that like I mentioned before, in the chat, if you've got a good one that you're sitting on, let us know and we can bring you on to share too. But what I love about this panel, Matt, you were mentioning you've been out of hands on work for a little while now. But you told me some of your stories. And what I like is an ongoing theme is I think we're all I think yeah, good. So I guess also a lot of physical harm, which I think we're all here for, right? There's some fire getting hit by buses, it happens, right? Wait, hold on. Getting hit by buses. That's the one you have to lead with. No, no, no, that that's an offline conversation when the bus but uh, yeah, if you want you want to get me in the full screen so people can get the full effect. Ooh, look at that. Alright, so let me get some notes. I had to get some notes together because it was a big deal. But this first story took place between December 2000 and November 2001. So it's so far back, I can sit here and laugh about it. And I hope that people get some laughs from it as well. It's my first job. I was there for about 10 months and the CIO left. I was his assistant and we had a new CIO come in. And he just you could tell from the beginning wasn't a good fit. And it was really bad because the president goes, Hey, this is this is Callahan. He's really nerdy. It's gonna work. That's not a qualification. Show me something show me experience, especially. I'm like 20 years old. I'm like, No, I want to work with someone that knows what they're talking about. I'm sorry, we'll just call him Callahan because that's his last name. He probably isn't even around anymore. So he cares. Um, you ever work with someone that always tries to connect with you at a level that's more than it's not professional, like buddy, buddy, and you're just like, Sorry, it's not happening. This guy came to work one day dressed like me, like in jeans and in a shirt. And it was like the equivalent. He's like 25 years older than me. Just get that out there. It was like the Steve Buscemi. Hello, fellow teens meme. You know, and it's like, All right, stop it. Stop it. So anyway, one day he's talking to me. He's like, What are you listening to? You always listen to music. I was like, I'm listening to Poison the well. What do you want from me? I go back to work. The next thing he shows up wearing a poison t shirt, you know, like Tommy Lee poison. And I'm just looking at the guy like, Come on, man, this is getting really weird. So anyway, a couple weeks go by, I get into the office, can't get onto the internet. You know, I look at the smart jack for the team one, I look at the CSU, I look at the router, everything's looking good. We were running Microsoft proxy server back then. That's how long ago it was. And there was zero disk space on the machine, because this genius decided to install LimeWire. And when I went poking around, I realized what was going on. He had downloaded all of Poison the Wells catalog, Limp Bizkit, which was the thing he was rocking left and right at the time. And you know, that ended up in a full rebuild of everything to get internet access that really kept us down for a while. And that was just the beginning of Callahan. But I mean, think about that you've got a CIO who decided to install music sharing software in a proxy server. That's just a nightmare. I'm looking at anyway, okay. So anyway, a few weeks go by, everything's cool. We got a great donation of brand new cubicles, like really high end steel case ones. And I was asked to go ahead and clear out the space, work with the contract, do all this great work. And as I was learning a little bit more about the building, I learned that someone had actually been murdered on our floor. And we're on the 29th floor of a building on Wall Street. There was some kind of financial company that was in there before us. They had a like a gym and there was an argument and someone got their skull crushed with a weight. And, you know, that led to us getting a discount, apparently. So that was great. Anyway, this story's in my head. I decided, you know what, these cubicles are beautiful. I want to make sure that they're properly hardwired into the network. I got the network drops done. I'm gonna, you know, punch everything down directly into them. I'm like, three in the morning, I've got music playing, I'm just doing what I gotta do. And I hear this kind of type sound. And I'm like, whoa, whoa, whoa, what was that? What was that? All right, it's the door. That's fine. All right, let me get back to work. And you start hearing like footsteps that were not right, like the gate was off, something's wrong. And I'm thinking, oh, man, well, someone got killed up here. It's got to be a ghost, right? I mean, I'm done. I'm done for. And I'm hanging out behind underneath one of the tables for the cubicles I was building. And I see a shadow underneath. At this point, I'm really getting nervous. And I just blow my fist up like this, like whatever's gonna happen, it's going down, it's just gonna be what it is. And the next thing you know, there are two legs in front of me at the desk. So I just swing away. I mean, like, full on. What's the guy's name from from signs? The whole swing away thing. Anyway, I nailed the dude in his crotch. And he starts to fall. But he doesn't just fall. He hits his face into the desk surface right here. Just like that. And then he falls back. He whacks the back of his head into the wall and then finally lands. I'm like, Oh, hey, Tom. Oh, shit. Now you got his whole name. Excuse me. My fault. Who's doxing who now? Yeah. Oops. All right. He's a nice guy. No more about him. Actually a lot more about him. But so yeah, he ended up getting a concussion and a serious groin injury because he decided to walk into the office three in the morning and see what was going on. You can tell this dude was a mess. Now the last part of him, let me just get into my notes because I don't want to say anything wrong. This is when you actually put him out of his misery and you murder him. Is this where this is going? I mean, kind of, you know, he ended up getting our entire cabinet in the data center hacked that we had some stuff in and it took like 20 hours to get that back up and running. And ultimately we knew he wasn't going to be a good fit for the company and nobody wanted to be the one to let him go. Think about that for a second. You're in a company, you've got a COO, you've got a CFO, you've got all these people, very nice people, by the way, I'm still in touch with a lot of them. And they were like, yeah, we're afraid to let this guy go. So I was like, the hell with it. I'll do it. I'll take care of it. And the CFO gave me his credit card. I took him to lunch and he didn't come back with me. And that was the end of it. After that, they let me have his office and they were cleaning it out. And I found something like 15 or 20 empty cups of miso soup, not empty, sorry, full containers of miso soup, like you'd get from like a takeout restaurant in his desk and all sorts of stuff. So we ended up just literally throwing everything in the office out and starting over because he was such a nightmare. I mean, he was a nightmare. And the thing that he said, and this is kind of like how I want to end the story, is we were in a meeting talking about a lot of things that were just not working out properly. It was a big full staff meeting. And he actually said, if I wasn't so good at my, if I wasn't, what was it? If I was, I want to get this right, I don't want to mess up. If I was, if I wasn't so good at my job, you wouldn't be so good at yours. Basically saying I'm terrible at what I do. He admitted that in the middle of a meeting, you know, and I think that sealed the coffin for him. That was like the final nail. And then that was the end of the Callahan saga. Um, how much more time do I have here, Jonathan? I got some funny stuff going on. Intentionally? Well, I think you're going to have a chance. Yeah, like, so so I'm going to get to the the machine on fire, the falling through ceiling stuff. But I think we need to pass it on to another person right here and come back to it. Man, ghoulish story of of a co you know, a coworker that wasn't working now. And I guess any other lesson learned other than like, having to like, make some hard decisions sooner? I well, I think the lesson learned from that, I mean, I was really young, I was not in a place to be able to make really big decisions. But you know, if there's something that you really love doing, you stick through it, you make sure it works out. And I think that the large, the big lesson I learned is that age doesn't always equate to wisdom. That was the big part for me. He's a CIO, he's accomplished, he's been doing this 2530 years, but everything is breaking. And I think that also is a lesson that, you know, technology education is important. It's ongoing, it's nonstop. And the moment that you decide you don't want to continue learning is the moment you become talented. And that's always, it's always been in the back of my head, and it's always driven me to learn new things and do other stuff. Because the sense of stagnancy in expertise with him is really what, man, it was just, it was like Spidey sent central, you know, you started realizing it left, right. But yeah, man, go to someone else. I'll come back with some fire stories. I do have some fire stories. All right, sounds good. How about Kelvin? So I only have like nerdy horror stories. And this is the one that always, that the one that I carried with me a long, long time was because it was pretty much one of the biggest mistakes I've ever made in my IT career. Actually, two of them. Same client, co-managed client that specializes in big data. And these clients are amazing, because clients that specialize in big data are often nerds themselves. So they want the best technology, they want the best kind of stuff. They know a little bit, a thing or two about tech. So the first thing that happened when we took this client over was, they said, Okay, hey, we want a new data warehouse server. It's for development purposes only. And that was already a trigger I should have said, No, it's not. You're going to run production on it. I know it. But they were like, Okay, you know, it's for development. And this was in 2008 ish. And they said, We want to fully stack with SSDs. And 2008 and SSDs was pricey. And then they went like, we want to stack with one terabyte SSDs. So we went to Dell, asked for a quote for one for a single one terabyte SSD. Dell in 2008 ish went like, Yeah, that's going to be $1,000 per SSD. So the client, of course, blown away. So yeah, that's not that's not possible. So one of the things they figured themselves was, okay, you know what, just order the server with just normal disks, and we'll replace them with other disks. That was already trigger number two, where I should have said, No, stop, what you're thinking right now is not going to work. So eventually, we did order the server, because you know what, it was still like, we weren't completely managing them, it wasn't going to be our problem, we'd still make some money. So we decided, you know what, let's do this. Ordered the server, they came to our office with their SSDs. Oh, my camera just dropped out, I think. They came to our office with the SSDs. And that was the moment where I was like, okay, they came to the office with the first Samsung Evo SSDs that were just released back then. So that was already one of the moments like, okay, you know what, this is just going to be terrible. This is this is going to end poorly. I everything, all the SSDs installed, and the server we bought was like an R710, or R700. So a Dell server with 24, one terabyte SSDs. We started putting in the disks, we installed everything, and that worked perfectly. It booted, everything worked. So we installed it into the data center. They started working on it. About two months later, they called us, hey, the server is no longer working. Can you check what's happening? So that was actually a data center that was a two hour drive away. So back then, I was still like a technology technology guy. I decided I'll get in the car and a two hour drive in the Netherlands. I mean, I know most of the people here are Americans. A two hour drive in the Netherlands is like going from Texas to California for you guys that that that's the equivalent here. So I got into the car, drove those So I got into the car, drove those two hours, two hours terrible two hours, brought two lunch bags, stopped five times to get a meal, all that kind of stuff. And eventually got in their data center. And I was shocked. I connected the monitor and I was shocked the first thing I saw. I looked at the monitor and it said rate failure. I was like, okay, you know what? One of the disks failed or something and it just didn't boot. And I rebooted the server and suddenly magically it booted again. So I was like, okay, what's going on here? Rebooted the server again and again, rate failure. So it turns out that putting 24 disks into a Dell R7 10 or 700, something like that, but just a big box with lots of disks. The rate controller will actually time out at the moment that you have that many SSDs in there because it's expecting the disks to be slower, to spin them up and to get access to them. And if it gets instant access, the rate controller will actually just choke itself. So I rebooted the server one more time and poof, all of the data was gone. All of the data was gone. That 24 terabytes or those 24 disks, one terabyte per disks, 12 terabyte usable space, all of it was gone. And I was like, holy crap, what just happened? I called the client, explained to them the situation. They said, yeah, it's no problem. It's a development server. I called Dell, they'll actually explain to us, yeah, this is something that happens with that controller. There is an internal firmware update we have available for it, but you'll have to install it before you lose any data. So eventually we spoke to the client again and they were like, yeah, we don't have backups of that server. And actually our biggest production database was on there. And that was the moment that, yeah, we started shouting at them a little, getting into, well, yeah, standard client, semi-co-managed MSP stuff. So yeah, that was my horror story. No backups, rate failure, completely unsupportive configuration in each and every way. I mean, there was no way you could have figured that this would have been supported at any point in time. So yeah. Wow. And Kelvin, hopefully we can get your video back on, sir. But man, like, where do you even go from there? I mean, is it just like... Home. Home. Home. You go home, exactly. You take a two hour drive back home and you sit down, you stop a couple of times, you take a meal, and then you come home and just, yeah. I just imagined Kelvin being like, forget this, I'm going to Applebee's and walking right out the door. That's actually, so I was actually really like, okay, I'm not going to curse, but I was like, fudge this, I'm going home. This is not my problem. Well, then actually, okay, so that's going to go right to our rules for survival too. I love that that kind of teased that up. Kelvin, thank you so much for sharing that. And before we jump over Tom, Tom Lawrence, I'm going to see you up in a second, but let's see if James wants to come on because James in the chat was hitting that he has a good story and Tom was saying like, yeah, let's get James on here. So Kelvin, I'm going to give you the boot for just a second while the video issue is happening to you. And James, I'm going to invite you on. And while we're waiting for you, James, Tom, do you want to tee up your story just a little bit? Give you a little teaser. Me time, make sure I got the time. Yes, I know, I'm sorry about that, Tom Lawrence. All right, Lawrence. Yeah, I made a video about this and it's a little bit longer because I have a lot of visuals in it, but we'll do the worst cabling job ever. We do a lot of physical infrastructure and this one is just, if some of you may from all my channel that you've seen the video, it's got a lot of views, but yeah, it's definitely going to be one to talk about. Yeah, well, actually, you know what, let's get into it. I've invited James on here, but while he's connecting, let's go ahead and get into your story. I'm going to give you the spotlight here. Okay, this was a lot of fun because I like that we have contractors and we do infrastructure. It's something we started doing years ago. I know it's not something every MSP does, but still the rules apply of when you manage a project of scope creep, managing expectations and dealing with crazy people. And the crazy people part was what made this so much fun. And the way you mitigate some of these problems when someone has unreasonable requests is not always by saying no, that's the right answer. But I still did that other answer of, I'll just say a price that no one should pay to get this done because it's a bad idea. And then they paid it. And you're like, well, I guess I have to do the thing they said. They're like, we want to complete it by this unreasonable date. I'm like, that would require bringing 13 extra contractors. And they reply with, well, how much will that cost? Okay, this much. So we started out as a simple wiring project we did. We did know that the people had originally bid it, dropped out for unknown reasons. We also know talking to the GC who was the replacement general contractor. These are a lot of red flags, by the way. And this is about 200 drops of cat six, a platinum. This is a decent size project, but we know that the GC was new and we also knew the electricians were new. And then there was this suspicious pile in the back fenced off of all brand new electrical boxes that were ripped out. And I said, why? Those look like brand new. Why are they being ripped out instead of being put in? They said, well, they underspec the original electrician before they fired them. They underspec the electrical store where the new electricians are replacing it. Like there's a few red flags, like this project may not be well managed, but we keep going. So we get the bid and then the walls are going up and we are starting to wire the walls before the drywall's on. And then we come in one day and the wall's gone. And we're like, what happened to the wall we wired? And they're like, oh, we moved it. So they would build another wall. And they're like, we have to charge you again to wire it. Yeah, we just cut all your wires. So we pull them all again. By the way, these are a 30 foot ceiling. So we've got boom lifts. We got everything. So my crew rewires a new wall and then they moved it again. And then they moved an entire, there was three IDFs. They moved one of our IDFs. They just decided it was going to be something else. And we had our cable falls in, but we hadn't put the racks in yet. And they, once again, just said, it goes this way now. They dropped all the wires. They actually took our J hooks down and just dropped it and left a giant pile. And I got some of the huge, we wasted thousands and thousands of feet of cable that was already run that they would cut back. But my favorite one is one, these are multiple locations that were being built. We were handling the one that's in our area is the CIO comes in, he flies in. I'm not there when he gets there. And he's wandering around pointing about things he wants done and just how he thinks the place should be run differently. Whatever, it wasn't like an official meeting. But then he grabbed a can of spray paint and he cut the server room in half. He just drew a line with spray paint. So this is the new wall we're putting up the server room. The problem is we've already installed all the ladder rack at a certain height. And it went across for the cable falls to come to where the servers are. The split AC units were already installed in the room. He drew a wall to put it on the other side of the split. And we're all looking at each other when we seen, because we didn't know what the paint was on the floor. And we're like, this is where new wall's going. We're like, well, the wrong part of the room where the servers are going won't be AC. And the smaller closet he made will be AC. And AC guys are like, good, not my problem. No one told me to move it. So it was like one thing after another after another that kept happening. One of my favorite ones is the drop ceiling move. They decided arbitrarily to take the drop ceilings and move them down. This is after fire and safety had come through and put all the, if you don't know how it comes to a drop ceiling, like the fire things, they come through the drop ceiling and they cut them to height. They had to, when he moved the drop ceiling, they had to redo unpressurized fire and safety, drain it, which was a mess, recut the pipes and move them. And then we had to remove all of our racks and everything again to the new adjusted ceiling height. But some of the lessons learned we had in there that was really every step of the way, we took a million pictures. And everything I got in writing, I would verbally say something and the company need to try to avoid paying things and always not want to reiterate it back. But we would always force them before we'd actually show up to give us more than the usual deposits, or even because we quoted so much for the extra contractors, was we were just like, this is what you gotta do. You gotta have to pay us. We would refer to them as another mobility thing, where we're like, if you wanted to get us deployed 13 more contractors than were originally done, this is how much it's gonna cost. We need the money now, because we don't trust you're gonna pay us in. We were like as bold and honest as, we made the situations awkward with them. Like, we don't think you'll actually pay us because I gave you a price that's not reasonable. And they said, well, we need it done. And we want this, we have an open date we wanna meet. So it was kind of fun. And then when the open date came, only half the building worked because we got it wired. We did our part. We did do some stupid things. Like we share the people who did the drink fountains, cause this is part of a family fun center. The tubes underground that come up. So you have like standalone kiosks. The cat six cable is run along next to the Coke machine, how they have all the little soda cable stuff. It's all in the same tubes. But some of the electrical stuff was just incapable of being done by the time they open. But in the end, although it took a long time, they paid us 90% of the bill. And the last 10%, they trickled over like six months, which took a while before I did the video because they actually paid. But having everything in writing meant every time they wanted to argue with us, we're like, here's a photo, here's what you said. Here's what happened. Here's the updated blueprints. Here's all this. I mean, we were taking pictures as they marked up blueprints and told us different things, but it was one of the most complicated, for no reason other than people kept changing their mind, jobs, but in the end, we actually made money out of it. That's the part that blew my mind. Everyone hates that job and no one wants to talk about it. We actually put a lot of hours in to make it happen, but everyone got bonused out for it too. So it was kind of like, it was rough. It was a rough one. I said, I have a lot of visuals in my video. Oh man, yeah, you should definitely drop a link to the video in the chat so we can check out those photos. You're also a great follower on YouTube, of course, and Twitter, because you're always sharing amazing photos from cabling jobs and things like that. I love this story because we hear often, people will say, oh yeah, just charge, come up with an amount that is stupid, that's a dumb amount of money, and if they pay it, then somehow it'd be worth it. But here, in your case, it ended up being that way, but I kind of wondered too if, I'm sure you regretted it multiple times. So, so regretted it. We were there till three in the morning a couple of days. We put in 16 hour days to complete it, but we billed them for it. It was like, oh, I'd work for that much, but when you're actually working for that much, you're like, no, no, I don't, I'd rather be at home. I wanna, there's anything I'd rather be doing than this, like, because this is cable crimping. We have people in my office who don't crimp cables, crimping cables, because a lot of it was just testing, not necessarily crimping, but there's so many drops, they all wanted them tested, so I'm like, you guys have to run around and plug these things in, sit around with label printers and put stickers on things. Like, how long are we doing it for? Till it's done. How many are there? I'm like, 240 drops. Amazing. Oh my God. Well, hey, Tom, thank you for sharing that one, and yeah, definitely drop that link in. And James, thanks for sitting tight, buddy, and thank you for coming on. So tell us a little bit about yourself before you dive into your story real quick. Sure. Yeah, you guys are flashing me back, or in particular, flash me back to my MSPPM days. It's one of those projects that's like, every day is what's it gonna be today, and that's one of those things where you come in and like, hey, we moved the IDF. No big deal, right? Yeah, so I am currently the, I feel, I don't even wanna admit this with all the talk about CIOs that you guys have been talking about, but I'm currently the CIO at a behavioral healthcare organization in upstate New York. Before that, I worked for several different MSPs in various roles from engineering to project management, as well as some VCIO consulting. And then before that, and where this story will take place, I cut my teeth originally in a small town hospital setting in healthcare IT, coming up to like the whole technical ranks. Started when I was 17 as a help desk guy, you know, like lowest man on a totem pole, kinda, and then kinda worked my way up technically from there. Wow, nice. All right, well, I can't wait to hear this story. So yeah, let's get into it. Yeah, so I've never told this story on, I guess in virtual before, in virtual format, we'll see how it goes, but there's gonna be some charades that happen. But I guess some things you need to know to start the story is that, like I said, this is hospital IT. I'm probably 18, 19 years old. I'm like a help desk, level one help desk guy. I'm probably about a year into the job or so, and we work in really tight quarters. So I sit facing forward, behind me is kind of like a junior engineer, and then behind him would be like the senior engineer, senior analyst, and the director's office is somewhere off to the side. So when help desk calls come in, it's not infrequent for me to take calls. And then, you know, if I have a question, someone can lean over and be like, oh, you know, help. We can help from behind, basically. So, and like I said, very tight space. We're crammed in a hospital basement. I'm sure anyone who's worked in healthcare understands. And then there's always the stigma of, for whatever reason, people in healthcare, I'm not afraid to admit it, something about technology and healthcare, just they don't love each other. But the, so the story starts where I'm, I pick up a help desk call, and it's from the emergency room. They dial in on a specific help desk line, and in particular, because there's a bunch of things that we support. We have obviously an EHR and an IT division. But I pick up the help desk call, and it's a nurse that is known to the department. You guys know where I'm going with that. And she says, hey, I'm having some trouble. You know, can you, whatever. I forget what the issue was. It's not important. But hey, can you sign in and take a look? And I said, sure, I'm happy to. At the time, we were using VNC viewer as the remote tool you had deployed internally. So, so I said, sure, okay. You know, well, I gotta get in, and I, you know, I said, what's the asset tag? And of course, you couldn't find the asset tag after 10 minutes of gesticulating. We were unable to secure an asset tag. So, all else fails. I said, okay. And you know, in the bottom right-hand corner on your toolbar, you have an icon. It's blue, red, and green, says VNC. Can you hover your mouse over the icon for me? A box will pop up. You'll have a bunch of numbers, it was an IP address. Give me the IP, I'll get in that way. So she said, okay. And I said, great. I said, so this is what the icon looks like. You see it? She said, yeah, I see it. It's blue, red, and green, says VNC. I'm like, that's the one. Put your mouse over it, let me know what happens. She goes, okay. So I'm sitting there, cue Jeopardy music. I'm like, okay, you should see a little box pop up. What do you see? And she said, nope, nothing. And I said, okay. You know, I said, all right, well, why don't you try moving your mouse off it? You know, come back on. Again, this is what the icon looks like. It's these colors, it says this. You see that icon, and she calls it out back to me. She goes, yep, it's white, it looks exactly like this. I know she's looking at the right thing. So I was like, okay, you know, hover the mouse over the icon again. Let me know what pops up. Now, we've been doing this for like 15 minutes at this point. You know, we're well past the point of my, you know, if we weren't jammed from a project, I would have already sent the desktop guy just to go in person. It's, you know, whatever. Probably something's gonna knock out in three minutes anyway. So the guys behind me at this point are, you know, they've figured out who it is. They know, they see the struggle, and they're somewhat entertained by this already. So she goes to hover the mouse over the icon again, and she goes, you know, I'm like, what do you see? And any pop up? And she goes, no, no pop up. So we try a few more things, and long story short is I'm thinking she's either, she's either not hovering the mouse over the right thing, even though she's telling me she is, or whatever. So eventually I said, okay, you know, I'm gonna rip a project guy off of a project, and we're gonna send the desktop guy to come get you. They were off wiring and sealing somewhere else in the hospital. You know, this is 25 minutes deep. Yeah, for sure. So I call, I said, okay, well, I'll tell you what. You know, I'm gonna have someone come down to you. Just sit tight, you know, thanks for your patience. We'll get someone right over to you. We'll take care of it. And she goes, okay. And I was like, all right. So I go to hang up the phone, and as I'm hanging up the phone, I hear her go, hey, hey, James. Like, put the phone back up to my ear. I'm thinking maybe she got the pop up. And she goes, hey, can I put my arm down while I wait for them to come up? And I said, it struck me. It took me aback. I said, can I put my arm down? And then I just realized I'm holding the mouse to my computer. I take and I hold, and I just did this. And I just held it up to the screen. And I realized that's what she was doing. She was holding her mouse literally over the icon. So I hold my mouse up to the screen, right? And I kind of just ineptly did it, just thinking about it. And then, you know, hover my mouth almost immediately. The guy behind me sees obviously what I just did. Spits out coffee all over his monitor. I mean, all over his monitor. Goes to like catch it, knocks the monitor off. I'm wearing shorts. Monitor falls off his desk, hits the back of my leg and scrapes the hell out of the back of my leg. Long story short, I ended up going to the emergency room to get my leg bandaged and wrapped up by the nurse that I couldn't tell why my leg had just been cut. And the reason because the whole thing. And I said, yeah, first of all, there was the hustle and all the things crashing and my stifling, you know, a scream as my leg gets sliced open. And I was like, yeah, you can put your arm down. You know what? I think I'm gonna be there in about five minutes. And so I was. And then, of course, the issue took 22 seconds to resolve total. It was a Citrix thing. Isn't it always? So yeah, so that's the story that's held up. But it's just, she's like, yeah, so I'm sorry. She's sitting there and, you know, she's literally the one, you know, doing the triage. She's wrapping up my leg while we're doing that. And I'm laughing and I'm trying to think now I gotta dance for 10 minutes around, you know, not telling the story of how I just cut my leg being an IT employee sitting at a desk. But that's a story I still tell to this day. And it's kind of what I think of as like the quintessential healthcare help desk IT story. That is amazing. As people in the comments are chiming in too. Thank you so much. That's incredible. Thank you so much for coming out for that. And James, we're gonna get you one of those IT survival facts for coming on. For your bravery. That's dope. Yeah, that sounds great. Thank you very much. Yeah, thanks for having me. This has been awesome too. Thanks for a couple of awesome laughs in an otherwise busy week. Yeah, well, hey, glad to provide. And thank you so much for giving us some too. That's awesome. Oh man, that was great. And so guys, like we blew through our time here. And Jason, I know you've got your client meeting. So we gotta kind of like hustle here. So what I would suggest is, obviously we gotta get our challenge on. But before we do that, so Jason, I know you've got some stories. I have a short one. Okay, give us a short one. And while you're doing that, I'm gonna pull up some stuff on the screen too. Okay, yeah. So this was an enterprise company that I worked at a handful of years ago. We get a call one day, they're re-roofing the data center, or actually the whole building. And it's, you know, thunderstorm comes through. We get a call one day, and I'm like, hey, like, you know, some of the servers are down and we're on call. So I roll in, I get there. One of my coworkers is mopping the floor. As it would happen, the roofer has clogged the roof drain with the roofing material, and it is raining water down on a 50 kilowatt EPS. And my coworker is sitting there mopping the aluminum raised floor with a mop. So that was like the one and only time I've ever gotten to press the big red button. So, I mean, that's a super short one. That was a, it was a good day. Like, luckily no one got electrocuted, but it is never a good day when it is raining on the EPS and the data center. Oh my God. Yeah, I mean, and while you mentioned the electrocution thing, and that's actually happened to some people that you've worked with, right? And I know it's happened to Matt, if I'm gonna bring on the screen back too. I'm gonna get him and Calvin back on. And then Tom too, like, so you got, really quickly, while I'm pulling this last thing up, and then we got a hustle here. You wanna, do you have a story, anything about like, I know you have one of your guys falling through a ceiling at one point. Oh yeah, I had this one guy, he would fall through ceilings, drop those big corner, like cubicle desks on himself. And then probably the greatest was, just one day rolls into my office. He's like, listen, I gotta go to jail. I'm like, come on, man. And he had broken down a door at a bar and not really an IT story, but I had to pull funds out and go and pay for an $800 door, so he could keep working and not go to jail. So I consider that a horror story. I would say that qualifies. Okay, I'm gonna bring Matt back on too, so we can do this challenge, but really quickly, let me just go through some stuff. I'm gonna send out this deck to people, so you can see, you can have this to keep. And also like, maybe we can continue some of this, even if it's just like, me and Matt has offered, but maybe we can do a little bit of a continuation of this, like after hours or something, because this is pretty fun, guys. But yeah, so based off our rules here, I'm just gonna breeze through these, because you wanna see us suffer through this Lollipop, right? How to survive an IT horror movie. Never assume something works until you test it, right? Always bake on restoration taking longer than you think. Jason, I mean, you called that one out for sure. Kelvin, this is one too, you never trust a user. Everyone lies, you call that house. And Tom, Lawrence, you were saying you had the house poster in your office, actually. Yes, we do, and it says, everybody lies. I remind all my help desk people. Yep, yep, the next two are the classics, no local admin, no exposed RDP, right? Then we've got some more fun ones. Never make changes on a Friday. Actually monitor AV and EDR alerts, especially on Friday. Never assume risk over stuff that's not yours. That kind of came up with some of the things you were talking about. This next one definitely did. The document everything. People in the comments responding to Tom Lawrence's story with the CYA comments too. And then the don't just plan for the worst, tabletop it. So guys, I'm sorry we had to kind of like rush through this stuff, but oh man, we gotta pick out our winner too. So really quickly, we asked people to share on social their six word IT horror stories. And man, you guys came through in a big way. And we got a poll down here. I see the winner by a pretty clear amount is the hey, while you're here, can you option. Let's go through here and see who's IT horror story that was. Admin, oh, that one. Yeah, that was a bad one. It looks like Cody. Cody, you're gonna be the winner of another IT survival pack, sir, for your submission there if you're on the line here. We got a couple of other good ones. I just wanna scroll through this really quickly. I mean, the DNS is classic. I love these two both a lot. Just a temporary one. There's nothing as permanent as that. Yeah, that is the permanent one. Oh, fantastic. Okay, all right, guys, here we go. Let's do this thing. So I'm gonna get Matt back on the line here. And while I'm doing that, some context for the folks at home. So the Toe of Satan described as the world's spiciest lollipop, nine million Scoville unit. Jason, I think you were saying like, what's pepper spray? It's like three million? Three million, I think. Three to six million for pepper spray. All right, so 900 times hotter than a jalapeno. What we're gonna do here is we're all gonna, let's just, we're gonna adjust this a little bit since we're running out of time. And we're going to go ahead and I'm gonna pull up a timer. I think it actually smells good. It actually smells good. It smells like cinnamon. I don't believe you there at all. There's some cinnamon in it. Are you doing it, Jonathan? I'm gonna do it too, don't worry. I wouldn't make you guys do a thing that I wouldn't do. Oh my God, no, Tom. Oh, is he going already? Icy teeth. Oh, and Matt, so Matt has already warned us. Oh, Matt. Yeah, sorry about that, guys. I stepped away to tie a damsel to some train tracks. I'm back, I'm back. Good, and Kelvin, your video has conveniently gone out too. I can't help but notice. I'll just put yourself on Twitter doing it. Yeah, right at the moment of truth. But okay, guys, let's go ahead and dive into this. We were gonna be answering questions, but I think we should just go for this and see how long we can last. I'm not doing that. I'm just gonna sit here with my mom. There you go. On your mark, get set, go. Oh, well, Tom, look right in there. Oh, I can't, I can't even. I can't, oh, wow. Taking it out of your mouth is a terrible idea. You just want to go back in. Yeah. Oh, this iced coffee is so relaxing. Don't take it out, just leave it in. It's kind of, I mean, oh my God, oh, wow. Guys. I'm gonna use these on myself. Is Kelvin doing this? Oh, God. It's just getting increasingly hotter. It's just like going around my whole mouth, like spreading. How long are you supposed to last with this, Tom? Like how? I saw the video the guy did in like five minutes. I got the hiccups. I lose. I can take this. Hiccups. This is not good. Hiccups are coming. Oh, God. Oh, this is a terrible mistake. This is really bad. Now you know why we didn't do it at the beginning. Oh, God. I gotta, oh my God. Guys, oh. Come on, guys, you got this. Keep going, keep going. The problem is it doesn't just stop when you take it out of your mouth. How do I make it stop? Oh, God. Oh, my God. I can't stop hiccuping. Guys, we're coming up on the hour and I wanted to plug a couple of things. We got MSP live chats. We're gonna have one next Thursday. Oh, my God. And you should come. I'm out, I can't do it. Oh, man. This is a terrible mistake. Guys, it's been nice having you here on this Ninja RMM live chat. Ninja won. Come on. Ninja won. Come on, come on. You got it. You can do this. Check out Tom Lawrence's YouTube channel. Jason is always on the community. He's doing amazing things in security. Tom Watson is with our live chats, too. Kelvin has this amazing project on GitHub. You should check out, Cybertrain. Oh, my God, guys. Oh, I think we gotta go. Jason, good luck with your client call. I'm crying. I'm crying. Okay, guys. This was a lot of fun. We'll never do this again. Was it? Was it? Oh, God, I'm dizzy. And I think with that, happy Halloween, everyone. Yep. My nose is clear. See you later.
