2021's Biggest IT Nightmares
The panel opens with a discussion of the year's most challenging IT incidents, with Print Nightmare emerging as the dominant pain point for MSPs. Jason Slagle notes that his team spent 30-40% of service time over two months dealing with printer-related issues stemming from the vulnerability and subsequent patches. The initial response of disabling printing entirely backfired when Microsoft failed to deliver promised patches for weeks. Tom Lawrence describes the anxiety now associated with any printer ticket, as technicians must determine whether issues stem from vendor drivers or Microsoft's remediation efforts. While Kelvin Tegelaar's cloud-focused client base largely avoided print server issues, he identifies the Kaseya ransomware attack as the year's most significant industry event, calling it the largest ransomware attack in history targeting MSP clients.
Construction Project Horror Story
Tom Lawrence shares an extensive infrastructure project nightmare involving a 200-drop Cat6 installation at a family entertainment center. The project exemplified scope creep and poor project management, with the general contractor, electricians, and previous vendors all being replacements after earlier teams abandoned the work. Walls were repeatedly moved after wiring was complete, requiring crews to re-pull cable through 30-foot ceilings multiple times. The CIO arbitrarily spray-painted a new wall location in the server room that would have left servers without air conditioning. Drop ceilings were lowered after fire suppression systems were installed, requiring complete rework of sprinkler pipes and cable infrastructure. Lawrence's team mitigated risk through extensive documentation, requiring upfront payment for additional contractors, and photographing every stage of work. Despite the chaos, the project was completed and the client eventually paid 90% immediately with the remaining 10% trickling in over six months.
Rules for Surviving IT Horror
The panel establishes survival rules for IT professionals, drawing parallels to horror movie tropes. Core principles include never trusting users ("everybody lies" from House M.D.), eliminating local admin rights, never exposing RDP, and avoiding changes on Fridays. Documentation emerges as critical for both technical and business protection, with Lawrence's construction story demonstrating how written records and photographs protected against payment disputes. The group emphasizes the importance of tabletop exercises rather than just disaster planning, actively monitoring security alerts especially before weekends, and never assuming risk for infrastructure you don't control. Jason Slagle advocates for saying no to unreasonable requests rather than pricing them prohibitively high, noting that clients sometimes accept outrageous quotes and force you to deliver on bad ideas.
Help Desk Hellfire Challenge
The session concludes with participants attempting the Toe of Satan challenge, consuming a 9-million-Scoville-unit lollipop (three times hotter than pepper spray and 900 times hotter than a jalapeño). Tom Lawrence, having attempted this previously, serves as the experienced veteran while others struggle with increasing heat, hiccups, and tears. The challenge demonstrates the camaraderie and willingness of IT professionals to subject themselves to absurd situations for community entertainment, with Jason Slagle facing an immediate client meeting afterward. The event reinforces NinjaOne's community-focused approach to MSP engagement, combining technical discussion with entertainment and prizes including IT survival packs with Timbuk2 backpacks, Death Wish coffee, and multi-tools.
