Truth in IT
    • Sign In
    • Register
        • Videos
        • Channels
        • Pages
        • Galleries
        • News
        • Events
        • All
Truth in IT Truth in IT
  • Data Management ▼
    • Converged Infrastructure
    • DevOps
    • Networking
    • Storage
    • Virtualization
  • Cybersecurity ▼
    • Application Security
    • Backup & Recovery
    • Data Security
    • Identity & Access Management (IAM)
    • Zero Trust
    • Compliance & GRC
    • Endpoint Security
  • Cloud ▼
    • Hybrid Cloud
    • Private Cloud
    • Public Cloud
  • Webinar Library
  • TiPs
  • DRAW
  • CPE

PDQ: Server Running 11 Years with No Reboots or Patches

PDQ
08/11/2026
0 (0%)
Share
  • Comments
  • Download
  • Transcript
Report Like Favorite
  • Share/Embed
  • Email
Link
Embed

Transcript


It was running a critical billing process. Nobody knew what would happen if it rebooted. Sent OS 6.4, end of life since 2020. Kernel from 2012, a custom Python 2.6 billing script written by a developer who left in 2015. It processes every invoice the company sends, 1,200 a month, for 11 years, no patches, no updates, no owner, no documentation. I called the developer. He doesn't remember writing it. I can't leave it. It's an unpatched security risk that's been sitting exposed since 2020. I can't touch it. It's 4,000 lines of uncommented Python 2 and nobody alive knows what it does. We spent six weeks reverse engineering it, tested the reboot 37 times in staging. Then on a Saturday morning with the CEO aware, we rebooted it. It came back up in 40 seconds. Everything worked perfectly. 11 years of fear for 40 seconds.

TL;DR

  • A sysadmin discovered a Linux server with 4,018 days of uptime running an undocumented, unpatched Python 2.6 billing script that processed every company invoice for 11 years.
  • The original developer, who left in 2015, was tracked down but no longer remembered writing the 4,000-line uncommented script, leaving the team with no documentation or owner.
  • After six weeks of reverse engineering and 37 staged reboot tests, the team successfully rebooted the server — it came back up in 40 seconds with everything working perfectly.

Summary

This short-form video, sourced from a Reddit sysadmin story, recounts the discovery of a Linux server with 4,018 days of uninterrupted uptime — nearly 11 years — running a critical billing process that generated 1,200 invoices per month. The server ran CentOS 6.4 (end of life since 2020) on a 2012 kernel, executing a 4,000-line uncommented Python 2.6 script written by a developer who had left the company in 2015 and no longer remembered writing it. With no patches, no documentation, no owner, and no understanding of what the script actually did, the sysadmin faced an impossible dilemma: the server was an unacceptable security risk, but touching it risked breaking the company's entire invoicing operation. After six weeks of reverse engineering and 37 staged reboot tests, the team rebooted the server on a Saturday morning with the CEO present. It came back up in 40 seconds and worked perfectly — resolving over a decade of institutional fear in under a minute.

Chapters

0:00 - The Discovery
0:07 - The Technical Debt
0:26 - The Impossible Dilemma
0:40 - The Resolution

Key Quotes

0:05 "Nobody knew what would happen if it rebooted."
0:29 "I can't leave it. It's an unpatched security risk that's been sitting exposed since 2020."
0:34 "I can't touch it. It's 4,000 lines of uncommented Python 2 and nobody alive knows what it does."
0:53 "... 11 years of fear for 40 seconds."

FAQ

Why didn't anyone reboot or patch the server sooner?

Nobody knew what would happen if the server rebooted, and the script had no documentation or living owner. The risk of breaking the company's entire invoicing operation outweighed the perceived urgency — until the security exposure became undeniable.

How did the team finally feel confident enough to reboot it?

They spent six weeks reverse engineering the 4,000-line Python 2.6 script and tested the reboot 37 times in a staging environment before proceeding on a Saturday morning with the CEO aware and present.


Categories:
  • » Data Protection
Channels:
News:
Events:
Tags:
  • Security Operations
  • Vulnerability Management
  • Best Practices
  • Short Form
  • Getting Started
  • Legacy systems
  • Technical debt
  • Patch management
  • Server uptime
  • End-of-life software
  • Sysadmin horror stories
  • Undocumented code
  • Linux administration
  • Incident preparedness
Show more Show less

Browse videos

  • Related
  • Featured
  • By date
  • Most viewed
  • Top rated
  •  

              Video's comments: PDQ: Server Running 11 Years with No Reboots or Patches

              View CPE Certificates

              Upcoming 360 View Events

              • Nov
                19

                Securing the Microsoft Enterprise: Strengthening Configuration Management, Identity Security, and Data Protection

                11/19/202601:00 PM ET
                More events

                XStreaminars (watch here)

                • Nov
                  12

                  Leaving VMware Means Leaving vSAN: Your VMware Exit Is Really a Storage Decision

                  11/12/202601:00 PM ET
                  More events

                  Industry Events (Sponsor Hosted)

                  • Nov
                    12

                    Hands-On Workshop: Connecting On-Prem Data to Databricks Using MinIO AIStor (Americas)

                    11/12/202601:00 PM ET
                    • Nov
                      18

                      Practical Session: Connecting On-Prem Data to Databricks via MinIO AIStor

                      11/18/202608:00 AM ET
                      • Nov
                        19

                        Zendesk Deep Dive: Get the work done faster with Employee Service AI Agents

                        11/19/202601:00 PM ET
                        • Nov
                          19

                          A Deep Dive into Muse, Dots, and Duet's Impact on Your Website

                          11/19/202601:00 PM ET
                          • Dec
                            03

                            CISO Insights: Steering Cybersecurity Strategies in an AI-Driven Era

                            12/03/202601:00 PM ET
                            More events

                            Upcoming Webinar Calendar

                            • 11/12/2026
                              01:00 PM
                              11/12/2026
                              Leaving VMware Means Leaving vSAN: Your VMware Exit Is Really a Storage Decision
                              https://www.truthinit.com/index.php/channel/2192/leaving-vmware-means-leaving-vsan-your-vmware-exit-is-really-a-storage-decision/
                            • 11/12/2026
                              01:00 PM
                              11/12/2026
                              Hands-On Workshop: Connecting On-Prem Data to Databricks Using MinIO AIStor (Americas)
                              https://www.truthinit.com/index.php/channel/2202/hands-on-workshop-connecting-on-prem-data-to-databricks-using-minio-aistor-americas/
                            • 11/18/2026
                              08:00 AM
                              11/18/2026
                              Practical Session: Connecting On-Prem Data to Databricks via MinIO AIStor
                              https://www.truthinit.com/index.php/channel/2194/practical-session-connecting-on-prem-data-to-databricks-via-minio-aistor/
                            • 11/19/2026
                              01:00 PM
                              11/19/2026
                              Securing the Microsoft Enterprise: Strengthening Configuration Management, Identity Security, and Data Protection
                              https://www.truthinit.com/index.php/channel/2076/securing-the-microsoft-enterprise-strengthening-configuration-management-identity-security-and-data-protection/
                            • 11/19/2026
                              01:00 PM
                              11/19/2026
                              Zendesk Deep Dive: Get the work done faster with Employee Service AI Agents
                              https://www.truthinit.com/index.php/channel/2181/zendesk-deep-dive-get-the-work-done-faster-with-employee-service-ai-agents/
                            • 11/19/2026
                              01:00 PM
                              11/19/2026
                              A Deep Dive into Muse, Dots, and Duet's Impact on Your Website
                              https://www.truthinit.com/index.php/channel/2228/a-deep-dive-into-muse-dots-and-duets-impact-on-your-website/
                            • 12/03/2026
                              01:00 PM
                              12/03/2026
                              CISO Insights: Steering Cybersecurity Strategies in an AI-Driven Era
                              https://www.truthinit.com/index.php/channel/2231/ciso-insights-steering-cybersecurity-strategies-in-an-ai-driven-era/
                            • 01/19/2027
                              02:00 PM
                              01/19/2027
                              HUMAN Dialogue: Embracing the Rise of the Empowered AI Consumer.
                              https://www.truthinit.com/index.php/channel/2160/human-dialogue-embracing-the-rise-of-the-empowered-ai-consumer/
                            • 03/18/2027
                              01:00 PM
                              03/18/2027
                              Securing Access for People, Partners and AI Agents
                              https://www.truthinit.com/index.php/channel/2251/securing-access-for-people-partners-and-ai-agents/
                            • 06/17/2027
                              01:00 PM
                              06/17/2027
                              Stop the Leak: Data Loss Prevention for the AI Era
                              https://www.truthinit.com/index.php/channel/955/360view-stop-the-leak-data-loss-prevention-for-the-ai-era/
                            • 08/19/2027
                              01:00 PM
                              08/19/2027
                              Who Has Access to What...and Should They?
                              https://www.truthinit.com/index.php/channel/2252/who-has-access-to-what-and-should-they/
                            • 10/21/2027
                              01:00 PM
                              10/21/2027
                              After VMware: Building the Private Cloud You Actually Want
                              https://www.truthinit.com/index.php/channel/2253/after-vmware-building-the-private-cloud-you-actually-want/
                            • 11/18/2027
                              01:00 PM
                              11/18/2027
                              Assume Breach, Prove Recovery
                              https://www.truthinit.com/index.php/channel/2254/assume-breach-prove-recovery/
                            • 12/16/2027
                              01:00 PM
                              12/16/2027
                              The Self-Running Service Desk
                              https://www.truthinit.com/index.php/channel/2255/the-self-running-service-desk/
                            Truth in IT
                            • Sponsor
                            • About Us
                            • Terms of Service
                            • Privacy Policy
                            • Contact Us
                            • Preference Management
                            Desktop version
                            Standard version