TechnologyBusinessManagementExplicit

IT Horror Stories with Jack Smith

Jack Smith

IT Horror Stories with Jack Smith tells real stories from the world of IT: outages, failed migrations, broken infrastructure, cybersecurity incidents, software failures, networking disasters, cloud problems, Shadow IT, and the occasional decision that seemed like a good idea at the time.

Jack Smith talks with the people who were there about what happened, how they got out of it, and what others can learn from the incident. Stories are anonymized where needed, and the focus is on technology, processes, and organizations rather than blaming individuals. New episodes weekly, with Jack's Rants in between.

You are one of us.

Play

Open in

24 episodes
weekly
Avg 24 min
English

Episodes 24

Newest Oldest Longest

All20 Unplayed20 In progress0 Played0

Counted on this page — what you have heard stays on this device, so it is not something the list can be paged by.

S2 · E14
Monday · 45 min
Cleared for Disaster : Air Traffic Control Technology: Outages, Failures, and Recovery
Jack talks with Harvey Scolnick about his years working with air traffic control technology and the failures that came with it. From construction work accidentally cutting a critical cable and knocking an air traffic control facility offline, with systems unavailable and aircraft still in the sky, to the people on the ground having to fall back on procedures, neighboring facilities, and some very human problem-solving. In addition, we look at the redundancy behind air traffic control, what recovery looks like when safety-critical infrastructure fails, and why even carefully designed systems can still be brought down by something remarkably mundane. Want even more info ? Read our show notes and related blog post related to this episode : https://blog.ithorrorstories.eu/episode-26-cleared-for-disaster/ Find out more about Harvey Scolnick : https://www.scolnickatc.com/ 00:00 Introduction 00:55 Harvey's Journey 02:13 Becoming ATC 05:34 Airways and Radar 07:03 Separation Standards 11:21 Busy JFK 14:33 Iconic Aircrafts 21:12 Technology Upgrades 29:27 Preventative Measures 38:10 AI and Future 41:27 Weather Challenges 43:57 Conclusion
- Transcript
October 5 · 19 min
Jack's Rants - Meshcore : Off-Grid Messaging, LoRa, Privacy, and Decentralised Networks
Someone asked us why we weren't on MeshCore yet. First we had to establish that it wasn't a new kind of thrash metal. Then we discovered an off-grid messaging network built from little radios, antennas, repeaters, and licence-free spectrum. So we started digging. How does it route messages ? How private is it ? What happens when repeaters disappear ? And haven't we built something suspiciously similar before ? MeshCore turns out to be considerably more interesting than just another messaging app. Want even more info ? Read our show notes and related blog post related to this episode : https://blog.ithorrorstories.eu/jacks-rants-meshcore/ MeshCore documentation used : - FAQ : https://docs.meshcore.io/faq/ - Packet Format : https://docs.meshcore.io/packet_format/ - Payloads : https://docs.meshcore.io/payloads/ - Companion Protocol: https://docs.meshcore.io/companion_protocol/ - CLI Commands : https://docs.meshcore.io/cli_commands/ MeshCore Map : https://meshcore.co.uk/map.html 00:00 Introduction 00:46 Discovering Mesh Core 02:09 Radio and Repeaters 05:19 Free Radio Insights 10:35 Encryption and Privacy 14:26 Old Meets New 15:19 Infrastructure Challenges 18:16 Conclusion
- Transcript
S2 · E13
September 28 · 21 min
Cultural Differences in IT : When Global Teams Don't Work the Same Way
IT may run on the same protocols everywhere, but the people operating it certainly don't. Put teams from different countries, companies, and working cultures onto the same project and apparently simple questions about responsibility, communication, deadlines, and escalation can suddenly have very different answers. This episode looks at what happens when those differences become part of the technical problem. From assumptions that nobody realised they were making to communication styles that work perfectly well locally but collide spectacularly across borders, we discuss how international IT teams get themselves into trouble — and what years of working across cultures can teach you about avoiding it. And no, the solution isn't another mandatory cultural-awareness PowerPoint. Want even more info ? Read our show notes and related blog post related to this episode : https://blog.ithorrorstories.eu/episode-25-cultural-differences-in-it/ 00:00 Introduction 00:41 Cultural Intricacies 03:12 Sales Challenges 06:02 Engineering Insights 10:32 Crisis Management 14:10 Data Center Differences 16:00 Luxembourg's Unique Edge 17:52 European Culture Wrap-up
- Transcript
S2 · E12
September 21 · 32 min
When UNPK Broke – Diagnosing an IBM Mainframe Hardware Failure
Texas, 1978. Robert was working at a cotton co-op running what may have been one of the world's earliest online auction systems. Farmers received punch cards representing their bales of cotton - occasionally complete with enough stray cotton fibres to jam the card readers - while buyers placed bids remotely using leased IBM 3270 terminals and acoustic coupler modems. After an upgrade, the mainframe refused to come back properly. An IPL produced a corrupted device address before dropping the machine into a hard wait. Single-cycling the processor indicated that an UNPK (Unpack) instruction was producing the wrong output from perfectly good input. The instruction itself had effectively broken. And yes, this episode contains a lot of old IBM hardware and mainframe terminology. We know. That's part of the fun. Want even more info ? Read our show notes and related blog post related to this episode : https://blog.ithorrorstories.eu/episode-24-when-unpk-broke/ 00:00 Introduction 00:43 Robert's Background 02:14 IBM DOS 03:27 Cotton Auction System 05:37 Online Auction 07:14 SNA Protocol 09:05 Hardware Issue 13:20 Debugging Unpack 17:13 IBM Fix 21:23 Conclusion
- Transcript
S2 · E11
September 14 · 37 min
Pinball as IoT : When Arcade Machines Go Online
A modern pinball machine isn't just flippers, bumpers, lights, and a very large box making expensive noises. It's a connected computer with software updates, online score tracking, player accounts, remote management, and an Internet connection. Evert Brochez from Torius joins Jack to explain how pinball evolved from electromechanical machines into something that looks suspiciously like an IoT device. But once you connect a pinball machine to the Internet, familiar IT questions start appearing. Who maintains the software? How long will it be supported? What can operators manage remotely? How much do we trust the manufacturer with security and access? We look at modern pinball from the operator's perspective, the realities of software quality, and the surprisingly unanswered cybersecurity questions that appear when the arcade machine becomes another device on the network. Find out more about Torius : Website : https://www.horecaspelen.be/ Facebook : https://www.facebook.com/Bingovit Flippers & Bier Podcast (in dutch) : https://open.spotify.com/show/5kvyHkIfo6HfdlulPeDScL More about Comic Sans : Gent : https://cafecomicsans.be/ Brussels : https://www.comicsansbrussels.be/ Facebook : https://www.facebook.com/cafecomicsans Want even more info ? Read our show notes and related blog post related to this episode : https://blog.ithorrorstories.eu/episode-23-pinball-as-iot/ 00:00 Introduction 00:34 Pinball Enthusiast 02:20 Pinball Operations 03:54 Maintenance Insights 05:16 IoT Integration 06:53 Player Engagement 09:44 Tech Challenges 13:41 Future Features 21:57 Software Updates 30:19 Conclusion
- Transcript
S2 · E10
September 7 · 25 min
The Three Envelopes: Blame, Reorganize, Repeat
The Three Envelopes: Blame, Reorganize, Repeat There is an old management joke about an outgoing manager leaving three numbered envelopes for their successor. When the first crisis arrives: blame your predecessor. The second: reorganize. And when the third arrives... prepare three envelopes. We look at why this joke has survived for decades, and why anyone who has worked in corporate IT will recognise far too much of it. From inherited problems and reorganisations to management changes that somehow leave the underlying problems untouched, they explore what the Three Envelopes tells us about blame, accountability, and the wonderfully repetitive nature of IT organisations. Want even more info ? Read our show notes and related blog post related to this episode : https://blog.ithorrorstories.eu/episode-22-the-three-envelopes/ 00:00 Introduction 01:01 The Three Envelopes 05:02 Blame Management 10:28 Management Insights 19:10 Entertaining Anecdotes 23:35 Conclusion
- Transcript
August 31 · 9 min
Jack's Rants - Why You Need to Reboot : Why Rebooting Your Computer Actually Fixes Problems
Jack's Rants - Why You Need to Reboot : Why Rebooting Your Computer Actually Fixes Problems "Have you tried turning it off and on again ?" became an IT cliché for a reason. Rebooting clears accumulated state, resets resources, and fixes a surprising number of problems. From Unix uptime bragging rights to mainframe IPLs and Windows servers nobody is allowed to restart, every system has its own relationship with rebooting. Maybe what we really need is a reboot SLA. Want even more info ? Read our show notes and related blog post related to this episode : https://blog.ithorrorstories.eu/jacks-rants-can-i-keep-your-id/ 00:00 Introduction 02:17 Developer Habits 04:01 Mainframes & Servers 06:18 Reboot Realities 08:19 Conclusion
- Transcript
S2 · E9
August 24 · 38 min
Testing Software Properly : Software Testing Done Right – Where, When and What Should You Test?
Testing software sounds simple enough. But what exactly should you test, where should those tests happen, and how do you decide which parts of a complex system need the most attention? Jack talks with Mooly Beeri about software testing from a developer's perspective: identifying the right test points, understanding how different parts of a system interact, and building testing into the development process rather than treating it as something that happens at the end. A practical look at what it actually means to test software properly. You can read more from Mooly at https://www.bettersoftware.dev/ Want even more info ? Read our show notes and related blog post related to this episode : https://blog.ithorrorstories.eu/episode-21-testing-software-properly/ 00:00 Introduction 01:05 Mooly's Background 01:43 Automotive Software 04:01 Software Challenges 06:21 Bug Management 12:11 Improving Processes 15:42 Communication Importance 28:07 Summary
- Transcript
S2 · E8
August 17 · 25 min
Microcode Mayhem : How a Mainframe Timer Bug Slowed an Entire Development Team
One day, a busy IBM-compatible mainframe began slowing to a crawl. Hundreds of programmers noticed immediately—some even resorted to banging on the server room window whenever the system became unusable. The problem appeared after a microcode upgrade, but nothing pointed to an obvious cause. In this episode, Jack Smith talks to Matt about an extraordinary piece of low-level troubleshooting that led deep into the internals of VM, interrupt handling and timer queues. Armed with nothing more than hex dumps, careful reasoning and a willingness to modify live memory, Matt uncovered a vendor bug and devised a temporary workaround that kept the system running until a permanent fix arrived. It's a fascinating story of how understanding a system at its lowest levels can solve problems that no diagnostic tool could explain. Want even more info ? Read our show notes and related blog post related to this episode : https://blog.ithorrorstories.eu/episode-20-microcode-mayhem/ 00:00 Introduction 01:07 Mainframe Tech 04:38 Microcode Issues 07:19 Timer Interrupts 15:02 Modern Comparisons 19:44 Final Insights 22:18 Conclusion
- Transcript
August 10 · 11 min
Jack's Rants - AI Is The New Shadow IT : Why Employees Are Quietly Building the Next IT Risk
Artificial intelligence didn't invent Shadow IT—it made it invisible. In this Jack's Rant, Jack Smith explores how AI tools like ChatGPT, Copilot and Claude have fundamentally changed the way employees solve problems. No servers under desks. No rogue software installations. Just a browser tab and a prompt. From confidential data and AI-generated scripts to fragmented workflows and "ChatGPT says..." becoming the new expert opinion, this episode examines why AI has lowered the barrier to creating business-critical systems outside IT's view. But this isn't a rant against AI. It's a reminder that Shadow IT has always been a symptom, not the disease. Employees don't create workarounds because they enjoy breaking policy—they do it because they're trying to get their jobs done. Instead of asking how to stop people from using AI, perhaps IT should be asking why they felt the need to use it in the first place. Because the organizations that succeed won't be the ones with the strictest AI policy, but the ones that make the secure, supported way the easiest way. Want even more info ? Read our show notes and related blog post related to this episode : https://blog.ithorrorstories.eu/jacks-rants-ai-is-the-new-shadow-it/ 00:00 Introduction 01:11 Shadow IT Explained 02:43 Feedback Loops 04:01 Modern Shadow IT 06:24 Trust in AI 08:25 AI Fragmentation 10:08 Leadership Solutions
- Transcript
S2 · E7
August 3 · 30 min
Melbourne Cup Meltdown : How a Betting System Crashed at the Worst Possible Moment
Every year, the Melbourne Cup pushes Australia's betting infrastructure to its limits. In 1991, a known software bug turned one of the busiest days of the year into an IT disaster. As millions of bets poured in, CPU utilisation unexpectedly collapsed and the betting system ground to a halt at the worst possible moment. Jack Smith is joined by veteran mainframe specialist Neale to revisit the incident, explain why the bug had been accepted for so long, how the team recovered, and what modern IT professionals can learn about technical debt, capacity planning, and the risks of carrying known defects into critical production systems. Want even more info ? Read our show notes and related blog post related to this episode : https://blog.ithorrorstories.eu/episode-19-melbourne-cup-meltdown/ 00:00 Introduction 00:53 Mainframe Basics 02:01 Melbourne Cup Explained 05:01 System Failure 09:06 Aftermath & Reflection 17:03 Learning from Errors 21:53 Modern Vs. Past Tech 29:51 Conclusion
- Transcript
July 27 · 10 min
Jack's Rants - The Missing Five Minutes : Why scambaiting videos leave security professionals with more questions than answers
Scambaiters like Kitboga, Jim Browning and Pierogi educate millions, waste scammers' time, and undoubtedly prevent people from becoming victims. As a security professional, I genuinely appreciate what they do. But every time I watch one of these videos, I find myself asking the same question: What happened in the missing five minutes? How does a legitimate remote support session suddenly become full control over the scammer's computer? Is it a vulnerability? Poor operational security? Social engineering? Or simply the part of the story that gets left on the cutting room floor? This isn't a criticism of scambaiters. It's an exploration of the unanswered security questions that every IT professional should be asking, and why understanding how something happened can be just as important as enjoying the outcome. Want even more info ? Read our show notes and related blog post related to this episode : https://blog.ithorrorstories.eu/jacks-rants-the-missing-five-minutes/ 00:00 Introduction 00:22 Scambaiter Styles 02:01 The IT Dilemma 04:23 Compromised Systems 06:48 The Editing Reality 09:00 Seeking Security Answers 09:59 Closing Thoughts
- Transcript
S2 · E6
July 20 · 37 min
One Network to Rule Them All : How a Missing VLAN Strategy Turned Smart City Infrastructure into a Cybersecurity Nightmare
Smart cities promise better transportation, connected infrastructure, and improved services—but what happens when everything is connected a little too well? In this episode, our guest shares the story of a security assessment in a major metropolitan smart city where network segmentation had been overlooked, creating unexpected paths between public networks and critical infrastructure. One memorable discovery showed that, with nothing more than the correct IP address, it was possible to operate a parking garage barrier from the city's public Wi-Fi while riding a bus on the opposite side of town. Together, we explore why VLANs and network segmentation remain some of the most fundamental principles in cybersecurity, how convenience can quietly undermine security, and why the smartest technology still depends on good architecture. It's a fascinating story that demonstrates how seemingly small design decisions can have city-wide consequences—and why "everything connected" should never mean "everything can talk to everything else." Want even more info ? Read our show notes and related blog post related to this episode : https://blog.ithorrorstories.eu/episode-18-one-network-to-rule-them-all/ 00:00 Introduction 01:58 Smart City Chaos 10:47 Security Camera Mess 25:09 ANPR Issues 36:08 Concluding Thoughts
- Transcript
July 13 · 10 min
Jack's Rants - Can I Keep Your ID? : Privacy, Identity Theft, Data Minimization, and Why Your Driver's License Is Not a Bar Tab Deposit
Opening a tab at a bowling alley shouldn't require handing over your driver's license or national ID card. Yet many businesses still ask customers to leave government-issued identity documents behind as collateral, often storing them in plain sight with little thought for privacy, physical security, or the risks involved. In this Rant, we ask a simple question: just because a practice has existed for years, does that make it acceptable? Want even more info ? Read our show notes and related blog post related to this episode : https://blog.ithorrorstories.eu/jacks-rants-can-i-keep-your-id/ 00:00 Introduction 01:34 Questioning Practices 02:46 Value of ID Cards 05:13 Consequences 06:01 Secure Solutions 07:51 Broader Implications 09:35 Conclusion
- Transcript
S2 · E5
July 6 · 32 min
Cardboard Confidential : A Cybersecurity Incident, a Golf Course Conversation, and Why No Industry Is Too Boring to Be Targeted
No one expects a cybersecurity incident to begin in the cardboard industry. Yet as our guest Kayne McGladrey explains, cybercriminals don't care what your company manufactures—they care about opportunity. In this episode, Kayne shares how an unexpected conversation with a golf buddy led to a cybersecurity contract, and why one of the most unassuming industries turned out to have a story worth telling. Together, we explore the incident itself, how trust and professional networks can open doors in unexpected ways, and why manufacturing organizations of every kind have become attractive targets for cyberattacks. It's a fascinating reminder that behind every product—even a simple cardboard box—there's a complex IT infrastructure that deserves to be protected. You can read more from Kayne at https://kaynemcgladrey.com/ Want even more info ? Read our show notes and related blog post related to this episode : https://blog.ithorrorstories.eu/episode-17-cardboard-confidential/ 00:00 Introduction 00:57 Cardboard Industry Focus 03:22 Manufacturing Breach 06:15 Incident Escalation 12:26 Operation Recovery 22:56 Lessons Learned 30:26 Conclusion
- Transcript
June 22 · 12 min
Jack's Rants - The New Change Management : ITIL Change Control, Organizational Change, and Why Project Managers Suddenly Need Psychology Degrees
Once upon a time, Change Management meant raising an RFC, preparing a rollback plan, and surviving the Change Advisory Board. Today, it also means stakeholder engagement, communication plans, adoption metrics, workshops, and apparently having the emotional intelligence of a licensed therapist Whether you're a project manager, systems engineer, change manager, or simply someone who's ever heard the phrase "we've always done it this way," this rant will probably feel uncomfortably familiar. Want even more info ? Read our show notes and related blog post related to this episode : https://blog.ithorrorstories.eu/jacks-rants-the-new-change-management/ 00:23 Introduction 01:23 Evolution of Change 02:39 Change Challenges 04:14 Human Factors 06:56 Practical Solutions 10:25 Conclusion
- Transcript
S2 · E4
June 15 · 14 min
GDPR Enters the Chat : The Day a Hiring Exercise Became an Information Security Incident
It was supposed to be a routine hiring exercise. A candidate receives a technical assignment, reviews the provided material, and prepares a solution. Then someone notices that the "anonymous" dataset isn't anonymous at all. What follows is an uncomfortable discovery involving confidential business information, questions about data handling, and an unexpected conversation with legal. In this episode of IT Horror Stories with Jack Smith, we discuss how good intentions, poorly sanitized data, and assumptions about anonymity can quickly transform a recruitment exercise into an information security incident. Along the way, we explore data governance, confidentiality, risk management, and what organizations should do when they discover they've shared information they never intended to expose. Want even more info ? Read our show notes and related blog post related to this episode : https://blog.ithorrorstories.eu/gdpr-enters-the-chat/ 00:00 Introduction 02:18 Security Breach 03:25 Social Engineering Risk 10:54 Company Response 12:46 Final Thoughts
- Transcript
S2 · E3
June 1 · 26 min
Spooling Out of Control : Enterprise Printing Explained and Why Printing Still Breaks IT
Printing is one of those technologies everyone assumes should have been solved years ago — until someone can't print invoices, shipping labels stop coming out, or 50,000 customer letters suddenly disappear into a print queue somewhere. In this episode, we take a tour through the strange and surprisingly complex world of printing: from small desktop printers and office multifunction devices to enterprise print servers and industrial mass-mailing environments that still move serious business. We explore why printing continues to survive every digital transformation, what actually happens behind the scenes when you click "Print," and why reliability suddenly becomes very important when paper turns into payroll, warehouse labels, customer communication, or operational downtime. Because nothing reminds people that IT exists faster than when the printer stops working. Want even more info ? Read our show notes and related blog post related to this episode : https://blog.ithorrorstories.eu/episode-15-spooling-out-of-control/ 00:00 IT Horror Stories 02:00 Industrial Printing 06:00 Printer Installation 11:00 Mission Critical Printing 15:00 High Volume Printing 17:30 DTP and Mac Systems 24:00 Printing Challenges
- Transcript
S2 · E2
May 18 · 23 min
Sleep Mode in Production : How a Laptop Took Down a Warehouse
Welcome to the story of a perfectly known and visible laptop that somehow became critical production infrastructure inside a warehouse environment — without going through proper validation, testing, or operational review. The system worked, solved an immediate problem, and was quickly integrated into daily operations... right up until the laptop entered sleep mode and warehouse activities ground to a halt. What followed was a frantic investigation into why scanners stopped working, workflows froze, and logistics operations suddenly stalled because of a standard power-saving setting. It's a story about how quickly "temporary" operational solutions can become business-critical, and why governance, testing, and basic operational checks matter far more than people realize — especially when convenience quietly reaches production. Want even more info ? Read our show notes and related blog post related to this episode : https://blog.ithorrorstories.eu/episode-14-sleep-mode-in-production/ 00:00 Introduction 01:32 Early 2000s IT 03:07 Warehouse Down 05:07 Troubleshooting Begins 09:23 Cable Chaos 11:18 Director's Laptop 15:57 Lessons Learned 21:31 Conclusion
- Transcript
S2 · E1
May 4 · 35 min
Ransomware Lockdown : What a Real-World Ransomware Attack Looks Like, Incident Response and Recovery
We explore a hypothetical ransomware scenario that mirrors what many organizations could face today. Imagine a normal day where systems suddenly become inaccessible, data is encrypted, and the scope of the attack starts to unfold in real time. What follows is a race against the clock, with teams trying to understand what's happening while keeping critical operations alive. More importantly, we focus on the response and recovery: the trade-offs, the communication challenges, and the lessons organizations can take away before something like this actually happens. Because ransomware isn't just a technical problem — it's an operational and human one, where preparation and clarity matter as much as the tools in place. Want even more info ? Read our show notes and related blog post related to this episode : https://blog.ithorrorstories.eu/episode-13-ransomware-lockdown/ 00:00 Introduction 01:05 Ransomware Explained 03:08 Attack Entry Points 06:16 Employee Training 13:00 AI Phishing Threats 20:18 Data Leaks Aftermath 25:26 Cybersecurity in Mergers 28:18 Recovery Steps 33:04 Conclusion
- Transcript

Showing 1–20 of 24 episodes