The Seven Stages of (IT) Hell

We’ve all been there – where a specific IT scenario or event can be compared to a horror film, filled with suspense and terror.

We’ve all been there – where a specific IT scenario or event can be compared to a horror film, filled with suspense and terror.

I’m not talking about a situation with an application or two going offline, I’m talking about the red-faced, heart-pounding, sweat-pouring, tight-collared, labor-inducing adventure when your whole IT infrastructure is down. Thinking about those sorts of incidents, there always seems to be seven common stages of any architecture outage:

  1. The whisper.  A single report trickles in.  One of your core services is not available.  You hear the famous words of Roy Trenneman from the IT crowd, “Have you tried turning it off and on again?”  It must be a user error.
  2. The flood.  Help desk phones go haywire.  Escalation requests are coming in rapid succession. All of a sudden, your mouth is agape.
  3. Panic.  Your boss is standing right behind you, and his boss behind him.  They are asking what the problem is and you just know that something is broken.  They want an ETA on resolution, and you just sat down at your desk.
  4. Investigation.  The sleeves are rolled up at this point.  You check basic network connectivity.  Nope.  DNS and DHCP are still working, but a large swath of layer-3 network appliances is having trouble.  You start digging in, checking manuals, googling, and just figuring stuff out. You have found the right pile of hay, now if you could just find that elusive needle.
  5. The light bulb.  You know of a few things to investigate and fiddle with, and they seem to fit and make sense.  There may be one light bulb, or many light bulbs during the course of an IT “exercise.”
  6. Relief.  Ahhh, problem solved.  It was Tommy again.  He decided to change the common OSPF area.  Help desk phone lights go out.  Reports from application teams verify that everything is back online.
  7. Prevention. This is the longest stage.  You never want to experience that kind of panic again.  You plan upgrades, you introduce auditing, and you change security permissions and policies.  Most importantly, you change the passwords and never (ever!) let Tommy have access to the layer-3 devices in your network.
  8. Bonus stage: Déjà vu.  The phone rings….

And then, you wake up and smile.  It was all just a dream – more like a nightmare.  You’re smiling because you’re the DDI architect and you’ve just invested in a reliable and resilient BlueCat DNS, DHCP and IPAM infrastructure and can finally sleep soundly at night, knowing that all those horrors are in the past.

An avatar of the author

BlueCat provides core services and solutions that help our customers and their teams deliver change-ready networks. With BlueCat, organizations can build reliable, secure, and agile mission-critical networks that can support transformation initiatives such as cloud adoption and automation. BlueCat’s growing portfolio includes services and solutions for automated and unified DDI management, network security, multicloud management, and network observability and health.

Related content

Detect anomalies and CVE risks with Infrastructure Assurance 8.4 

The Infrastructure Assurance 8.4 release features an anomaly detection engine for outliers and a CVE analysis engine to uncover device vulnerabilities.

Read more

Get fast, resilient, and flexible DDI management with Integrity 9.6

With Integrity 9.6, network admins can get support for new DNS record types, architect and configure multi-primary DNS, and automate IP assignments.

Read more

Deepen your security insight with Infrastructure Assurance 8.3

BlueCat Infrastructure Assurance 8.3, with an enhanced analytics dashboard, including interactive widgets and top 10 alerts, is now available.

Read more

Security, automation, cloud integration keys to DDI solution success

Only 40% of enterprises believe they are fully successful with their DDI solution. Learn how to find greater success with new research from EMA and BlueCat.

Read more

Our commitment to Micetro customers and product investment

From CEO Stephen Devito, a word on BlueCat’s ongoing commitment to supporting Micetro customers and Micetro’s evolution as a network management tool.

Read more

Seven reasons to rethink firewall monitoring and boost automation 

With BlueCat Infrastructure Assurance, you can better protect your network with automated alerts and suggested remedies for hidden issues in your firewalls.

Read more