How networks collapse: gradually, then suddenly

In his fabulous 1926 novel, The Sun Also Rises, Ernest Hemingway famously wrote that bankruptcy happens in two ways: “gradually and then suddenly”.

This rings a bell for us. (You might even say that the bell tolls…?) In our conversations with network administrators of all stripes, we’ve found that the decline of DNS, DHCP, and IP address management (DDI) infrastructure often happens in a similar fashion.

Gradually

Most enterprises experience a decline in their DDI as a gradual process. There are hints, of course, that the network’s foundation is slowly being eaten away. Yet in the absence of an event that draws attention to DDI specifically, most of the impact of these things happens slowly enough that nobody notices.

There are many symptoms of gradual DDI decline, most of which happen in parallel:

Service ticket volume: In a small network, DNS service tickets are easily handled through a manual process. Yet as networks grow more complex, the volume of requests gradually morphs into a significant burden. This is particularly true as organizations move into the cloud, where DevOps teams become very demanding with IP address provisioning requests.

Integration creep: Complexity also becomes a gradual challenge for DNS admins when they’re trying to handle integrations with other networking tools such as SD-WAN controllers, network virtualization engines, and SDN platforms. As these technologies are gradually rolled out across the enterprise, they exact a similarly gradual cost on network admins who support their DDI requirements.

Security gaps: Given the difficulty of deploying DNSSEC in Microsoft DNS and BIND, many network admins either don’t do it well or don’t do it at all. As the network scales and grows more complex, that task only becomes harder.

Lack of visibility:  When you’ve only got a handful of servers, compiling DNS logs to trace the source of security or operational issues is relatively easy. Yet as the network grows, that information becomes steadily difficult to gather and analyze at scale – to the point that few admins bother to do it at all.

Shadow IT: When DevOps and cloud teams can’t provision IP addresses quickly, they’ll often just stand up a BIND server and keep going. Over time, the probability of IP conflicts and the challenge of managing DDI across hybrid environments grows.

Managing customization: Building and maintaining custom scripts for Microsoft DNS and BIND starts off as a manageable exercise, but over time it morphs into a full-time job. One person gradually sheds all other duties and focuses on core infrastructure exclusively. We call this person “Mr. DNS”, but there’s probably an Old Man and the Sea analogy in there somewhere…

Single-threaded dependence: The home-grown fixes and custom architectures built over many years by Mr. DNS leave network operations highly dependent on the institutional knowledge of one person. But nobody realizes it until that person decides to retire or take another job.

Suddenly

After all of these things gnaw away at the foundation of the network for several years, the “suddenly” part happens. Usually, it’s a single event – often a large-scale outage – that puts the severity of the situation into sudden focus. Sometimes it’s when Mr. DNS retires – or threatens to quit – that the IT team realizes the trouble they’re in.

By the time most network administrators and IT executives find that their DDI is broken, the situation is usually desperate. They come to solution providers like BlueCat practically begging for a solution to the constant network outages, the flood of service tickets, and the fragility of their network infrastructure.

To have and have Not

DDI-related collapses aren’t inevitable. The gradual impact of DDI problems only means that with enough planning and foresight, the foundation of your network infrastructure can be addressed with enough time to stave off the “suddenly” part.

We know that it’s tempting to kick the can down the road. We also know that it’s a worse mistake to let DDI problems gradually creep up on you.

A crash migration to a purpose-built DDI solution like BlueCat is always possible. (We’ve done it in a weekend.) At the same time, we prefer to take a more methodical approach – one that migrates your infrastructure with zero downtime and creates an architecture built around your business needs.

Don’t let DDI collapse your network gradually, then suddenly. Taking a strategic approach to your DDI infrastructure will pay immediate dividends – greater stability, security, efficiency – while at the same time providing flexibility to address future needs. In other words, it’s much easier to prevent a problem than it is to clean it up.

Which phase are you in – gradual or sudden? If you’re ready to build your network around the best DDI solution (before it’s too late), we should talk. (Hemingway had a bunch of six-toed cats, so we feel like it’s only natural.)