How networks collapse: gradually, then suddenly

Hemingway said that bankruptcy comes “gradually, then suddenly”. We’ve discovered that network collapses often follow a similar pattern.


May 20, 2020

In his fabulous 1926 novel, The Sun Also Rises, Ernest Hemingway famously wrote that bankruptcy happens in two ways: “gradually and then suddenly”.

This rings a bell for us. (You might even say that the bell tolls…?) In our conversations with network administrators of all stripes, we’ve found that the decline of DNS, DHCP, and IP address management (DDI) infrastructure often happens in a similar fashion.


Most enterprises experience a decline in their DDI as a gradual process. There are hints, of course, that the network’s foundation is slowly being eaten away. Yet in the absence of an event that draws attention to DDI specifically, most of the impact of these things happens slowly enough that nobody notices.

There are many symptoms of gradual DDI decline, most of which happen in parallel:

Service ticket volume: In a small network, DNS service tickets are easily handled through a manual process. Yet as networks grow more complex, the volume of requests gradually morphs into a significant burden. This is particularly true as organizations move into the cloud, where DevOps teams become very demanding with IP address provisioning requests.

Integration creep: Complexity also becomes a gradual challenge for DNS admins when they’re trying to handle integrations with other networking tools such as SD-WAN controllers, network virtualization engines, and SDN platforms. As these technologies are gradually rolled out across the enterprise, they exact a similarly gradual cost on network admins who support their DDI requirements.

Security gaps: Given the difficulty of deploying DNSSEC in Microsoft DNS and BIND, many network admins either don’t do it well or don’t do it at all. As the network scales and grows more complex, that task only becomes harder.

Lack of visibility:  When you’ve only got a handful of servers, compiling DNS logs to trace the source of security or operational issues is relatively easy. Yet as the network grows, that information becomes steadily difficult to gather and analyze at scale – to the point that few admins bother to do it at all.

Shadow IT: When DevOps and cloud teams can’t provision IP addresses quickly, they’ll often just stand up a BIND server and keep going. Over time, the probability of IP conflicts and the challenge of managing DDI across hybrid environments grows.

Managing customization: Building and maintaining custom scripts for Microsoft DNS and BIND starts off as a manageable exercise, but over time it morphs into a full-time job. One person gradually sheds all other duties and focuses on core infrastructure exclusively. We call this person “Mr. DNS”, but there’s probably an Old Man and the Sea analogy in there somewhere…

Single-threaded dependence: The home-grown fixes and custom architectures built over many years by Mr. DNS leave network operations highly dependent on the institutional knowledge of one person. But nobody realizes it until that person decides to retire or take another job.


After all of these things gnaw away at the foundation of the network for several years, the “suddenly” part happens. Usually, it’s a single event – often a large-scale outage – that puts the severity of the situation into sudden focus. Sometimes it’s when Mr. DNS retires – or threatens to quit – that the IT team realizes the trouble they’re in.

By the time most network administrators and IT executives find that their DDI is broken, the situation is usually desperate. They come to solution providers like BlueCat practically begging for a solution to the constant network outages, the flood of service tickets, and the fragility of their network infrastructure.

To have and have Not

DDI-related collapses aren’t inevitable. The gradual impact of DDI problems only means that with enough planning and foresight, the foundation of your network infrastructure can be addressed with enough time to stave off the “suddenly” part.

We know that it’s tempting to kick the can down the road. We also know that it’s a worse mistake to let DDI problems gradually creep up on you.

A crash migration to a purpose-built DDI solution like BlueCat is always possible. (We’ve done it in a weekend.) At the same time, we prefer to take a more methodical approach – one that migrates your infrastructure with zero downtime and creates an architecture built around your business needs.

Don’t let DDI collapse your network gradually, then suddenly. Taking a strategic approach to your DDI infrastructure will pay immediate dividends – greater stability, security, efficiency – while at the same time providing flexibility to address future needs. In other words, it’s much easier to prevent a problem than it is to clean it up.

Which phase are you in – gradual or sudden? If you’re ready to build your network around the best DDI solution (before it’s too late), we should talk. (Hemingway had a bunch of six-toed cats, so we feel like it’s only natural.)

Published in:

An avatar of the author

BlueCat is the Adaptive DNS company. The company’s mission is to help organizations deliver reliable and secure network access from any location and any network environment. To do this, BlueCat re-imagined DNS. The result – Adaptive DNS – is a dynamic, open, secure, scalable, and automated DDI management platform that supports the most challenging digital transformation initiatives, like adoption of hybrid cloud and rapid application development.

Related content

Detect anomalies and CVE risks with Infrastructure Assurance 8.4 

The Infrastructure Assurance 8.4 release features an anomaly detection engine for outliers and a CVE analysis engine to uncover device vulnerabilities.

Read more

Get fast, resilient, and flexible DDI management with Integrity 9.6

With Integrity 9.6, network admins can get support for new DNS record types, architect and configure multi-primary DNS, and automate IP assignments.

Read more

Deepen your security insight with Infrastructure Assurance 8.3

BlueCat Infrastructure Assurance 8.3, with an enhanced analytics dashboard, including interactive widgets and top 10 alerts, is now available.

Read more

Security, automation, cloud integration keys to DDI solution success

Only 40% of enterprises believe they are fully successful with their DDI solution. Learn how to find greater success with new research from EMA and BlueCat.

Read more

Our commitment to Micetro customers and product investment

From CEO Stephen Devito, a word on BlueCat’s ongoing commitment to supporting Micetro customers and Micetro’s evolution as a network management tool.

Read more

Seven reasons to rethink firewall monitoring and boost automation 

With BlueCat Infrastructure Assurance, you can better protect your network with automated alerts and suggested remedies for hidden issues in your firewalls.

Read more