Hired a Network Engineer? Here Are 3 Things They Will Forget to Do
Notice: This blog post was originally published on Indeni before its acquisition by BlueCat.
The content reflects the expertise and perspectives of the Indeni team at the time of writing. While some references may be outdated, the insights remain valuable. For the latest updates and solutions, explore the rest of our blog
Key takeawaysThis key takeaway was generated through LLMs crawling the page and coming up with an overview of the content.This article outlines three common operational mistakes that slow new network and security engineers and can cause major outages: leaving debug mode enabled, misaligned static routing tables, and ignoring packet drop counters. It explains the technical environment—firewalls, static routes, and packet inspection tools like Palo Alto Networks’ Packet Drop Counter—how each error manifests in production, and the operational impacts such as reduced throughput, complete outages, or prolonged troubleshooting. The piece concludes with practical pro tips to prevent these failures (disable debug flags, mirror static routes on active/passive devices, and train staff to use packet drop counters), and suggests automated continuous checks to accelerate ramp-up of junior staff.
Why is leaving debug mode enabled on devices harmful to network operations?
Debug mode is used during troubleshooting to test device capabilities without impacting end users, but when debug flags remain enabled they consume extra resources on the device. That additional load can reduce throughput or cause service interruptions, producing spotty connectivity or apparent slowness for customers. The article’s pro tip is to ensure debug mode is disabled on all devices when not actively troubleshooting and to implement daily checks of each firewall to catch forgotten debug settings quickly.
How can misaligned static routing tables cause an outage after deploying a new service?
Static routes are manually configured paths that direct traffic between network segments and are commonly used for predictable communication, such as when launching a new application. In high-availability environments with active and passive devices, failing to apply the same static routes to both devices means that if the active firewall fails and control passes to the standby, traffic won’t follow the expected path and the external service can be disrupted. The article recommends having a senior engineer review static routes before going live and ensuring routes are mirrored across active/passive devices.
What role does the Packet Drop Counter play in troubleshooting and why is it often underused?
The Packet Drop Counter on Palo Alto Networks firewalls categorizes reasons packets are dropped, which speeds troubleshooting and provides granular visibility when analyzing many counters. It is an advanced feature that requires additional proficiency to interpret each counter and to track increment rates over time. Because of that learning curve, customers often underutilize it; the article advises training new hires on the historical causes of packet drops at the business and teaching them how to use the Packet Drop Counter and related CLI commands so issues are identified and resolved faster.
When you start a new job there are is a lot to learn. When you are a security engineer, the learning curve is especially steep. There are hundreds of network and security appliances and supporting software needed to build, manage and scale a network. To complicate matters, every business’ network architecture and topology is different. The more complex your network is, the longer it will be until your new hire is fully ramped. While we’d like every new hire to be superhuman, the reality is we are all human and smart people forget to do simple things now and then.
Unfortunately for Network and Security professionals these simple mistakes cost companies hundreds of thousands of dollars every minute their network is down. If you are the new hire, or the manager of the IT operations team, you can avoid these soul crushing mistakes. Here are top items that catch the best of us and need to be reminded to have on the top of our to do lists:
#1. Disable Debug Mode
What is Debug mode? When you are troubleshooting a device, you need to test different capabilities of the machine to determine what part of the product is malfunctioning or otherwise not behaving as intended. Many software solutions have a “Debug mode” that allows you to do this testing without disrupting the end users experience.
Why this simple mistake causes major mayhem. When troubleshooting a system debug flags are often enabled. When enabled they use extra resources, and forgetting to turn them off after troubleshooting has finished can mean service interruptions or reduced throughput.
How do you know this has happened? Your customers may experience spotty connectivity or say their connection is slow.
Pro tip: Make sure this feature is disabled when not in use on all devices. Find a way to check each firewall daily.
#2 – Align Static Routing Tables
What are static routes? Static routing is when you manually configure the path of a segment of your network. These paths are usually managed by your network administrator. Said simply, these are the approved highways for packets of information to get from point A to point B. Static routes are used in scenarios where the network parameters and environment are expected to remain constant.
Why this simple mistake causes major mayhem. When you launch a new application or service, many IT pros opt for setting up a static route on the firewall. This allows for communication to pass through your firewall in a predictable manner. If your administrator sets up an static route for a new business service, and say the mobile application performance is slow, the admin knows precisely where information is expected to be transmitted from, and can exert precise control over it. Now, if your company has high availability requirements you will have active and passive devices on your network. The same routes need to be applied to both active and passive devices so that in the event the active firewall fails, and control is passed to the stand by device, and your external service isn’t disrupted.
How do you know this has happened: Unfortunately no warning. You would be experiencing an outage.
Pro tip: Static routes are used typically when defining network segments, common in micro-segmentation strategies. When defining and setting up your network segments have a more senior engineer review their static routes before going live.
#3 – Inspect Packet Drops
What is a packet drop? A packet drop is when when one or more packets of information travelling across your network fail to reach their destination. Packet loss can happen for various reasons such as congestion on the highway, intentional throttling by IT, or a result of a malicious attack to name a few. Palo Alto Networks, similar to other firewall vendors, has a Packet Drop Counter feature that categorizes why packets are dropped. This helps cut down troubleshooting time and provides more granularity in packet inspection especially when you have to inspect hundreds of counters.
Why this simple mistake causes major mayhem. The Packet drop counter is an advanced feature of the Palo Alto Networks firewalls. Therefore it would take an additional layer of proficiency in this device to understand what each counter means. This is a feature included in Palo Alto Firewalls that provides great efficiencies if you know how to use it. Customers today do not highly leverage this because the counters increment and an engineer would have to track rate.
How do you know this is happening? Typically a call to the help desk saying their information is not being sent or received.
Pro tip: Ensure your new hires understand the historical causes for packet drops at your business, and have access to trainings that teach them how to use the Packet Drop Counter and/or implement CLI commands.
What are the other things that you wish you would of known when you started as a network engineer? Let us know by commenting on this post in the community.
Indeni will continuously check for these issues and more in your environment. Ramp your new hires and junior engineers quickly. Try Now.
Related content
How to map your network with user-defined links in Integrity X
Map your network with user-defined links in Integrity X to define and manage custom relationships, such as dual-stack and NAT environments.
Automate your DDI modernization path by migrating with Micetro
Automate cross-platform DNS and DHCP migration with Micetro to reduce risk, eliminate manual effort, and modernize infrastructure faster.
Your journey to intelligent NetOps begins at Cisco Live
Visit BlueCat’s booth or book a meeting now to learn more about how our solutions can help you build a network that supports constant change.
Replace BIND and ISC with Micetro DNS/DHCP Server (MDDS)
Tired of patching and manually configuring BIND DNS and ISC DHCP? Discover how Micetro MDDS appliances can replace them for modern DDI.