Hired a Network Engineer? Here Are 3 Things They Will Forget to Do
When you start a new job there are is a lot to learn. When you are a security engineer, the learning curve is especially steep. There are hundreds of network and security appliances and supporting software needed to build, manage and scale a network. To complicate matters, every business’ network architecture and topology is different. The more complex your network is, the longer it will be until your new hire is fully ramped. While we’d like every new hire to be superhuman, the reality is we are all human and smart people forget to do simple things now and then.
Unfortunately for Network and Security professionals these simple mistakes cost companies hundreds of thousands of dollars every minute their network is down. If you are the new hire, or the manager of the IT operations team, you can avoid these soul crushing mistakes. Here are top items that catch the best of us and need to be reminded to have on the top of our to do lists:
#1. Disable Debug Mode
What is Debug mode? When you are troubleshooting a device, you need to test different capabilities of the machine to determine what part of the product is malfunctioning or otherwise not behaving as intended. Many software solutions have a “Debug mode” that allows you to do this testing without disrupting the end users experience.
Why this simple mistake causes major mayhem. When troubleshooting a system debug flags are often enabled. When enabled they use extra resources, and forgetting to turn them off after troubleshooting has finished can mean service interruptions or reduced throughput.
How do you know this has happened? Your customers may experience spotty connectivity or say their connection is slow.
Pro tip: Make sure this feature is disabled when not in use on all devices. Find a way to check each firewall daily.
#2 – Align Static Routing Tables
What are static routes? Static routing is when you manually configure the path of a segment of your network. These paths are usually managed by your network administrator. Said simply, these are the approved highways for packets of information to get from point A to point B. Static routes are used in scenarios where the network parameters and environment are expected to remain constant.
Why this simple mistake causes major mayhem. When you launch a new application or service, many IT pros opt for setting up a static route on the firewall. This allows for communication to pass through your firewall in a predictable manner. If your administrator sets up an static route for a new business service, and say the mobile application performance is slow, the admin knows precisely where information is expected to be transmitted from, and can exert precise control over it. Now, if your company has high availability requirements you will have active and passive devices on your network. The same routes need to be applied to both active and passive devices so that in the event the active firewall fails, and control is passed to the stand by device, and your external service isn’t disrupted.
How do you know this has happened: Unfortunately no warning. You would be experiencing an outage.
Pro tip: Static routes are used typically when defining network segments, common in micro-segmentation strategies. When defining and setting up your network segments have a more senior engineer review their static routes before going live.
#3 – Inspect Packet Drops
What is a packet drop? A packet drop is when when one or more packets of information travelling across your network fail to reach their destination. Packet loss can happen for various reasons such as congestion on the highway, intentional throttling by IT, or a result of a malicious attack to name a few. Palo Alto Networks, similar to other firewall vendors, has a Packet Drop Counter feature that categorizes why packets are dropped. This helps cut down troubleshooting time and provides more granularity in packet inspection especially when you have to inspect hundreds of counters.
Why this simple mistake causes major mayhem. The Packet drop counter is an advanced feature of the Palo Alto Networks firewalls. Therefore it would take an additional layer of proficiency in this device to understand what each counter means. This is a feature included in Palo Alto Firewalls that provides great efficiencies if you know how to use it. Customers today do not highly leverage this because the counters increment and an engineer would have to track rate.
How do you know this is happening? Typically a call to the help desk saying their information is not being sent or received.
Pro tip: Ensure your new hires understand the historical causes for packet drops at your business, and have access to trainings that teach them how to use the Packet Drop Counter and/or implement CLI commands.
What are the other things that you wish you would of known when you started as a network engineer? Let us know by commenting on this post in the community.
Indeni will continuously check for these issues and more in your environment. Ramp your new hires and junior engineers quickly. Try Now.