Building PAN-OS Air Traffic Control

This blog post gives a high-level overview of Indeni’s “Air Traffic Control for your NGFW” webinar. You can access the recording here and accompanying cheat-sheet here. 

Overview

PANW provides good device visibility in their UI and management tools, but unexpected obscure conditions can threaten to take traffic down. You can create your own air traffic control to provide measurements to help you sustain confidence in your firewall. Here, we share insight we’ve accumulated so that you can become better practitioners at running and gaining visibility into your own Palo Alto NGFWs through XML/API. For more details, access our webinar recording and/or one-pager with main takeaways. 

What do we mean by visibility?

Many existing tools focus on management and policy. For instance: “What do I want to let in?” “What do I want to block?” “What analysis do I want on which layers?” This means the visibility provided by these tools focuses on high level activity, like accepts, drops, and findings, versus low-level activity, like basic up-time. 

However, fundamental level misconfigurations have the biggest impact. It is this level at which monitoring – and thus visibility – is most valuable.

Why monitoring?

As analysts and administrators, our attention needs to be focused on monitoring. Over time, this term has become less and less interesting. But specific information can be pulled, that, when done right, i.e. SSH with the CLI, isn’t visible through external tools. Doing so, we can pre-diagnose and prevent problems before they happen.

Change management issues?

When one makes a configuration change, it is documented via committee and processes. But when we monitor via the command line, we are not making any configuration changes. We are doing something fairly lightweight that will not change the performance envelope of our devices. If we used role-based access control to set up service-only login with root only permissions, this is the only configuration change. Then we can use monitoring to focus on problems that we have, in order to build up the visibility necessary to keep up and running. 

Methodology

Troubleshooting vs. Tracking Mindset

A lot of monitoring revolves around a troubleshooting mindset: “Tell me what broke and what the reason was. I’ll go fix it.”

Instead, we advocate for a tracking mindset. This mindset keeps prioritizes continuous monitoring, allowing us to understand the system when it’s normal and to do real projection planning. 

Ongoing monitoring allows us to accomplish 2 goals. The first is speeding time for remediation. Ongoing monitoring helps to pinpoint when a firewall was last working and what has changed since. It also enables us to determine what is actually broken versus odd, but normal behavior. The second goal is enabling an admin to predict trouble. By having deep visibility, we are able to determine when an upgrade is needed in the long term, or if something is potentially going to break in the short term. 

If we stick to the troubleshooting mindset, we may find some odd behavior turns out is totally normal. We wouldn’t be able to determine in advance at what point the capabilities of this hardware platform will no longer be sufficient.

Method

There’s a tendency to default to Panorama for NGFW, which provides management information. But this is the wrong place to look. We can’t pull out the running state of the devices beyond what’s built in the existing UI. 

To track the running state of firewalls, we must go through the CLI or XML/API (For clarification on how we refer to XML/API vs RESTful API, watch our webinar recording.)

What about SNMP?

SNMP is really good at providing high level metrics, and exposing individual first-level inputs you can use in your calculations. A lot of us have NOCs that rely on dispatch tools and SNMP traps to help figure out, “Is something going on right now?” The problem with SNMP is it requires its own set of skills and knowledge on top of the deep knowledge required for PAN-OS and NGFW. It is difficult to find someone with both skill sets, and it is easy to get lost in this complex mix. If all your time is spent here, it becomes a time sink. 

And then we begin to forget the skills we have. Instead, build on your current skill base. If you’re comfortable with the command line or SSH, pick up Expect. If you’re comfortable using Wget or curl to do RESTful type of calls, build on that as well.

Getting Started with XLM/API

Use API Key, not username/password

API Keys are useful as a password-equivalent, with limited severability from the password. API keys can be revoked (all at once, not individually) without changing the password. Changing the account password will also revoke all API keys.

Best Practices

Use a dedicated account for monitoring, set RBAC permissions for read-only, and create an account and setting permissions.

CLI Debug

Here, you can find a short-cut to generate XML/API calls: CLI Debug. 

Applying Knowledge

Tracking drops and general inabilities to connect

The single best starting point to get targeted detection of firewall misbehavior is to use counters via the CLI: 

• Get absolute count (since reboot or similar previous arbitrary point)
  • Show counter global filter severity drop
• Get relative count by running this query 1/minute with “delta yes”
  • Show counter global filter delta yes severity drop

You could also use the XML/API:

• curl -k -X GET ‘https://${ngfw_IP}/api?type=op&cmd=<show><counter><global><filter><delta>yes</delta><severity>drop</severity></filter></global></counter></show>&key=${api_key}’

A hybrid solution would involve enabling logging for specific global counters, and combining this with targeted packet capture. 

Utilizing external feeds and services

External feeds are the basis for application, anti-virus, and threat detection. Services include Identity (basis for zero trust architecture), Demisto SOAR, Cortex content data lake, and Prisma cloud security.

To check for the frequency and action for Threats/Anti-Virus, you could execute the following commands. The response would include both frequency and policy. 

• curl -k -X GET ‘https://${ngfw_IP}/api?type=config&action=get&xpath=/config/devices/entry/deviceconfig/system/*/threats&key=${api_key}’
• curl -k -X GET ‘https://${ngfw_IP}/api?type=config&action=get&xpath=/config/devices/entry/deviceconfig/system/*/anti-virus&key=${api_key}’

About Indeni

Indeni is the crowd-sourced automation platform for network and security infrastructure. With Indeni Crowd and Indeni Knowledge, organizations gain access to a living repository of automation tasks across maintenance, high availability, network visibility, security, compliance and vendor best practices. Teach your team co-development processes alongside the largest community of certified IT professionals and reduce the total cost of ownership with prescriptive steps to resolve issues across firewalls, routers and switches.