• How To's
  • Building Powerful Grafana Dashboards

Building Powerful Grafana Dashboards

By using Indeni, security infrastructure teams have access to many metrics. Between Check Point and Palo Alto Networks firewalls, Indeni collects over 500 unique metrics. While the Indeni WebUI provides dashboards and custom reporting capabilities, enterprises are often relying on third party tools for even more compelling diagrams and customized dashboards. They want to bring together unique data sources from many vendors and different device types. One such tool is the very popular open source Grafana

This post is intended for a technical audience who is planning to develop custom dashboards from scratch. Knowledge of Grafana and JSON is assumed. The post takes you from the initial integration with Grafana to understanding details about the Indeni metrics and the structure of the data. It will cover tools you can use to discover the metrics, how to interpret them and correctly parse them while building out powerful dashboards. This post will not cover the Grafana installation as there are a lot of great resources out there. Configuring Grafana to query the Indeni platform is straightforward and it is covered in our documentation

Indeni integration with Grafana

There are two types of integration.

1. Indeni Plugin for Grafana

Grafana-Indeni is a plugin for Grafana allowing users to visualize a subset of the data collected by the Indeni platform. The plugin is intended to make plotting your data easy using a number of graphs and reports. There is no need to understand the structure of the data and no parsing is required. You can find more information about the plugin here

2. JSON API

The JSON API plugin is a popular open source data source plugin for loading JSON APIs into Grafana using JSONata. It executes requests against Indeni’s in-memory database and parses JSON responses into Grafana dataframes. It provides the flexibility a programmer expects. Indeni’s in-memory database consists of data collected from all the devices in the last 90 minutes. Effectively, this has a lot more data points than the Indeni plugin for Grafana. We used the JSON API plugin to build out compelling dashboards. With the JSON API plugin, you need to understand how the data is structured and have knowledge on query languages such as JSONata. Although it is more complex than the Indeni plugin, it is more powerful and flexible. 

Introducing iktool (Indeni Knowledge Tool)

Indeni Knowledge tool (iktool) is a powerful CLI tool we use to develop Auto-Detect elements. It provides a lot of useful information about the devices Indeni is managing, device unique identifiers and the metrics collected for the device. Open the SSH terminal on your indeni server and run the iktool command.

indeni@indeni-server:~$ iktool
Usage: iktool [-h] <options> [parameters]

op
    devices_show_all
    device_info_show            <device_id>
    alert_delete_all            <device_id>
    alert_delete_single         <alert_id>
    metric_get                  <metric_name> <device_id>
    metric_show_all             <device_id>
    overwrite_file_create       <path_to_stable_ind_file>
    overwrite_files_show
    overwrite_file_delete       <all|<specific_ind_file>
    knowledge_version_show
    catalog                     <vendor> <metric>

workflow
    real_device_workflow        <config_file>
    real_device_block           <config_file>
    mock_data_workflow          <config_file>
    mock_data_block             <config_file>

snapshot
    create
          json
              ind_per_device    <target_file_name> <devices.txt>
    compare
          json                  <old_json_file> <new_json_file>

logs
    devices
          backup
    show
          device                <device_id>
    show                        <collector.log|indeni.log|backup.log>

--help, -h
--version

indeni@indeni-server:~$

We will show you how to use the tool to retrieve the available metrics for a device and see the data structure. Before we do that, let us introduce you to the Indeni metrics.

Understanding the Indeni metrics

Indeni Time Series Data

Indeni collects data at different points in time. Depending on the metric, the collection interval varies from one minute to one hour. Plotting time series data on a graph makes it much easier to digest the data and tracks changes over time. 

Most metrics collected by Indeni are numeric in form and are called double metrics. They are a number with a decimal point, such as 0.0, 1.0, 123.45, etc. A lot of the metrics are boolean representing true or false, up or down state, enabled or disabled. We use double metrics to represent a current state. The value of 1.0 means true, up or enabled. Some of the metrics are measurements such as number of packets. Some metrics are percentages such as CPU usage. 

We also have complex metrics that are not numeric, e.g. serial number, a list of files. These metrics are used to check the current configuration of a device. A complex metric can be a simple string and it is stored in a JSON format. 

{“value”:”America/San_Francisco”}

A complex metric can also be a list of items, stored in a JSON array, e.g. a list of core dump files. 

[{“path”: “/somepath1”, “created”: “14502281”}, {“path”: “/somepath2”, “created”: “14502351”}]

Now that you have some idea of what the metrics look like, let’s take a look at a simple example. 

What does the data look like? 

First, let’s pick a device by retrieving the list of device identifiers using iktool. The queries we’ll be using require the device’s unique identifier.

indeni@indeni-server:~$ iktool op 'devices_show_all'
Device info for all devices
id                                    ip_address       name
------------------------------------  ---------------  ---------------------
22e63ff5-2dc4-472c-be4f-195b010f3ecb  10.11.94.142     CP-R81-VSX1-1
78c11d23-f90e-4a8c-b247-fb1f6f03241f  10.11.95.5       PAN-FW1-10.0
8815612b-c984-4a63-88c2-ef70450e7f4e  10.11.94.134     CP-R81-MGMT-1
5b8f2e69-cddb-4580-ae28-247073bb391e  172.16.20.81     panos
8b934d98-c1a5-46f4-bd21-aebe743031b0  10.11.93.71      FortiGate-V7_0_5-FW01
8b7bdc17-94a3-42d5-b957-beac8acaf35e  10.11.94.132     CP-R81-VRRP-1
6ef0290e-1273-41bd-89cb-ead8ec0ff828  10.11.94.131     CP-R81-GW2
b582b403-beca-4ba3-9bab-f247c1d57cd7  10.11.94.133     CP-R81-VRRP-2
964a252d-d6c6-4319-9d73-0804645bffe3  10.11.94.137     CP-R81-MDS-2
28785bcf-dac9-4f8e-81c8-529ad1fae2b9  10.11.94.108     CP-R80.20SP-VSX
2095d544-809e-445d-aa2e-1a31d8510282  10.11.89.22      Arbor-AED-6.7.0.0
31a68a41-734a-4b90-abfc-0d766687cfe6  10.11.80.1       CP-R80.40-GW1-1
724d8cff-ad4f-4fc5-aa4a-76157d979343  10.11.94.136     CP-R81-MDS-1
ce360d5b-9918-424d-832f-e2381b842243  10.11.94.150     CP-R81.10-GW1
2a7a25b6-b971-47e0-bb54-ee6768dc51dd  10.11.95.12      PAN-Panorama02-10.0
c9ce94a2-57e3-42f8-a154-beb75f64ad25  10.11.95.11      PAN-Panorama01-10.0
cc2916e3-28da-453a-a25c-7e429f33a8e4  10.11.95.6       PAN-FW2-10.0
d29a1e94-6104-4204-b773-752b2096f20f  10.11.80.41      RedHat7
edd8de4b-82e5-47d6-8ff9-beea5f8f0483  10.11.94.135     CP-R81-MGMT-2
52837394-b6e7-4a4a-ad54-a814f5b6b6d9  10.11.80.11      CP-R80.40-VSX1-1
b104f20b-908b-4d85-859a-0fde1bf393c9  10.11.93.72      FortiGate-V7_0_5-FW02
46f2bf23-1be4-4c74-8bdd-605cc53e4adc  10.11.80.2       CP-R80.40-GW1-2
09317a32-6b5e-4692-ad5f-73fd190be7fa  10.10.10.131     UbuntuIndeniShouky
d96360f1-7d40-4c9f-bc42-8ea8cef27ee3  10.11.80.40      RedHat
b305d370-c48a-4404-a9ff-89cb3b3cfac6  10.11.94.143     CP-R81-VSX-1-2
245312f1-5557-4e8d-8dbc-ee7f09d9228d  10.11.94.144     CP-R81-MLM
8db53fec-4386-4bac-9d93-d0b296ebeb34  10.11.88.1       IL-Fortinet-VPN
961f3eeb-8dc3-4709-9a9a-1727a8c41ffc  10.11.93.62      FortiGate-V6_4_8-FW02
80fed3ad-1d8b-4a8c-9975-64f3bfe9c264  10.11.80.52      Maestro Orchestrator
dd6f15f4-b9b0-40d8-81dd-4f5a41668ab0  10.11.94.130     CP-R81-GW1
ba1ff0ce-6d3a-4c6e-8f6d-37e98f7a37c5  192.168.193.223  ASA1
410881c5-533d-43ff-8395-b7bd822bdc7c  172.17.0.1       indeni Server
ab8df5f1-e44d-4082-9cea-244474d0fa0c  192.168.193.224  ASA2

Let’s look at a simple complex metric example. Say we want to find out the device vendor. Let’s pick PAN-FW1-10.0, a Palo Alto Networks firewall. Grab the device-id. Select the metric vendor.

indeni@indeni-server:~$ iktool op "metric_get" vendor 78c11d23-f90e-4a8c-b247-fb1f6f03241f
[{'ind_name': 'panos-show-system-info-monitoring',
  'metric_type': 'complex_metric',
  'points': [{'fs-ts': 1649194598519,
              'jsval': {'value': 'Palo Alto Networks'},
              'ts': 1649347199421},
             None,
             None],
  'start': 1649342003441,
  'step': -1,
  'tags': {'device-id': '78c11d23-f90e-4a8c-b247-fb1f6f03241f',
           'display-name': 'Vendor',
           'im.name': 'vendor',
           'im.step': '300',
           'ind.name': 'panos-show-system-info-monitoring',
           'live-config': 'true'}}]
indeni@indeni-server:~$

The response to the command is in a JSON array format. The metric vendor is a complex metric since it is not a number. The 3rd item in the array, “points”,  is a JSON array itself. As expected, you can see the value “Palo Alto Networks”. Later on, we will see how we can extract just the vendor from the JSON document. 

Let’s look at a double metric example, say a metric showing states. Using the same device, let’s select the metric bgp-state.  

indeni@indeni-server:~$ iktool op "metric_get" bgp-state 78c11d23-f90e-4a8c-b247-fb1f6f03241f
[{'ind_name': 'panos-show-protocol-bgp-peer',
  'metric_type': 'double_metric',
  'points': [None,
             None,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0,
             None,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0,
             1.0],
  'start': 1649261280000,
  'step': 60000,
  'tags': {'device-id': '78c11d23-f90e-4a8c-b247-fb1f6f03241f',
           'display-name': 'BGP Peers - State',
           'im.dstype': 'gauge',
           'im.dstype.displaytype': 'boolean',
           'im.identity-tags': 'name',
           'im.name': 'bgp-state',
           'im.step': '60',
           'ind.name': 'panos-show-protocol-bgp-peer',
           'live-config': 'true',
           'name': 'BGP Group: PAN_10_BGP_peer_1 Peer Name: PAN_10_peer_bgp_1 '
                   'Peer Address: 10.0.102.3'}}]
indeni@indeni-server:~$
indeni@indeni-server:~$

You can see the metric name “bgp-state”. The 3rd item in the array, “points”,  is a JSON array itself. It consists of actual bgp states measured at a 1 minute interval, starting at time “1649261280000” in epoch format. The BGP state “1.0” means bgp is up in the last 90 minutes. The last item “name” is the name of the BGP peer that is helpful to include in the display.

Building Grafana Queries

Now that you know what the data looks like, you can build a Grafana query using the query editor to fetch the information you want to display in Grafana. Let’s look at a simple one first. In this case, we want to grab just the vendor name from the query response. 

The Grafana query request: 

GET /metrics?query=(device-id==’$device_id’;im.name==’vendor’)&start=$__from

From the Path tab of the query editor: 

You use a query language such as JSONata to extract the vendor name from the JSON query response. 

The query expression to extract the value “Palo Alto Networks”:

*.points[0].jsval.value.$string()

The Fields tab is where you specify the query expression. 

Let’s look at a more interesting example, the bgp-state metric. In this case, you want to extract several values in a time series data format to draw a graph. Again, you use JSONata to extract the states, the corresponding timestamps and the BGP peer name from the JSON response. 

The grafana query: 

GET /metrics?query=(device-id==’$device_id’;im.name==’bgp-state’)&start=$__from 

From the Path tab of the query editor: 

Specify the query expression in the  Fields tab. 

The query expression for BGP states:

$map($, function($v, $i, $a){$map([$v].points, function($vv, $ii, $aa) {{“point”:[$v].points[$ii],”timestamp”:[$v].($number(start)+(($number($ii)*$number(step)))),”tag”:[$v].tags.name}})}).point

The query expression for timestamps:

$map($, function($v, $i, $a){$map([$v].points, function($vv, $ii, $aa) {{“point”:[$v].points[$ii],”timestamp”:[$v].($number(start)+(($number($ii)*$number(step)))),”tag”:[$v].tags.name}})}).timestamp

The query expression for the BGP peer: 

$map($, function($v, $i, $a){$map([$v].points, function($vv, $ii, $aa) {{“point”:[$v].points[$ii],”timestamp”:[$v].($number(start)+(($number($ii)*$number(step)))),”tag”:[$v].tags.name}})}).tag

You can see the result of the query expression by enabling Table view from the query editor.  

You can see the data is correctly extracted and populated in the form of a Grafana dataframe. 

Getting Started

While it’s easy to get started with Grafana and there are plenty of resources out there to help you, it can be challenging if you have to build the dashboard from scratch. It is certainly very time consuming. That’s why we did all the heavy lifting by building curated dashboards for you. Check out our Grafana-indeni dashboard repository in Bitbucket. You will find the JSON files of these dashboards containing everything you need such as layout, variables, styles, data sources, queries, query expressions to extract the data, etc. 

Once you import these JSON files into your Grafana environment, you can easily customize the dashboards to meet your needs. It is also easy to build your own query now that you’ve samples of all the queries and query expressions. We’ve plans to add a lot more dashboards going forward so remember to check back from time to time for new panels and dashboards. 

Tips & Tools

1. Retrieving a list of full metrics from Indeni

Using iktool, you can retrieve a full list of metrics of a device. Let’s get the list from a Check Point GAiA device.

indeni@indeni-server:~$ iktool op 'metric_show_all' 31a68a41-734a-4b90-abfc-0d766687cfe6
arp-limit
arp-table
arp-total-entries
bgp-active-routes-counter
bgp-hidden-routes-counter
bgp-state
ca-accessible
chkp-allowed-host-any
chkp-auditlog-enabled
chkp-max-login-failed-attempts
chkp-mgmtauditlogs-enabled
chkp-os-bit
chkp-os-build
chkp-os-kernel
chkp-password-complexity
chkp-systemlog-enabled
cluster-member-active
cluster-member-active-live-config
cluster-member-states
cluster-mode
cluster-preemption-enabled
cluster-state
cluster-state-live-config
cluster-vip
clusterxl-ccp-mode
clusterxl-pnote-state
concurrent-connections
concurrent-connections-limit
concurrent-connections-limit-snapshot
concurrent-connections-usage
config-unsaved
configuration-content
connected-networks-table
core-dumps
coredumping-enabled
cores-locked
cphaprob-required-interfaces
cphaprob-required-secured-interfaces
cphaprob-up-interfaces
cphaprob-up-secured-interfaces
cpu-usage
current-datetime
debug-status
destination-cache-limit
destination-cache-usage
disk-total-kbytes
disk-usage-percentage
disk-used-kbytes
dns-response-time
dns-server-state
dns-servers
domain
failed-logins
fast-accel-status
fast-accel-table-entries
fast-accel-table-status
features-enabled
file-modifed-last-day
hostname
hostname-exists-etc-hosts
hotfix-jumbo-take
hotfixes
inode-total-count
inode-used-count
interface-hw
ipassignment-conf-errors
kernel-table-actual
kernel-table-limit
license-expiration
licenses
lines-config-files
listening-ports
load-average-fifteen-minutes
load-average-fifteen-minutes-live-config
load-average-five-minutes
load-average-five-minutes-live-config
load-average-one-minute
load-average-one-minute-live-config
local-logging-status-live-config
log-kernel-connection-refused
log-server-communicating
logged-in-users
login-banner
memory-free-kbytes
memory-total-kbytes
memory-usage
min-passwd-length
model
network-interface-admin-state
network-interface-admin-state-logical
network-interface-duplex
network-interface-ipv4-address
network-interface-ipv4-subnet
network-interface-mac
network-interface-mtu
network-interface-rx-bits
network-interface-rx-dropped
network-interface-rx-dropped-percentage
network-interface-rx-error-percentage
network-interface-rx-errors
network-interface-rx-frame
network-interface-rx-overruns
network-interface-rx-packets
network-interface-rx-util-percentage
network-interface-speed
network-interface-state
network-interface-state-logical
network-interface-tx-bits
network-interface-tx-carrier
network-interface-tx-dropped
network-interface-tx-dropped-percentage
network-interface-tx-error-percentage
network-interface-tx-errors
network-interface-tx-overruns
network-interface-tx-packets
network-interface-tx-util-percentage
network-interface-type
network-interfaces
not-logging-locally
ntp-servers
os-name
os-version
pbr-rules
policy-install-last-modified
policy-installed
policy-name
process-cpu
process-memory
process-state
process_state_snmpd_state
process_state_syslogd_state
radius-servers
routes-missing-kernel
securexl-accept-template-status
securexl-drop-template-status
securexl-nat-template-status
securexl-status
serial-numbers
snmp-communities
snmp-contact
snmp-enabled
snmp-location
snmp-traps-receiver
snmp-traps-status
snmp-users
snmp-version
software-eos-date
ssh-timeout
ssh-version-1-enabled
static-arp
static-routing-table
syslog-servers
tacacs-enabled
tacacs-servers
tasks-zombies
telnet-enabled
timezone
unencrypted-snmp-configured
uptime-milliseconds
user-id
users
vendor
vmalloc-total-kbytes
vmalloc-used-kbytes
vpn-tunnel-state
indeni@indeni-server:~$

2. Query Inspector

Throughout development, one of the built-in tools we used frequently to troubleshoot issues is Query Inspector. This is helpful if your graph doesn’t show any data or the result isn’t what you expected. You can see the error response from the query. 

Here, you can see if the data is correctly extracted and interpreted for display. The actual value of the state is 1.0, but instead you want to translate the value 1.0 to “Up”.

3. Exploring the Indeni API

To explore the Indeni API using CLI or tools such as Postman, use this API URL Path. 

See below an example of making a GET request from Postman:

4. Useful Resources

JSONata resources:

JSONPath resources:

Conclusion

There is a lot of data you can extract from the Indeni server. With Grafana, you can now visualize literally millions of data points. Humans respond to and process visual data better than any other type of data. Since we are visual by nature, we plan to invest more on Grafana so you can process the data more effectively. 

Go ahead and experiment with the dashboards. Enjoy!

BlueCat to acquire LiveAction

BlueCat adds LiveAction’s network observability and intelligence platform, which helps large enterprises optimize the performance, resiliency, and security of their networks.