Exporting Data to Prometheus and Grafana

To merge data from Traffic Sentinel into dashboards that also display data from other sources you can export it into a time-series database such as Prometheus or Influx-DB. This tutorial will show you how to do that. To run this example you will need a system running Docker (which may be your laptop or desktop) that has access to your Traffic Sentinel server.

In these examples the IP addresses are:

  • Traffic Sentinel = 10.0.0.114
  • Prometheus Container = 10.0.0.72:9090
  • Grafana Container = 10.0.0.72:3000

You will need to substitute your own IP addresses where directed below.

1. Install "Scraper" script under File>REST

Install the following script under File>REST, call it prometheus_query.js and substitute your Prometheus IP address into the authClients: setting at the top to allow unpassworded access to the REST call from Prometheus.

// author: InMon
// date: 3/15/20
// version: 1.0
// description: See https://inmon.com/tutorials8/prometheus_grafana.php
// ttl: 60
// resultFormat: txt
// authClients: 10.0.0.72
// input_view     : Database view
// input_value    : list of value fields (for Sentinel query)
// input_metric   : list of value names (Prometheus metrics)
// input_key      : list of key fields (for Sentinel query)
// input_label    : list of key names (Prometheus labels)
// input_scale    : (optional) list of value scale factors
// input_over     : (optional) suppress values below this
// input_other    : (optional) substitute for null key
// input_n        : (optional) truncate for topN query
// input_where    : (optional) filter expression
// input_interval : (optional) time interval

include("prometheus-1.0.js");

var view     = view     || "ifcounters";
var metric   = metric   || null;
var key      = key      || "name(agent),name(interface),ifalias";
var label    = label    || null;
var value    = value    || "llbytes";
var scale    = scale    || null;
var n        = n        || null;
var other    = other    || null;
var over     = over     || null;
var where    = where    || null;
var interval = interval || "final - 1 min, final";

var keys = key ? key.split(",") : [];
var labels = label ? label.split(",") : [];
var values = value ? value.split(",") : [];
var metrics = metric ? metric.split(",") : [];
var scales = scale ? scale.split(",") : [];

var select = keys.concat(values).join(",");
// only sort (by first value) if truncating to topN
var sort = n ? values[0] : null;
var query = Query.topN(view, select, where, interval, sort, n);
// null key eliminates row unless 'other' string specified
query.nullsinkeys = (other != null);
query.run(function(row) {
  var kSet = {};
  for(var kk in keys) {
    kLabel = labels[kk] || keys[kk];
    var ky = row[kk];
    if(ky == null) ky = other;
    kSet[kLabel] = ky;
  } 
  for(var vv in values) {
    var met = metrics[vv] || values[vv];
    var val = row[keys.length + parseInt(vv)];
    if(scales[vv]) val *= scales[vv];
    if(over == null || val > over) printPrometheus(met, val, kSet);
  }
});

2. Run Prometheus

On your Docker server, create a new directory called something like SentinelPrometheus and cd into it. Then inside create a directory called data. And at the same level add the file prometheus.yml containing the following config, substituting the target IP of your Traffic Sentinel server:

global:
  scrape_interval:     60s
  evaluation_interval: 60s

rule_files:
  # - "first.rules"
  # - "second.rules"

scrape_configs:
  - job_name: 'sentinel-topn'
    metrics_path: /inmsf/Q/prometheus_query.js
    static_configs:
      - targets: ['10.0.0.114']
    params:
      view: ['traffic']
      metric: ['sentinel_traffic_bps']
      value: ['rate(bytes)']
      scale: ['8']
      key: ['sourcezone,sourcegroup,destinationzone,destinationgroup,serverport,protocolstack,country(sourceaddress),country(destinationaddress),name(asn(sourceaddress)),name(asn(destinationaddress)),domain(sourcename 2),domain(destinationname 2)']
      label: ['srcZone,srcGroup,dstZone,dstGroup,protocol,stack,srcCountry,dstCountry,srcAS,dstAS,srcDomain,dstDomain']
      n: ['10']
      other: ['other']

Still in the SentinelPrometheus directory, start the Prometheus database with this command:

docker run --name prometheus --rm -v $PWD/data:/prometheus -v $PWD/prometheus.yml:/etc/prometheus/prometheus.yml -p 9090:9090 -d prom/prometheus

You should now be able to connect to Prometheus by substituting the IP address into this URL:

http://10.0.0.72:9090

Now navigate to Status>Targets and test the Scraper script by clicking on its EndPoint URL:

You should see something like this:

sentinel_traffic_bps{srcZone="Zone 1",srcGroup="Group 1",dstZone="Zone 1",dstGroup="Group 1",protocol="TCP:3260",stack="MAC.IP.TCP",srcCountry="other",dstCountry="other",srcAS="ASN0",dstAS="ASN0",srcDomain="inmon.com",dstDomain="inmon.com"} 3037440
sentinel_traffic_bps{srcZone="Zone 1",srcGroup="Group 1",dstZone="Zone 1",dstGroup="Group 1",protocol="UDP:6343",stack="MAC.IP.UDP",srcCountry="other",dstCountry="other",srcAS="ASN0",dstAS="ASN0",srcDomain="inmon.com",dstDomain="inmon.com"} 112604.8
sentinel_traffic_bps{srcZone="Zone 1",srcGroup="Group 1",dstZone="Multicast",dstGroup="Multicast",protocol="UDP:5353",stack="MAC.IP.UDP",srcCountry="other",dstCountry="other",srcAS="ASN0",dstAS="ASN0",srcDomain="inmon.com",dstDomain="other"} 57654.64
sentinel_traffic_bps{srcZone="Zone 1",srcGroup="Group 1",dstZone="Zone 1",dstGroup="Group 1",protocol="UDP:26343",stack="MAC.IP.UDP",srcCountry="other",dstCountry="other",srcAS="ASN0",dstAS="ASN0",srcDomain="other",dstDomain="inmon.com"} 39903.44
sentinel_traffic_bps{srcZone="Zone 1",srcGroup="Group 1",dstZone="Zone 1",dstGroup="Group 1",protocol="UDP:161",stack="MAC.IP.UDP",srcCountry="other",dstCountry="other",srcAS="ASN0",dstAS="ASN0",srcDomain="inmon.com",dstDomain="other"} 5056.264
sentinel_traffic_bps{srcZone="Zone 1",srcGroup="Group 1",dstZone="Zone 1",dstGroup="Group 1",protocol="ICMP:3",stack="MAC.IP.ICMP",srcCountry="other",dstCountry="other",srcAS="ASN0",dstAS="ASN0",srcDomain="inmon.com",dstDomain="inmon.com"} 3826.4
sentinel_traffic_bps{srcZone="EXTERNAL",srcGroup="EXTERNAL",dstZone="EXTERNAL",dstGroup="EXTERNAL",protocol="ETHERNET:2054",stack="MAC.ARP",srcCountry="other",dstCountry="other",srcAS="ASN0",dstAS="ASN0",srcDomain="other",dstDomain="other"} 0

Prometheus will now be collecting data from Traffic Sentinel every minute. You can confirm this, and check the performance, by visiting the File>REST>Statistics page on Traffic Sentinel.

3. Run Grafana

Still in the SentinelPrometheus directory on your Docker Server, create another directory called GrafanaData. Then run Grafana like this:

docker run --name grafana -v $PWD/grafanaData:/var/lib/grafana -p 3000:3000 -d grafana/grafana

You can now connect to Grafana by substituting your IP into this URL:

http://10.0.0.72:3000

and log in using the default user=admin pass=admin.

Next, click to add the Prometheus data source. The URL for the data source will be the http://10.0.0.72:9090 URL that you connected to before:

4. Install Dashboard

On the Graphana page, click on the '+' and select import to install a pre-made dashboard by pasting the json spec from here.

The result should be a dashboard looking like this that updates every minute:

5. Adding Dashboard Panels in Grafana

Now you are free to add additional dashboard panels using the metric sentinel_traffic_bps that was defined in the prometheus.yml configuration above. When you define chart panels you should set the value units to data(rate) / Bits/sec to match the numbers provided by the Traffic Sentinel script. The metric is broken out by a number of dimensions (labels), but you can roll sub-totals together when you define dashboard panels using an expression such as:

topk(10,sum(sentinel_traffic_bps) by (srcZone,srcGroup))

Note that the tokens in this expression are the metric name and the labels from the prometheus.yml file above.

6. Adding Metrics for Interface Counters and Host Counters

To make additional metrics available to Grafana you need only add more jobs to the prometheus.yml config file and then restart the Prometheus container. So the sequence looks like this:

docker stop prometheus
### edit prometheus.yml ###
docker run --name prometheus --rm -v $PWD/data:/prometheus -v $PWD/prometheus.yml:/etc/prometheus/prometheus.yml -p 9090:9090 -d prom/prometheus

Below is an expanded prometheus.yml file that uses the same Traffic Sentinel File>REST script to extract a subset of the interface counters and host counters every minute:

global:
  scrape_interval:     60s
  evaluation_interval: 60s

rule_files:
  # - "first.rules"
  # - "second.rules"

scrape_configs:
  - job_name: 'sentinel-topn'
    metrics_path: /inmsf/Q/prometheus_query.js
    static_configs:
      - targets: ['10.0.0.114']
    params:
      view: ['traffic']
      metric: ['sentinel_traffic_bps']
      value: ['rate(bytes)']
      scale: ['8']
      key: ['sourcezone,sourcegroup,destinationzone,destinationgroup,serverport,protocolstack,country(sourceaddress),country(destinationaddress),name(asn(sourceaddress)),name(asn(destinationaddress)),domain(sourcename 2),domain(destinationname 2)']
      label: ['srcZone,srcGroup,dstZone,dstGroup,protocol,stack,srcCountry,dstCountry,srcAS,dstAS,srcDomain,dstDomain']
      n: ['10']
      other: ['other']
  - job_name: 'sentinel-counters'
    metrics_path: /inmsf/Q/prometheus_query.js
    static_configs:
      - targets: ['10.0.0.114']
    params:
      view: ['ifcounters']
      metric: ['sentinel_ifinoctets,sentinel_ifoutoctets,sentinel_ifinunicasts,sentinel_ifoutunicasts,sentinel_ifinmulticasts,sentinel_ifoutmulticasts,sentinel_ifinbroadcasts,sentinel_ifoutbroadcasts,sentinel_ifinerrors,sentinel_ifouterrors,sentinel_ifindiscards,sentinel_ifoutdiscards']
      value: ['ifinoctets,ifoutoctets,frames_in,frames_out,ifinmulticasts,ifoutmulticasts,ifinbroadcasts,ifoutbroadcasts,ifinerrors,ifouterrors,ifindiscards,ifoutdiscards']
      key: ['zone,group,agent,ifindex,name(agent),name(interface),ifalias']
      label: ['zone,group,agentIP,ifindex,agent,interface']
      over: ['1']
      where: ['ifalias!=""']
  - job_name: 'sentinel-host-counters'
    metrics_path: /inmsf/Q/prometheus_query.js
    static_configs:
      - targets: ['10.0.0.114']
    params:
      view: ['host']
      metric: ['loadAvg,CPU,MEM,DISK,TCP_retransmissions']
      value: ['load_one,cpu_util,mem_used_pc,diskpartmax,tcp_retrans_segs']
      key: ['zone,group,agent,hostname,os_release']
      label: ['zone,group,agentIP,host,os']

If you install this prometheus.yml file, substitute the correct target IP address, and restart the Prometheus container the new metrics will appear in the Grafana UI. Note that the filter is being used to select counters only from interfaces that have a non-empty ifAlias string, and the parameter over=1 is suppressing all zero counter values.

7. Explanation

The fields in each job-name section in the prometheus.yml file are used to assemble a URL, and the list of params become the variables passed with it. So the sentinel-host-counters job above will call the URL:

http://10.0.0.114/inmsf/Q/prometheus_query.js

and pass these parameters:

view=host&metric=loadAvg,CPU,MEM,DISK,TCP_retransmissions&value=load_one,cpu_util,mem_used_pc,diskpartmax,tcp_retrans_segs&key=zone,group,agent,hostname,os_release&label=zone,group,agentIP,host,os

Those parameters will become the variables in the File>REST script in Traffic Sentinel:

  • view = Traffic Sentinel database view.
  • value = the value fields for the Traffic Sentinel query as a comma-separated list.
  • metric = Grafana names that you choose for those value fields (e.g. so you can prefix them with "sentinel_" to distinguish easily from other data sources).
  • key = the key fields for the Traffic Sentinel query.
  • label = Grafana label names that you choose for those key fields.
  • scale = optional scale factors for the values.
  • n = optional topN truncation. This typically only makes sense when there is just one value. It allows you to make queries that would otherwise return huge datasets and cause scalability problems in Prometheus.
  • minval = optional minimum value. The script will only export data to Prometheus that exceeds this value.
  • other = optional substitution string when a key result is otherwise null.

Related Topics