Exporting Data to Prometheus and Grafana
To merge data from Traffic Sentinel into dashboards that also display data from other sources you can export it into a time-series database such as Prometheus or Influx-DB. This tutorial will show you how to do that. To run this example you will need a system running Docker (which may be your laptop or desktop) that has access to your Traffic Sentinel server.
In these examples the IP addresses are:
- Traffic Sentinel = 10.0.0.114
- Prometheus Container = 10.0.0.72:9090
- Grafana Container = 10.0.0.72:3000
You will need to substitute your own IP addresses where directed below.
1. Install "Scraper" script under File>REST
Install the following script under File>REST, call it prometheus_query.js and substitute your Prometheus IP address into the authClients: setting at the top to allow unpassworded access to the REST call from Prometheus.
// author: InMon
// date: 3/15/20
// version: 1.0
// description: See https://inmon.com/tutorials8/prometheus_grafana.php
// ttl: 60
// resultFormat: txt
// authClients: 10.0.0.72
// input_view : Database view
// input_value : list of value fields (for Sentinel query)
// input_metric : list of value names (Prometheus metrics)
// input_key : list of key fields (for Sentinel query)
// input_label : list of key names (Prometheus labels)
// input_scale : (optional) list of value scale factors
// input_over : (optional) suppress values below this
// input_other : (optional) substitute for null key
// input_n : (optional) truncate for topN query
// input_where : (optional) filter expression
// input_interval : (optional) time interval
include("prometheus-1.0.js");
var view = view || "ifcounters";
var metric = metric || null;
var key = key || "name(agent),name(interface),ifalias";
var label = label || null;
var value = value || "llbytes";
var scale = scale || null;
var n = n || null;
var other = other || null;
var over = over || null;
var where = where || null;
var interval = interval || "final - 1 min, final";
var keys = key ? key.split(",") : [];
var labels = label ? label.split(",") : [];
var values = value ? value.split(",") : [];
var metrics = metric ? metric.split(",") : [];
var scales = scale ? scale.split(",") : [];
var select = keys.concat(values).join(",");
// only sort (by first value) if truncating to topN
var sort = n ? values[0] : null;
var query = Query.topN(view, select, where, interval, sort, n);
// null key eliminates row unless 'other' string specified
query.nullsinkeys = (other != null);
query.run(function(row) {
var kSet = {};
for(var kk in keys) {
kLabel = labels[kk] || keys[kk];
var ky = row[kk];
if(ky == null) ky = other;
kSet[kLabel] = ky;
}
for(var vv in values) {
var met = metrics[vv] || values[vv];
var val = row[keys.length + parseInt(vv)];
if(scales[vv]) val *= scales[vv];
if(over == null || val > over) printPrometheus(met, val, kSet);
}
});
2. Run Prometheus
On your Docker server, create a new directory called something like SentinelPrometheus and cd into it. Add the file prometheus.yml containing the following config, substituting the target IP of your Traffic Sentinel server:
global:
scrape_interval: 60s
evaluation_interval: 60s
rule_files:
# - "first.rules"
# - "second.rules"
scrape_configs:
- job_name: 'sentinel-topn'
metrics_path: /inmsf/Q/prometheus_query.js
static_configs:
- targets: ['10.0.0.114']
params:
view: ['traffic']
metric: ['sentinel_traffic_bps']
value: ['rate(bytes)']
scale: ['8']
key: ['sourcezone,sourcegroup,destinationzone,destinationgroup,serverport,protocolstack,country(sourceaddress),country(destinationaddress),name(asn(sourceaddress)),name(asn(destinationaddress)),domain(sourcename 2),domain(destinationname 2)']
label: ['srcZone,srcGroup,dstZone,dstGroup,protocol,stack,srcCountry,dstCountry,srcAS,dstAS,srcDomain,dstDomain']
n: ['10']
other: ['other']
Add a persistent data volume to prometheus to use, like this:
sudo docker volume create prometheus-data
Still in the SentinelPrometheus directory, start the Prometheus database with this command:
sudo docker run --name prometheus --rm -v prometheus-data:/prometheus -v $PWD/prometheus.yml:/etc/prometheus/prometheus.yml -p 9090:9090 -d prom/prometheus
You should now be able to connect to Prometheus by substituting the IP address into this URL:
http://10.0.0.72:9090
Navigate to Status>Targets and test the Scraper script by clicking on its EndPoint URL:
You should see something like this:
sentinel_traffic_bps{srcZone="Zone 1",srcGroup="Group 1",dstZone="Zone 1",dstGroup="Group 1",protocol="TCP:3260",stack="MAC.IP.TCP",srcCountry="other",dstCountry="other",srcAS="ASN0",dstAS="ASN0",srcDomain="inmon.com",dstDomain="inmon.com"} 3037440
sentinel_traffic_bps{srcZone="Zone 1",srcGroup="Group 1",dstZone="Zone 1",dstGroup="Group 1",protocol="UDP:6343",stack="MAC.IP.UDP",srcCountry="other",dstCountry="other",srcAS="ASN0",dstAS="ASN0",srcDomain="inmon.com",dstDomain="inmon.com"} 112604.8
sentinel_traffic_bps{srcZone="Zone 1",srcGroup="Group 1",dstZone="Multicast",dstGroup="Multicast",protocol="UDP:5353",stack="MAC.IP.UDP",srcCountry="other",dstCountry="other",srcAS="ASN0",dstAS="ASN0",srcDomain="inmon.com",dstDomain="other"} 57654.64
sentinel_traffic_bps{srcZone="Zone 1",srcGroup="Group 1",dstZone="Zone 1",dstGroup="Group 1",protocol="UDP:26343",stack="MAC.IP.UDP",srcCountry="other",dstCountry="other",srcAS="ASN0",dstAS="ASN0",srcDomain="other",dstDomain="inmon.com"} 39903.44
sentinel_traffic_bps{srcZone="Zone 1",srcGroup="Group 1",dstZone="Zone 1",dstGroup="Group 1",protocol="UDP:161",stack="MAC.IP.UDP",srcCountry="other",dstCountry="other",srcAS="ASN0",dstAS="ASN0",srcDomain="inmon.com",dstDomain="other"} 5056.264
sentinel_traffic_bps{srcZone="Zone 1",srcGroup="Group 1",dstZone="Zone 1",dstGroup="Group 1",protocol="ICMP:3",stack="MAC.IP.ICMP",srcCountry="other",dstCountry="other",srcAS="ASN0",dstAS="ASN0",srcDomain="inmon.com",dstDomain="inmon.com"} 3826.4
sentinel_traffic_bps{srcZone="EXTERNAL",srcGroup="EXTERNAL",dstZone="EXTERNAL",dstGroup="EXTERNAL",protocol="ETHERNET:2054",stack="MAC.ARP",srcCountry="other",dstCountry="other",srcAS="ASN0",dstAS="ASN0",srcDomain="other",dstDomain="other"} 0
Prometheus will now be collecting data from Traffic Sentinel every minute. You can confirm this, and check the performance, by visiting the File>REST>Statistics page on Traffic Sentinel.
3. Run Grafana
Add another persistent data volume for Grafana to use:
sudo docker volume create grafana-dataand then run Grafana like this:
sudo docker run --name grafana -v grafana-data:/var/lib/grafana -p 3000:3000 -d grafana/grafana
You can now connect to Grafana by substituting your IP into this URL:
http://10.0.0.72:3000
and log in using the default user=admin pass=admin.
Click the gear wheel icon for Configuration > Data Sources to add the Prometheus data source. The URL for the data source will be the http://10.0.0.72:9090 URL that you connected to before:
4. Install Dashboard
On the Graphana page, click on the '+' and select import to install a pre-made dashboard by pasting the json spec from here and then clicking the Load button below.
On the next page assign the Prometheus data source and click Import.
The result should be a dashboard looking like this that updates every minute:
5. Adding Dashboard Panels in Grafana
Now you are free to add additional dashboard panels using the metric sentinel_traffic_bps that was defined in the prometheus.yml configuration above. When you define chart panels you should set the value units to data(rate) / Bits/sec to match the numbers provided by the Traffic Sentinel script. The metric is broken out by a number of dimensions (labels), but you can roll sub-totals together when you define dashboard panels using an expression such as:
topk(10,sum(sentinel_traffic_bps) by (srcZone,srcGroup))
Note that the tokens in this expression are the metric name and the labels from the prometheus.yml file above.
6. Adding Metrics for Interface Counters and Host Counters
To make additional metrics available to Grafana you need only add more jobs to the prometheus.yml config file and then restart the Prometheus container. So the sequence looks like this:
docker stop prometheus ### edit prometheus.yml ### docker run --name prometheus --rm -v $PWD/data:/prometheus -v $PWD/prometheus.yml:/etc/prometheus/prometheus.yml -p 9090:9090 -d prom/prometheus
Below is an expanded prometheus.yml file that uses the same Traffic Sentinel File>REST script to extract a subset of the interface counters and host counters every minute:
global:
scrape_interval: 60s
evaluation_interval: 60s
rule_files:
# - "first.rules"
# - "second.rules"
scrape_configs:
- job_name: 'sentinel-topn'
metrics_path: /inmsf/Q/prometheus_query.js
static_configs:
- targets: ['10.0.0.114']
params:
view: ['traffic']
metric: ['sentinel_traffic_bps']
value: ['rate(bytes)']
scale: ['8']
key: ['sourcezone,sourcegroup,destinationzone,destinationgroup,serverport,protocolstack,country(sourceaddress),country(destinationaddress),name(asn(sourceaddress)),name(asn(destinationaddress)),domain(sourcename 2),domain(destinationname 2)']
label: ['srcZone,srcGroup,dstZone,dstGroup,protocol,stack,srcCountry,dstCountry,srcAS,dstAS,srcDomain,dstDomain']
n: ['10']
other: ['other']
- job_name: 'sentinel-counters'
metrics_path: /inmsf/Q/prometheus_query.js
static_configs:
- targets: ['10.0.0.114']
params:
view: ['ifcounters']
metric: ['sentinel_ifinoctets,sentinel_ifoutoctets,sentinel_ifinunicasts,sentinel_ifoutunicasts,sentinel_ifinmulticasts,sentinel_ifoutmulticasts,sentinel_ifinbroadcasts,sentinel_ifoutbroadcasts,sentinel_ifinerrors,sentinel_ifouterrors,sentinel_ifindiscards,sentinel_ifoutdiscards']
value: ['ifinoctets,ifoutoctets,frames_in,frames_out,ifinmulticasts,ifoutmulticasts,ifinbroadcasts,ifoutbroadcasts,ifinerrors,ifouterrors,ifindiscards,ifoutdiscards']
key: ['zone,group,agent,ifindex,name(agent),name(interface),ifalias']
label: ['zone,group,agentIP,ifindex,agent,interface']
over: ['1']
where: ['ifalias!=""']
- job_name: 'sentinel-host-counters'
metrics_path: /inmsf/Q/prometheus_query.js
static_configs:
- targets: ['10.0.0.114']
params:
view: ['host']
metric: ['loadAvg,CPU,MEM,DISK,TCP_retransmissions']
value: ['load_one,cpu_util,mem_used_pc,diskpartmax,tcp_retrans_segs']
key: ['zone,group,agent,hostname,os_release']
label: ['zone,group,agentIP,host,os']
If you install this prometheus.yml file, substitute the correct target IP address, and restart the Prometheus container the new metrics will appear in the Grafana UI. Note that the filter
7. Explanation
The fields in each job-name section in the prometheus.yml file are used to assemble a URL, and the list of params become the variables passed with it. So the sentinel-host-counters job above will call the URL:
http://10.0.0.114/inmsf/Q/prometheus_query.js
and pass these parameters:
view=host&metric=loadAvg,CPU,MEM,DISK,TCP_retransmissions&value=load_one,cpu_util,mem_used_pc,diskpartmax,tcp_retrans_segs&key=zone,group,agent,hostname,os_release&label=zone,group,agentIP,host,os
Those parameters will become the variables in the File>REST script in Traffic Sentinel:
- view = Traffic Sentinel database view.
- value = the value fields for the Traffic Sentinel query as a comma-separated list.
- metric = Grafana names that you choose for those value fields (e.g. so you can prefix them with "sentinel_" to distinguish easily from other data sources).
- key = the key fields for the Traffic Sentinel query.
- label = Grafana label names that you choose for those key fields.
- scale = optional scale factors for the values.
- n = optional topN truncation. This typically only makes sense when there is just one value. It allows you to make queries that would otherwise return huge datasets and cause scalability problems in Prometheus.
- minval = optional minimum value. The script will only export data to Prometheus that exceeds this value.
- other = optional substitution string when a key result is otherwise null.
| Related Topics |
|