Integration with a Customer Portal

Traffic Sentinel's scripted query capability provides a flexible mechanism for extracting traffic information that can be presented to network users. In many cases there may be an existing "customer portal" that provides access to information on network outages, scheduled maintenance and a mechanism for raising trouble tickets. This tutorial describes techniques for making traffic information accessible to customers in a secure and scalable way. This tutorial assumes that you are familiar with the script API (see the Scripting Queries and Advanced Scripting tutorials). Some of the examples in this tutorial require Traffic Sentinel version 4.0 or later.

Note: Traffic Sentinel's web interface is designed for network managers and operators. Traffic Sentinel's abililty to provide a detailed, network-wide view of traffic is very useful for network operations but acces to this sensitive information should be limited. In order to provide information to customers, you need to ensure that there is no leakage of confidential information between customers, or leakage of information about the internal configuration of the network that would compromise security.

There are two approaches to providing data to customers:

Standard Reports, create a standard set of charts and tables that describes each customer's traffic.
Ad-Hoc Queries, provide a way for customers to make queries to selected parts of the database.

The rest of this tutorial will describe these two techniques in more detail.

Standard Reports

The first task is to find a way to identify network and/or traffic objects that are associated with each customer. This association will tend to be network dependent:

switch port, in many networks customers are associated with the interface(s) on the edge switches that connect their servers to the network. This is often the case with colocation factilities.
mac address, the mac address(es) associated with customer routers may identify customers. This can be the case with a layer 2 exchange.
CIDR, customers might be allocated particular IP addresses or address blocks.

This customer to network object mapping must be made accessible to the script. Options include:

ifAlias,sysName, encode customer information in the names of devices or interfaces so that it is available using SNMP.
file, dump the information as a file in a location accessible from the Traffic Sentinel server.
url, make the information available through a web server. The information could be created dynamically through a php or CGI script, or it could simply be a way of exporting the file.
script, you can invoke the query using a scripting langauge such as Perl or PHP and pass the information in as part of the query.

Most reporting scripts will have a similar structure, following the same basic steps:

Retrieve customer data
Create a filter to select only the data relevant to the customers
Run a query to retrieve the data
Split the resulting data into individual customer tables
Create charts for each customer and write them to disk

Example: Interface Trends

In this first example we will assume that each customer facing switch port has an SNMP ifAlias of the form "C-id" where id is the customer number, for example an ifAlias of "C-1223" identifies customer number 1223.

Suppose that you want to create charts for each customer showing usage trends for each of their interfaces. The following script (trends.txt) creates the charts:

// regular expression to identify customer interfaces
var ifAlias_pattern = /^C-([\\\d]+)$/;

// query parameters
var interval = "last24hours";
var group    = 5;

// customer portal location
var rootdir = "/var/www/html/customers/";
var height  = 200;
var width   = 400;
var prefix  = "l24h";

// limit number of interfaces per query
var max_ifs_per_query = 400;

var network = Network.current();

// create list of customer interfaces
var ifx_to_cust = {};
for each (var ifx in network.interfaces()) {
  network.path = ifx;
  var match = ifAlias_pattern.exec(network.ifAlias());
  if(match) ifx_to_cust[ifx] = match[1];
}

// create all the charts, max_ifs_per_query at a time
var ifxs = [];
for (var ifx in ifx_to_cust) {
  ifxs.push(ifx);
  if(ifxs.length >= max_ifs_per_query) {
    createCharts(ifxs);
    ifxs = [];
  }
}
if(ifxs.length > 0) createCharts(ifxs);

// function to create charts for a list of interfaces
function createCharts(ifxs) {
  // query for trends for list of interfaces
  var query = Query.trend(
    "historycounters",
    "time,interface,rate(ifinoctets),rate(ifoutoctets)",
    "interface=" + ifxs,
    interval,
    group);

  var table = query.run();

  // split results by interface
  var trends = [];
  var pvt = table.pivotTime(0,1,2);
  for(var j = 1; j < pvt.ncols; j++) {
    var t = new Table();
    t.start = table.start;
    t.end = table.end;
    t.addColumn("Time","time",pvt.column(0));
    t.addColumn("In","double",pvt.column(j));
    t.scaleColumn(1,8);
    trends[pvt.cnames[j]] = t;
  }
  pvt = table.pivotTime(0,1,3);
  for(var j = 1; j < pvt.ncols; j++) {
    var t = trends[pvt.cnames[j]];
    t.addColumn("Out","double",pvt.column(j));
  }

  // create charts for each interface
  for (var ifx in ifx_to_cust) {
    var customer = ifx_to_cust[ifx];
    network.path = ifx;
    var t = trends[ifx];
    if(t) {
      var chart = Chart.multiSeries(
           "trend", network.agent() + ">" + network.displayName(),
           t, "Time", 0, "Bits per Second", [1,2]);
      chart.height = height;
      chart.width = width;
      var fn = prefix + "-" 
               + network.agentIP() + "-" + network.ifx() 
               + "_" + height + "-" + width + ".png";
      write(
        rootdir + customer + "/"
        + fn,
        chart);
    }
  }
}

Example: Top Users

In this second example, assume that instead of being associated with switch ports, each customer is associated with one or more IP address blocks and that the information is available as an xml document (customers.xml):

<customers>
  <customer name="InMon Corp." id="1001">
    <cidr address="10.1.1.0" bits="24" />
    <cidr address="10.1.2.0" bits="24" />
  </customer>
  <customer name="NewCo Inc." id="1002">
    <cidr address="10.1.4.0" bits="25" />
  </customer>
</customers>

Suppose that you want to create a chart for each customer showing the top consumers of bandwidth within their address range(s). The following script (users.txt) creates the charts:

// location of customer database
var url = "http://www.inmon.com/tutorials4/customers.xml";

// query parameters
var interval  = "last24hours";
var n         =  5;

// customer portal location
var rootdir = "/var/www/html/customers/";
var height  = 300;
var width   = 400;
var prefix  = "l24h";

// limit number of customers per query
var max_cust_per_query = 100;

// get customer data
var customersString = readurl(url);
var customers = new XML(customersString);

var cust_to_cidrs = {};
for each (var customer in customers..customer) {  
  for each (var cidr in customer..cidr) {
    var cidrs = cust_to_cidrs[customer.@id];
    if(!cidrs) {
      cidrs = [];
      cust_to_cidrs[customer.@id] = cidrs;
    }
    cidrs.push(cidr.@address + "/" + cidr.@bits);
  }
}

// create all the charts, max_cust_per_query at a time
var custs = [];
for (var cust in cust_to_cidrs) {
  custs.push(cust);
  if(custs.length >= max_cust_per_query) {
    createCharts(custs);
    custs = [];
  }
}
if(custs.length > 0) createCharts(custs);

// function to create charts for a list of customers
function createCharts(custs) {
  // create a multi-query
  var selects  = [];
  var filters  = [];
  var q = Query.topN("historytrmx",
                     selects,
                     filters,
                     interval,
                     "bytes",
                     n);
  q.multiquery = true;
  for each (var cust in custs) {
    selects.push("ipsource,bytes");
    filters.push("ipsource=" + cust_to_cidrs[cust]);
    selects.push("ipdestination,bytes");
    filters.push("ipdestination=" + cust_to_cidrs[cust]);
  }

  // get result tables
  var tables = q.run();

  // create charts
  var i = 0;
  for each (var cust in custs) {
    var chart = Chart.singleSeries(
                    "bar",
                    "Top Sources",tables[i++],
                    "Time", 0, "Bytes", 1);
    chart.height = height;
    chart.width  = width;
    var fn = prefix + "-" + "sources" + "_" + height + "-" + width + ".png";
    write(rootdir + cust + "/" + fn, chart);

    chart = Chart.singleSeries(
                    "bar",
                    "Top Destinations",tables[i++],
                    "Time", 0, "Bytes", 1);
    chart.height = height;
    chart.width  = width;
    fn = prefix + "-" + "destinations" + "_" + height + "-" + width + ".png";
    write(rootdir + cust + "/" + fn, chart);
  }
}

The script makes extensive use of the E4X extensions to javascript. The script uses E4X mechanisms for iterating over the XML document and accessing elements within the document.

Scheduling

The following steps use the crontab command to automatically run this script every day at 2:00am to update the customer charts. Type:

crontab -e

and add the following line to your list of tasks:

0 2 * * * /usr/local/inmsf/bin/query /home/pp/scripts/trends.txt
0 2 * * * /usr/local/inmsf/bin/query /home/pp/scripts/users.txt

You may want to construct multiple scripts, each of which updates a different type of chart. A script run every 10 minutes might update trend charts showing data over the last hour. A script run every month could provide charts that display top contributors to usage charges.

At this point we have a family of charts being maintained automatically. Each customer's charts are stored in a directory corresponding to their customer ID. The next step is to decide how to make this data available to the customer portal.

There are a number of options for exporting the data:

NFS you can use an NFS mount to either write the data directly onto the customer portal server, or to allow the customer portal server to access the customer data. This technique is very convenient if Traffic Sentinel server and the customer portal server are near each other on a secure network.
rsync you can use the rsync utility to copy updated information to the customer portal. Rsync can be used in conjunction with ssh to provide a secure mechanism for transferring the data between the servers.
HTTP you can configure Apache on the Traffic Sentinel server to export the data so that it is available via HTTP (or HTTPS if security is a concern). In a simple case where there is no existing customer portal, you could construct a simple web interface that allows users to see their traffic data through pages published on the Traffic Sentinel server.

In the next section we will look at techniques for allowing customers to make queries that access their own traffic data.

Ad-Hoc Queries

Ad-Hoc queries are useful for providing information that is needed immediately or information that is only occasionally accessed.

Most query scripts will have a similar structure, following the same basic steps:

Authenticate the customer
Extract query parameters
Construct a filter that restricts customer to their own data
Append additional customer supplied filters
Run query
Format the results

Example: Connection List

In this example, suppose you want to allow customers to make a query that would list all the hosts that connected to a particular IP address during a specified period. This type of query provides a useful audit trail to the customer in the case where they suspect a host might have been compromised.

Assume that each customer is associated with one or more IP address blocks and that the information is available as an xml document (customers.xml):

<customers>
  <customer name="InMon Corp." id="1001">
    <cidr address="10.1.1.0" bits="24" />
    <cidr address="10.1.2.0" bits="24" />
  </customer>
  <customer name="NewCo Inc." id="1002">
    <cidr address="10.1.4.0" bits="25" />
  </customer>
</customers>

The following script (connections.txt) allows a customer to query their data:

// the following variables must be passed in as arguments
//  - customerid
//  - target

var url = "http://www.inmon.com/tutorials4/customers.xml";

// get customer data
var customersString = readurl(url);
var customers = new XML(customersString);

// get cidrs for customerid
var cidrs = [];
for each(var cidr in customers.customer.(@id == customerid)..cidr) {
   cidrs.push(cidr.@address + "/" + cidr.@bits);
}
if(cidrs.length > 0) {
  var where = "(ipsource=" + cidrs + "|" + "ipdestination=" + cidrs + ")";
  where += "&(ipsource=" + target + "|" + "ipdestination=" + target + ")";
  var query = Query.topN(
    "historytrmx",
    "sourceaddress,sourceport,destinationaddress,destinationport,bytes",
     where, "today", "bytes", 100);
  var t = query.run();
  t.printHTML(true);
}

Finally, the script needs to be integrated with the customer portal. Assuming that the portal is written in php, here is an example script:

<?php
$customerid = '1001';
$target = $_GET['target']; 
if($target) {
 exec('/usr/local/inmsf/bin/query\
 -input customerid='.$customerid.'\
 -input target='.escapeshellarg($target).'\
 /home/pp/scripts/connections.txt',$result);
 foreach($result as $line) echo($line);  
}
else {
 echo '<form method="GET" action="'.$PHP_SELF.'">';
 echo '<input type="text" size="12" name="target">';
 echo '<button type="submit">OK</button>';
 echo '</form>';
}
?>

The customer needs be authenticated before they have access to the query form. The variable $customerid must be set to the authenticated customer's id.