Scripting Queries

Traffic Sentinel's scripted query capability provides a powerful mechanism for creating specialized queries. Most users will not need to use the scripting interface; the standard reporting capabilities can generate generate most common types of report. Scripting is typically used to extract and format data so that it can be imported into other tools (such as a billing application), or to develop new types of report that combine data from multiple sources into a single chart or table.

Running a query

The scripting interface is accesses from the Reports>Script menu. The following figure shows the script form with the default query script.

Enter a script or paste a script into the text input area of the form and click on the Submit button to evaluate the script. Click on the Defaults button to return to the default script. The default script makes a simple topN query. You can edit the view, where, interval, sort, n, format and heading values to experiment with the query and see the results.

The following figure shows a typical query result:

Click on the Back button to return to the script form.

Importing Data

When viewing a query result; click on the TXT button to get a simple text result with the query encoded in the URL. The URL can by copied and used to perform the query at any time. Web aware tools such as wget or Perl can use the query URL to extract data.

Note A script may be too long to be passed in the URL. In this case you will need to get the data using an HTTP POST operation (both wget and Perl support POST). You can see both the URL and the arguments needed for a POST on lines 3 and 4 of the IQY format (see below).

The IQY button shows the query in Microsoft Excel IQY format. Cut and paste the text text and use Notepad put it in a file with an .IQY extension and Microsoft Excel will be able to use it to perform Web Queries.

Note: Excel prefers to have the data as an HTML table rather than in CSV format. Just change the format to "html" in the default query script to get the result as an HTML table.

Writing Scripts

The scripting language used by Traffic Sentinel is JavaScript 1.6 with E4X extensions. Javascript is a simple scripting language (much like Visual Basic) that is commonly used within web pages. There are many free tutorials on Javascript (just search for "javascript tutorial" on Google). If you are looking for a book then Javascript: The Definitive Guide is an excellent reference.

The rest of this tutorial will provide examples that demonstrate how scripts can access network topology and traffic information, perform calculations and output results so that they can easily be imported into other tools.

Note You can cut and paste these example scripts into the Script form to try them out yourself. If you don't have access to a copy of Traffic Sentinel, you can run these scripts on the demo.inmon.com.

Printing

Print functions are not a standard part of the Javascript language. The following two global functions have been added to allow textual output to be returned from the script: print() prints a result and println() prints the result and starts a new line.

print("hello");
println(" world");
println("one plus one equals " + (1 + 1));

Print functions are very useful for testing scripts. Include print statements to make sure that you are getting the results you expect.

Network

The Network class is an interface to the current network state; providing information on agents and interfaces, network topology and address mappings.

The following example creates a new network object, asks for a list of all the agents and then iterates over the agents printing the agent ip address and system description.

var n = Network.current();
var agents = n.agents();
for(var i = 0; i < agents.length; i++) {
  n.path = agents[i];
  println("ip=" + n.agentIP() + " sysDescr=" + n.sysDescr());
};

The path attribute refers to the hierarchical grouping of network devices specified in the configuration (see the Traffic Sentinel Configuration tutorial for more information on the network hierarchy). Setting a path limits the scope of subsequent queries; in the previous example setting the path to an agent allows agent specific data to be queried.

The network object also has a number of Map functions that take an array of arguments and returns an array of results. In the following example uses the vendorMap() function to find the vendor names associated with an array of MAC addresses.

var n = Network.current();
var macs = ["000480F54600","001321B21CE1","0002556F2F78"];
var vendors = n.vendorMap(macs);
for(var i = 0; i < macs.length; i++) println(macs[i] + " " + vendors[i]);

Query

The Query class is used to query the traffic database for interface counter data and traffic flow information. The traffic database is organized into views that have a particular set of attributes:

historytrmx this view is used to query historical traffic flows (e.g. top web servers over the last month).
historycounters this view is used to query historical interface counter trends (e.g. trend utilization on a link over the last month).
rttraffic this view is used to query short term traffic flows, before they have been consolidated into the history (e.g. top broadcast sources in last 10 minutes).
rtcounters this view is used to query short term interface counter trends (e.g. trend errors on a link over the last 20 minutes).
host this view is used to query host performance counters (e.g. cpu, memory, IO).
application this view is used to query application transaction information (URL's, response times, etc.).
events this view is used to query the event log (e.g. identify top types of critical event).

Note Click on the links in this list to get details of all the attributes of each view.

Within a view, attributes fall into two major types: keys that will form categories when aggregated and values that will be accumulated. For example ipsource is a category and bytes is a value. A query that specifies the ipsource attribute and the bytes attribute will produce a result that sums the bytes for each source ip address.

To create a query you will need to specify a view, select attributes, choose a time interval and specify how the results are to be sorted. The following example constructs a simple query.

var q = new Query();
q.view     = "historytrmx";
q.select   = "ipsource,bytes";
q.where    = "sourcezone != EXTERNAL";
q.interval = "today";
q.sort     = "bytes";
q.truncate = 5;
var t = q.run();
t.printCSV(true);

This script makes a query for the top 5 local IP source addresses, sorted by bytes. The q.run() statement evaluates the query and returns a Table as a result.

A simpler way of constructing the same query is to use the Query.topN() function; the following script performs exactly the same query as before

var q = Query.topN("historytrmx",
                   "ipsource,bytes",
                   "sourcezone != EXTERNAL",
                   "today",
                   "bytes",
                   5);
var t = q.run();
t.printCSV(true);

In the example the data was sorted by one of the value attributes in the view. Sometimes it is useful to instead count the number of unique values of one of the key attributes and use that as a value. This type of query is particularly useful for identifying scanning behavior (often associated with Internet worms or port scanning). The following example demonstrated how a count() function can by applied in a query

var q = Query.topN("historytrmx",
                   "ipsource,count(ipdestination)",
                   "destinationport = TCP:80 & sourcezone != EXTERNAL",
                   "today",
                   "count(ipdestination)",
                   5);
var t = q.run();
t.printCSV(true);

The query counts the number of addresses scanned by hosts on the local network using port TCP:80 and lists the top hosts by number of addresses scanned.

Another common type of query is to generate a trend showing changes over time. The following script makes use of the Query.trend() function to create the query.

var q = Query.trend("historytrmx",
                   "time,bytes",
                   "protocol = TCP",
                   "yesterday",
                   "hour");
var t = q.run();
t.printCSV(true);

The script trends total bytes by hour for TCP traffic. If byte rates are required instead of totals (i.e. average bytes per second for each interval) then the expression rate(bytes) can be substituted for bytes as shown below

var q = Query.trend("historytrmx",
                   "time,rate(bytes)",
                   "protocol = TCP",
                   "yesterday",
                   "hour");
var t = q.run();
t.printCSV(true);

Table

The Table class is used to represent a query result. A table consists of information about the table columns (including column names and types) and a number of rows of data.

In the previous example we made a query that resulted in values measured in bytes per second. If we want the results in bits per second we need to scale them by a factor of 8. The Table scaleColumn() function can peform this task

var q = Query.trend("historytrmx",
                   "time,rate(bytes)",
                   "protocol = TCP",
                   "yesterday",
                   "hour");
var t = q.run();
t.scaleColumn(1,8);
t.printCSV(true);

The following example demonstrates how to extend a Table with additional data:

var q = Query.topN("historytrmx",
                   "ipsource,bytes",
                   "sourcezone != EXTERNAL",
                   "today",
                   "bytes",
                   5);
var t = q.run();
var n = Network.current();
var addresses = t.column(0);
var locations = n.locationMap(addresses);
t.insertColumn("location","interface",locations,1);
t.printCSV(true);

The script accesses the table column containing the addresses (column 0) and uses the Network locationMap() function to locate each address to the switch interface connecting it to the network. The insertColumn function is used to insert the location information immediately after the addresses in the table (naming the column location and declaring its type as interface).

SNMP

The SNMP class is used to make SNMP (Simple Network Management Protocol) requests to devices in the network. Many standard SNMP variables are already available using the Network class (e.g. sysName(), ifName() etc). It is much more efficient to retrieve information from the Network class where possible. Generally the SNMP class is used to retrieve data from devices that Traffice Sentinel doesn't routinely monitor, or to retrieve vendor specific information.

The following script shows how a vendor specific variable can be retrieved; in this case, temperature from Foundry Networks switches:

var selectedAgents = [];
var n = Network.current();
for each (var agent in n.agents()) {
  n.path = agent;
  var id = n.sysObjectID();
  if(id && id.indexOf(".1.3.6.1.4.1.1991.") == 0) selectedAgents.push(agent);
}

var s = new SNMP();
s.oid = ".1.3.6.1.4.1.1991.1.1.1.1.18.0";
var t = Table.create(["Agent","Name","Degrees C"],
                     ["agent","string","integer"]);
for each(var agent in selectedAgents) {
  n.path = agent;
  s.host = agent;
  try {
    var val = s.get();
    if(val[0]) t.addRow([agent, n.sysName(), parseInt(val[0]) / 2]);
  } catch(ex) {};
}
t.printCSV(true);

The first part of the script uses a Network object to enumerate all the agents. Foundry Networks agents are identified by checking each agent's sysObjectID to see if it starts with the Foundry Networks prefix, ".1.3.6.1.4.1.1991.". The second part of the script sends a request for the current chassis temperature to each of the selected agents and assembles the result in a table.

readurl

The readurl function is used to import data into a script. The following example shows how readurl can be used to access an online information source:

var addr = "64.151.76.40";
var url = "http://www.radb.net/cgi-bin/radb/advanced-query.cgi"
url +=  "?submit=Query&searchstr=" + addr;
var res = readurl(url);
var match = /descr:[\s]+([^\n]+)/.exec(res);
println(match[1]);

The script constructs a URL that queries the Merit Routing Asset Database (RADb) for information about an IP address. The resulting text is parsed using a regular expression and the value of the descr: field is printed.

This next example imports an XML file and uses it to lookup information:

var url = "http://www.inmon.com/tutorials3/db.xml";
var db = new XML(readurl(url));
println(db.server.(name=="Web").address);

The db.xml document contains records for a number of servers, each with a name and address. The E4X extensions to javascript provide native support for XML and provide mechanisms for querying, creating, modifying and printing XML documents.

Finally, a readurl can be used to load a library of javascript functions and values at the beginning of a script:

var initscript = readurl("http://www.inmon.com/tutorials3/init.js");
eval(initscript);
println(descr("64.151.76.40"));

The init.js script contains a descr() function that implements the RADb lookup function shown earlier.

Using Scripts

There are three main ways to use scripts: web-based queries using the Query URL, the query command or by incorporating them in xml templates for use by the Traffic Sentinel reporting tool.

URL

The Query URL has the following format:

/inmsf/Query?script=script&authenticate=basic&resultFormat=txt

where script is the javascript source.

It is also possible to pass additional arguments to the script via the URL. Any attribute of the form

input_name=value

in the URL is passed into the script as an assignment of the form

var name = value;

The following script demonstrates this technique:

var addrs = [];
if(typeof(address) != 'undefined') addrs = address.split(",");
var n = Network.current();
var locations = n.locationMap(addrs);
for(var i = 0; i < addrs.length; i++) {
  n.path = locations[i];
  print(addrs[i]+ " " + n.sysName() + " " + n.ifName()); 
};

The script tests to see if the address parameter is defined (i.e. if it has been passed into the script). If the address string is defined then it is treated as a comma separated list and split into an array of addresses. The locationMap function identifies the interface connecting each address to the network and the resulting agent and interface information is printed.

The following form shows how this script can be embedded into a web form; click on the button to see the result.

The source code for the form is shown below:

<form action="http://demo.inmon.com/inmsf/Query" method="get">
<input name="input_address" value="172.16.74.28" type="text" />
<input type="submit" />
<input type="hidden" name="authenticate" value="basic" />
<input type="hidden" name="resultFormat" value="txt" />
<input type="hidden" name="script" value="var addrs = [];
if(typeof(address) != 'undefined') addrs = address.split(',');
var n = Network.current();
var locations = n.locationMap(addrs);
for(var i = 0; i &lt; addrs.length; i++) {
  n.path = locations[i];
  print(addrs[i]+ ' ' + n.sysName() + ' ' + n.ifName());
};" />
</form>

Note: Single quotes were used to delimit strings in the script so that the script could be included as a form field. The "<" symbol had to be escaped as "<" (generally the characters "<", ">" and "&" need to be escaped as "<", ">" and "&" respectively when a script is used as an html attribute - as it is in this case).

Command Line

A command line query function is part of the Traffic Sentinel standard installation. The command can be found at /usr/local/inmsf/bin/query. The query command can take a script as standard input:

echo "println('hello world');" | /usr/local/inmsf/bin/query

or from a script file:

/usr/local/inmsf/bin/query hello.js

The command line query function provides a useful way to access traffic data using standard Linux tools. The command line query can be more reliable than a URL based query since it bypasses the web server and the network. One of the most common applications for the command line query is to periodically extract traffic accounting data for billing purposes. The Linux cron service can run a script periodically that uses a query to obtain usage data that is used to update billing records in the billing system.

XML Templates

The tutorial Creating Report Templates describes how you can create a report template based on a query script. The script can be extended to create customized charts and tables in a report. Once report template has been created it can be run on demand or scheduled to produce reports and generate events.

The tutorial Creating Widget Templates describes how you can create a widget template based on a query script. The script can be extended to display a chart or a table in a widget. Once a widget template has been created it can be included on the Dashboard.

Support