Security Stats API Guide

Overview

This document describes using the Instart security statistics API to make specific queries against the raw security event data and aggregated security statistics collected by our log processing system and data platform. You can access raw security event data, data on trends, Bot Defense, custom security rules, and Web Application Firewall (WAF) events.

Note

The WAF API has been deprecated, replaced by the WAF methods in the Security stats API described in this document.

You can find some query examples at the end of this document.

Events

These methods provide access to raw security event data.

Note

The retention period of the security event log is the previous three days plus whatever time has elapsed in the current day.

GET /stats/security/events/data

This provides raw event log data. You specify a time span and a list of the dimensions you want. You can optionally filter out some of the data you receive by either limiting the number of results shown or by adding an optional filter parameter to the query string.

The query structure is appended to the API URL as a standard query string, as follows:

https://api.instartlogic.com/ unique customer name>/v1/stats/security/events/data?
dimensions=<dimension1,dimension2...>
&startime=<UTC date/time of the desired beginning time>
&endtime=<UTC date/time of the desired end time>
[&filters=<filter expression 1,...>]
[&sortby=<dimension1,dimension2...>]
[&limit=<number of results desired>]

To be valid, each query requires

  • a set of dimensions
  • both start and end date/times for the span of time you want statistics for

Date/times are in standard ISO8601 date/time format, always in the UTC time zone. Don't specify a time zone designator or the request will be rejected. End timestamps must be later than the start timestamp to be valid.

The additional query parameters filters, sortby, and limit may be provided to further refine your query, but are optional.

All query parameters must be URL-encoded.

Dimensions

To get event data, provide the dimensions parameter with a valid list of dimensions. The list of supported dimensions can be found by querying the API, as described below.

The dimensions are the kinds of available data. They correspond to the column headings of a table listing event log records. For example, event_severity and event_type are two types of data that are recorded in the security events logs.

Starttime

The starttime parameter is the start time of the desired time range of the desired statistics, which corresponds to when the user requests were received as logged in the access logs. It is required and cannot be left empty. It must be specified in standard ISO8601 date/time format, but always in the UTC time zone. Don't specify a time zone designator or the request will be rejected. Start times must be earlier than the end time to be valid.

Endtime

The endtime parameter is the end time of the desired time range of the statistics, which corresponds to when the user requests were received as logged in the access logs. It is required and cannot be left empty. It must be specified in standard ISO8601 date/time format, but always in the UTC time zone (specifying another time zone designator will cause the request to be rejected). End times must be later than the start time to be valid.

Filters (optional)

The optional filters parameter consists of a comma-separated list of filters to be applied to the requested data. A filter is of the form:

<dimension><operator><value>

For example:

filters=property==public

where property is the dimension, == is the operator (equal to) and public is the value. This filter, if added to a query, would get back only the events that that occurred for the property named "public."

Valid operators are == and !=

Multiple individual filters can be combined using the OR(,) or AND(;) operators – but not both – to form a logical filter expression.

Note

If the special characters "," or ";" need to be treated as literal characters in a filter expression (for example, the comma character occurs in some country strings you might want to filter on), you need to escape them by prepending them with "\"

For example:

filters=property==public;client_browser==Chrome

contains two filters: the first one to allow only results from the property public, and a second one to allow only results that came from requests from the Chrome browser. They are separated by a semicolon, which means AND. So this pair of filters would return only results with an property named "public" AND were requests from Chrome.

Sortby (optional)

The optional sortby parameter contains a list of one or more dimensions and/or metrics over which the resultant statistics data should be sorted. For example:

sortby=event_timestamp_ms

Sorting order is specified by the left to right order of the metrics and dimensions listed. Sorting direction defaults to ascending and can be changed to descending by using a minus sign (-) prefix on the requested field.

One common use case would be if you are troubleshooting an issue that just started, and you want to see the latest events first. You can use

sortby=-event_timestamp_ms

to return results in reverse order.

Limit (optional)

The optional limit parameter is an integer that will restrict the total number of results that are returned by the query. For example, using

limit=25

tells the Stats API server to return the first 25 results.

If unspecified, the limit parameter defaults to returning 10 results.

The limit parameter has no upper limit. To get all events in a particular time span, you can query without the limit parameter, observe the total number of events, then query a second time with limit=<total number of events>.

HEAD /stats/security/events

The HEAD method is identical to GET except that the server doesn't return a message body in the response, just the same headers. This is useful for testing hypertext links for validity, accessibility, and recent modification.

GET /stats/security/events/dimensions

The dimensions are the kinds of available data. They correspond to the column headings of a table listing event log records. For example, event_severity and event_type are two types of data that are recorded in the security events logs.

WAF trends

These methods provide data on WAF (web application firewall) trends and analysis based on aggregated summary data.

GET /stats/security/waf/trends/data

This provides aggregated WAF trend data. You specify a time span and a list of the dimensions and metrics you want. You can optionally filter out some of the data you receive by either limiting the number of results shown or adding an optional filter parameter to the query string.

The query structure for each is appended to the API URL as a standard query string, as follows:

https://api.instartlogic.com/ unique customer name>/v1/stats/security/waf/trends/data?
dimensions=<dimension1,dimension2...>
&metrics=hits
&startime=<UTC date/time of the desired beginning time>
&endtime=<UTC date/time of the desired end time>
&granularity=<day|hour|5minute>
[&filters=<filter expression 1,...>]
[&sortby=<dimension1,dimension2...>]
[&limit=<number of results desired>]

where the parameters in square brackets are optional.

Dimensions

Dimensions are the kinds of available data. They correspond to the column headings of a table listing event log records.

The dimensions parameter contains a list of one or more elements which correspond to features that characterize the aggregated WAF trend statistics. For example, waf_action specifies the action taken when a request triggered a WAF rule.

The list of dimensions you specify must correspond to a valid grouping set. The list of supported grouping sets can be found by querying the API, as described below.

Metrics

Metrics are quantitative results. The only metric currently supported is hits.

Starttime

The starttime parameter is the start time of the desired time range of the desired statistics, which corresponds to when the user requests were received as logged in the access logs. It is required and cannot be left empty. It must be specified in standard ISO8601 date/time format, but always in the UTC time zone. Don't specify a time zone designator or the request will be rejected. Start times must be earlier than the end time to be valid.

Endtime

The endtime parameter is the end time of the desired time range of the statistics, which corresponds to when the user requests were received as logged in the access logs. It is required and cannot be left empty. It must be specified in standard ISO8601 date/time format, but always in the UTC time zone (specifying another time zone designator will cause the request to be rejected). End times must be later than the start time to be valid.

Granularity

The granularity parameter corresponds to the time granularity of the aggregated WAF trend data you want to retrieve. The possible values are

  • day
  • hour
  • 5minute

WAF trend data is aggregated as follows:

  • per 5 minutes for 1 week
  • per hour for 1 month
  • per day forever

When making a query you need to specify an appropriate granularity depending on the span of time you want the data for. For example, if you specify a span from last night at midnight to noon today, it doesn't make sense to specify a granularity of day. Similarly, if you specify a span that ended longer than three months ago, data is only available at the granularity of a day, so it doesn't make sense to specify it as hour or 5min. If you specify a span that ended longer than one week ago but less than three months ago, data is available at both day and hour granularities.

Filters (optional)

The optional filters parameter consists of a comma-separated list of filters to be applied to the requested data. A filter is of the form:

<dimension><operator><value>

For example:

filters=property==public

where property is the dimension, == is the operator (equal to) and "public" is the value. This filter, if added to a query, would get back only the events that that occurred for the property named "public."

Valid operators are == and !=

Multiple individual filters can be combined using the OR(,) or AND(;) operators – but not both – to form a logical filter expression.

Note

If the special characters "," or ";" need to be treated as literal characters in a filter expression (for example, the comma character occurs in some country strings you might want to filter on), you need to escape them by prepending them with "\".

For example:

filters=property==public;client_browser==Chrome

contains two filters: the first one to allow only results from the property public, and a second one to allow only results that came from requests from the Chrome browser. They are separated by a semicolon, which means AND. So this pair of filters would return only results with an property named "public" AND were requests from Chrome.

Sortby (optional)

The optional sortby parameter contains a list of one or more dimensions and/or metrics over which the resultant statistics data should be sorted. For example:

sortby=event_timestamp_ms

Sorting order is specified by the left to right order of the metrics and dimensions listed. Sorting direction defaults to ascending and can be changed to descending by using a minus sign (-) prefix on the requested field.

Limit (optional)

The optional limit parameter is an integer that will restrict the total number of results that are returned by the query. For example, using

limit=25

tells the Stats API server to return the first 25 results.

If unspecified, the limit parameter defaults to returning 10 results.

The limit parameter has no upper limit. To get all events in a particular time span, you can query without the limit parameter, observe the total number of events, then query a second time with limit=<total number of events>.

HEAD /stats/security/waf/trends

The HEAD method is identical to GET except that the server doesn't return a message body in the response, just the same headers. This is useful for testing hypertext links for validity, accessibility, and recent modification.

GET /stats/security/waf/trends/dimensions

This provides a list of the available dimensions of the WAF trend data. The dimensions are the kinds of available data. They correspond to the column headings of a table listing the WAF trend data. For example, waf_action and client_country_code are two types of data that are aggregated and stored in the data platform.

Only certain specific combinations of dimensions can be used in querying for security trend data. To see these, use the groupings method.

GET /stats/security/waf/trends/groupings

Use the groupings method to return the list of the valid dimension groupings of the aggregated WAF trend statistics. Only these combinations of dimensions can be used in querying for security trend data.

GET /stats/security/waf/trends/metrics

Use the metrics method to return the list of the valid metrics of the aggregated WAF trend statistics.

Metrics are quantitative measurements. Currently, the only metric available is hits.

WAF topk

This provides data on the top 20 (k=20) WAF events for a variety of data categories (each of which has its own dimensions and metrics) based on the aggregated WAF data. You specify a time span and a list of the dimensions and metrics you want. You can optionally filter out some of the data you receive by adding an optional filter parameter to the query string.

The query structure for each is appended to the API URL as a standard query string, as follows:

https://api.instartlogic.com/ unique customer name>/v1/stats/security/waf/topk/<category>/data?
dimensions=<dimension1,dimension2...>
&metrics=hits
&startime=<UTC date/time of the desired beginning time>
&endtime=<UTC date/time of the desired end time>
[&filters=<filter expression 1,...>]

where filters are optional. The possible values for category are:

  • browser
  • clientip
  • country
  • requestpath
  • ruleid
  • severity

The descriptions of the parameters are the same as for the WAF trend methods. Each category also has its corresponding dimensions, groupings, and metrics methods. 

HEAD /stats/security/waf/topk/<category>

There are corresponding HEAD methods for all the topK categories is identical to GET except that the server doesn't return a message body in the response, just the same headers. This is useful for testing hypertext links for validity, accessibility, and recent modification.

Trends

These methods provide data on overall security trends and analysis based on aggregated summary data.

GET /stats/security/request/trends/data

This provides security trends data. You specify a time span and a list of the dimensions and metrics you want. You can optionally filter out some of the data you receive by either limiting the number of results shown or adding an optional filter parameter to the query string.

The query structure is appended to the API URL as a standard query string, as follows:

https://api.instartlogic.com/ unique customer name>/v1/stats/security/request/trends/data?
dimensions=<dimension1,dimension2...>
&metrics=hits
&startime=<UTC date/time of the desired beginning time>
&endtime=<UTC date/time of the desired end time>
[&filters=<filter expression 1,...>]
[&sortby=<dimension1,dimension2...>]
[&limit=<number of results desired>]

where the parameters in square brackets are optional.

The descriptions of the parameters are the same as for the WAF trend methods.

GET /stats/security/request/trends/dimensions

Dimensions are the kinds of available data. They correspond to the column headings of a table listing event log records.

This provides a list of the available dimensions of the summary security trend data. For example, waf_action and domain are two of the dimensions of security trend data that are aggregated in the data platform.

GET /stats/security/request/trends/groupings

Use the groupings method to return the list of the valid dimension groupings of the summarized security trend statistics. Only these combinations of dimensions can be used in querying for security trend data.

GET /stats/security/request/trends/metrics

Use the metrics method to return the list of the valid metrics of the aggregated security trend statistics. Metrics are quantitative measurements. Currently, the only metric available is hits.

HEAD /stats/security/request/trends

The HEAD method is identical to GET except that the server doesn't return a message body in the response, just the same headers. This is useful for testing hypertext links for validity, accessibility, and recent modification.

Traffic shaping trends

These methods provide data on traffic shaping trends and analysis based on aggregated summary data. Traffic shaping is blocking and rate limiting of certain IP addresses/ranges and geo-location information. Rather than block or throttle, traffic shaping rules can also be set to warn, which means that no action os taken by the rule (the request is handled as an ordinary request) but the request is logged as a security event.

GET /stats/security/trafficshaping/trends/data

You specify a time span and a list of the dimensions and metrics you want. You can optionally filter out some of the data you receive by either limiting the number of results shown or adding an optional filter parameter to the query string.The query structure for each is appended to the API URL as a standard query string, as follows:

https://api.instartlogic.com/ unique customer name>/v1/stats/security/trafficshaping/trends/data?
dimensions=<dimension1,dimension2...>
&metrics=hits
&startime=<UTC date/time of the desired beginning time>
&endtime=<UTC date/time of the desired end time>
&granularity=<day|hour|5minute>
[&filters=<filter expression 1,...>]
[&sortby=<dimension1,dimension2...>]
[&limit=<number of results desired>]

where the parameters in square brackets are optional.

The descriptions of the parameters are the same as for the WAF trend methods.

GET /stats/security/trafficshaping/trends/dimensions

Dimensions are the kinds of available data. They correspond to the column headings of a table listing event log records.

Use the dimensions method to return a list of the available dimensions of the aggregated summary security trend data. For example, waf_action and domain are two types of security trend data that are aggregated in the data platform.

GET /stats/security/trafficshaping/trends/groupings

Use the groupings method to return the list of the valid dimension groupings of the aggregated security trend statistics. Only these combinations of dimensions can be used in querying for security trend data.

GET /stats/security/trafficshaping/trends/metrics

Use the metrics method to return the list of the valid metrics of the aggregated security trend statistics. Metrics are quantitative measurements. Currently, the only metric available is hits.

HEAD /stats/security/trafficshaping/trends

The HEAD method is identical to GET except that the server doesn't return a message body in the response, just the same headers. This is useful for testing hypertext links for validity, accessibility, and recent modification.

Traffic shaping topK

This provides data on the top 1,000 traffic shaping events for a variety of data categories (each of which has its own dimensions and metrics) based on the aggregated traffic shaping data. You specify a time span and a list of the dimensions and metrics you want. You can optionally filter out some of the data you receive by either limiting the number of results shown or adding an optional filter parameter to the query string.

The query structure for each is appended to the API URL as a standard query string, as follows:

https://api.instartlogic.com/ unique customer name>/v1/stats/security/trafficshaping/topk/<category>/data?
dimensions=<dimension1,dimension2...>
&metrics=hits
&startime=<UTC date/time of the desired beginning time>
&endtime=<UTC date/time of the desired end time>
[&filters=<filter expression 1,...>]
[&sortby=<dimension1,dimension2...>]
[&limit=<number of results desired>]

where the parameters in square brackets are optional. The possible values for category are listed below:

  • browser
  • clientip
  • country
  • ratelimitclientip
  • ratelimitcountry
  • requestpath

The descriptions of the parameters are the same as for the WAF trend methods. Each category method has its corresponding dimensions, groupings, and metrics methods.

HEAD /stats/security/trafficshaping/topk/<category>

There are corresponding HEAD methods for all the topK categories. These methods are identical to GET except that the server doesn't return a message body in the response, just the same headers. This is useful for testing hypertext links for validity, accessibility, and recent modification.

Examples

The following are some examples of using the security stats API to query for statistics. Line breaks are added for readability.

If you use cURL, execute the command

curl -v -u 'username:password' '<base URL>?<query string>'

If you use Postman or a similar REST client:

  1. Open a new Request window.

  2. In the URL field at the top enter the base URL and the query string.
  3. Select GET from the method list to the left of the URL field.
  4. Below, in the Authorization tab, select Basic Auth from the Type list, and enter your username and password in the fields provided.
  5. Click Send.

With either method you would of course use your own authorization string and company name in place of the placeholders. Also be sure that the starttime and endtimeparameters cover a time span that occurred within the last three days if you are querying raw event data from the access logs.

Example 1 – raw events query without optional parameters

The following query requests raw event data logged between 11 November 2017 at midnight (UTC) and 12 November 2017 at midnight (UTC). It is requesting a subset of dimensions: the remote IP address of the client connection, the client browser, the event type, event action, and the domain the request was sent to.

The base URL and query string for this query are

https://api.instartlogic.com/acme/v1/stats/security/events/data?
dimensions=client_connection_remote_ip,client_browser,event_type,event_action,domain
&starttime=2017-11-11T00:00:00
&endtime=2017-11-12T00:00:00

Note that line breaks were added to the query string for readability.

If successful, the API server responds with a JSON object similar to the following (for brevity only a few rows are shown of the full number actually returned). Note that the timestamp of each event (event_timestamp_ms) is included in the results by default. Also note that limit (how many results to display) is set at the default value of 10since that parameter was not specified in the request, and that the total number of events found in the specified time span is displayed as the value of the total_results field:

{
"query": {
"starttime": "2017-11-11T00:00:00",
"endtime": "2017-11-12T00:00:00",
"start": 0,
"limit": 10,
"dimensions": [
"client_connection_remote_ip",
"client_browser",
"event_type",
"event_action",
"domain"
],
"filters": "customer==acme"
},
"columns": [
{
"name": "event_timestamp_ms"
},
{
"name": "client_connection_remote_ip"
},
{
"name": "client_browser"
},
{
"name": "event_type"
},
{
"name": "event_action"
},
{
"name": "domain"
}
],
"rows": [
[
"2017-11-11T00:02:20",
"45.33.61.221",
"Chrome",
"CUSTOM",
"WARN",
"www.acme.com"
],
[
"2017-11-11T00:02:40",
"139.162.210.253",
"Chrome",
"CUSTOM",
"WARN",
"www.acme.com"
],
[
"2017-11-11T00:02:33",
"45.79.179.69",
"Chrome",
"CUSTOM",
"WARN",
"www.acme.com"
],
...
[
"2017-11-11T00:02:07",
"139.162.244.180",
"Chrome",
"CUSTOM",
"WARN",
"www.acme.com"
]
],
"total_results": 26584
}

Example 2 – raw events query with filters and limit

The following query requests the client IP address, ASN, and rule ID from raw event data logged between 11 November 2017 at midnight (UTC) and 12 November 2017 at midnight (UTC). A filter is applied so that only those log entries from Singapore (country code SG) are returned, and a limit of 50 is applied.

The base URL and query string for this query are

https://api.instartlogic.com/acme/v1/stats/security/events/data?
dimensions=event_timestamp_ms,client_ip,rule_id,client_asn
&starttime=2017-11-12T00:00:00
&endtime=2017-11-14T24:00:00
&limit=50
&filters=client_country_code==SG

Note that line breaks were added to the query string for readability.

If successful, the API server responds with a JSON object similar to the following (for brevity only a few rows are shown of the full number actually returned):


{
"query": {
"starttime": "2017-11-12T00:00:00",
"endtime": "2017-11-14T24:00:00",
"start": 0,
"limit": 50,
"dimensions": [
"event_timestamp_ms",
"client_ip",
"rule_id",
"client_asn"
],
"filters": "client_country_code==SG;customer==acme"
},
"columns": [
{
"name": "event_timestamp_ms"
},
{
"name": "client_ip"
},
{
"name": "rule_id"
},
{
"name": "client_asn"
}
],
"rows": [
[
"2017-11-12T00:02:46",
"52.76.85.66",
"D173128",
16509
],
[
"2017-11-12T00:10:32",
"52.76.85.66",
"D173128",
16509
],
[
"2017-11-12T00:06:33",
"52.76.85.66",
"D173128",
16509
],
...
[
"2017-11-12T01:37:18",
"52.76.85.66",
"D173128",
16509
],
[
"2017-11-12T01:33:19",
"52.76.85.66",
"D173128",
16509
]
],
"total_results": 4365
}