base/frameworks/sumstats/main.bro

SumStats

The summary statistics framework provides a way to summarize large streams of data into simple reduced measurements.

Namespace:SumStats
Source File:/scripts/base/frameworks/sumstats/main.bro

Summary

Types

SumStats::Calculation: enum The various calculations are all defined as plugins.
SumStats::Key: record Represents a thing which is having summarization results collected for it.
SumStats::Observation: record Represents data being added for a single observation.
SumStats::Reducer: record  
SumStats::Result: table Type to store results for multiple reducers.
SumStats::ResultTable: table Type to store a table of sumstats results indexed by keys.
SumStats::ResultVal: record Value calculated for an observation stream fed into a reducer.
SumStats::SumStat: record SumStats represent an aggregation of reducers along with mechanisms to handle various situations like the epoch ending or thresholds being crossed.

Redefinitions

SumStats::Reducer: record  

Functions

SumStats::create: function Create a summary statistic.
SumStats::key2str: function Helper function to represent a SumStats::Key value as a simple string.
SumStats::observe: function Add data into an observation stream.
SumStats::request_key: function Dynamically request a sumstat key.

Detailed Interface

Types

SumStats::Calculation
Type:

enum

SumStats::PLACEHOLDER
SumStats::AVERAGE

(present if base/frameworks/sumstats/plugins/average.bro is loaded)

Calculate the average of the values.

SumStats::HLL_UNIQUE

(present if base/frameworks/sumstats/plugins/hll_unique.bro is loaded)

Calculate the number of unique values.

SumStats::LAST

(present if base/frameworks/sumstats/plugins/last.bro is loaded)

Keep last X observations in a queue.

SumStats::MAX

(present if base/frameworks/sumstats/plugins/max.bro is loaded)

Find the maximum value.

SumStats::MIN

(present if base/frameworks/sumstats/plugins/min.bro is loaded)

Find the minimum value.

SumStats::SAMPLE

(present if base/frameworks/sumstats/plugins/sample.bro is loaded)

Get uniquely distributed random samples from the observation stream.

SumStats::VARIANCE

(present if base/frameworks/sumstats/plugins/variance.bro is loaded)

Find the variance of the values.

SumStats::STD_DEV

(present if base/frameworks/sumstats/plugins/std-dev.bro is loaded)

Find the standard deviation of the values.

SumStats::SUM

(present if base/frameworks/sumstats/plugins/sum.bro is loaded)

Sums the values given. For string values, this will be the number of strings given.

SumStats::TOPK

(present if base/frameworks/sumstats/plugins/topk.bro is loaded)

SumStats::UNIQUE

(present if base/frameworks/sumstats/plugins/unique.bro is loaded)

Calculate the number of unique values.

The various calculations are all defined as plugins.

SumStats::Key
Type:

record

str: string &optional

A non-address related summarization or a sub-key for an address based summarization. An example might be successful SSH connections by client IP address where the client string would be the key value. Another example might be number of HTTP requests to a particular value in a Host header. This is an example of a non-host based metric since multiple IP addresses could respond for the same Host header value.

host: addr &optional

Host is the value to which this metric applies.

Represents a thing which is having summarization results collected for it.

SumStats::Observation
Type:

record

num: count &optional

Count value.

dbl: double &optional

Double value.

str: string &optional

String value.

Represents data being added for a single observation. Only supply a single field at a time!

SumStats::Reducer
Type:

record

stream: string

Observation stream identifier for the reducer to attach to.

apply: set [SumStats::Calculation]

The calculations to perform on the data points.

pred: function (key: SumStats::Key, obs: SumStats::Observation) : bool &optional

A predicate so that you can decide per key if you would like to accept the data being inserted.

normalize_key: function (key: SumStats::Key) : SumStats::Key &optional

A function to normalize the key. This can be used to aggregate or normalize the entire key.

ssname: string &optional

calc_funcs: vector &optional

hll_error_margin: double &default = 0.01 &optional

(present if base/frameworks/sumstats/plugins/hll_unique.bro is loaded)

The error margin for HLL.

hll_confidence: double &default = 0.95 &optional

(present if base/frameworks/sumstats/plugins/hll_unique.bro is loaded)

The confidence for HLL.

num_last_elements: count &default = 0 &optional

(present if base/frameworks/sumstats/plugins/last.bro is loaded)

Number of elements to keep.

num_samples: count &default = 0 &optional

(present if base/frameworks/sumstats/plugins/sample.bro is loaded)

A number of sample Observations to collect.

topk_size: count &default = 500 &optional

(present if base/frameworks/sumstats/plugins/topk.bro is loaded)

Number of elements to keep in the top-k list.

unique_max: count &optional

(present if base/frameworks/sumstats/plugins/unique.bro is loaded)

Maximum number of unique elements to store.

SumStats::Result
Type:table [string] of SumStats::ResultVal

Type to store results for multiple reducers.

SumStats::ResultTable
Type:table [SumStats::Key] of SumStats::Result

Type to store a table of sumstats results indexed by keys.

SumStats::ResultVal
Type:

record

begin: time

The time when the first observation was added to this result value.

end: time

The time when the last observation was added to this result value.

num: count &default = 0 &optional

The number of observations received.

average: double &optional

(present if base/frameworks/sumstats/plugins/average.bro is loaded)

For numeric data, this calculates the average of all values.

hll_unique: count &default = 0 &optional

(present if base/frameworks/sumstats/plugins/hll_unique.bro is loaded)

If cardinality is being tracked, the number of unique items is tracked here.

card: opaque &optional

(present if base/frameworks/sumstats/plugins/hll_unique.bro is loaded)

hll_error_margin: double &optional

(present if base/frameworks/sumstats/plugins/hll_unique.bro is loaded)

hll_confidence: double &optional

(present if base/frameworks/sumstats/plugins/hll_unique.bro is loaded)

last_elements: Queue::Queue &optional

(present if base/frameworks/sumstats/plugins/last.bro is loaded)

This is the queue where elements are maintained. Use the SumStats::get_last function to get a vector of the current element values.

max: double &optional

(present if base/frameworks/sumstats/plugins/max.bro is loaded)

For numeric data, this tracks the maximum value given.

min: double &optional

(present if base/frameworks/sumstats/plugins/min.bro is loaded)

For numeric data, this tracks the minimum value given.

samples: vector &default = vector() &optional

(present if base/frameworks/sumstats/plugins/sample.bro is loaded)

This is the vector in which the samples are maintained.

sample_elements: count &default = 0 &optional

(present if base/frameworks/sumstats/plugins/sample.bro is loaded)

Number of total observed elements.

num_samples: count &default = 0 &optional

(present if base/frameworks/sumstats/plugins/sample.bro is loaded)

variance: double &optional

(present if base/frameworks/sumstats/plugins/variance.bro is loaded)

For numeric data, this calculates the variance.

prev_avg: double &optional

(present if base/frameworks/sumstats/plugins/variance.bro is loaded)

var_s: double &default = 0.0 &optional

(present if base/frameworks/sumstats/plugins/variance.bro is loaded)

std_dev: double &default = 0.0 &optional

(present if base/frameworks/sumstats/plugins/std-dev.bro is loaded)

For numeric data, this calculates the standard deviation.

sum: double &default = 0.0 &optional

(present if base/frameworks/sumstats/plugins/sum.bro is loaded)

For numeric data, this tracks the sum of all values.

topk: opaque &optional

(present if base/frameworks/sumstats/plugins/topk.bro is loaded)

unique: count &default = 0 &optional

(present if base/frameworks/sumstats/plugins/unique.bro is loaded)

If cardinality is being tracked, the number of unique items is tracked here.

unique_max: count &optional

(present if base/frameworks/sumstats/plugins/unique.bro is loaded)

unique_vals: set [SumStats::Observation] &optional

(present if base/frameworks/sumstats/plugins/unique.bro is loaded)

Value calculated for an observation stream fed into a reducer. Most of the fields are added by plugins.

SumStats::SumStat
Type:

record

name: string

An arbitrary name for the sumstat so that it can be referred to later.

epoch: interval

The interval at which this filter should be “broken” and the epoch_result callback called. The results are also reset at this time so any threshold based detection needs to be set to a value that should be expected to happen within this epoch.

reducers: set [SumStats::Reducer]

The reducers for the SumStat.

threshold_val: function (key: SumStats::Key, result: SumStats::Result) : double &optional

Provide a function to calculate a value from the SumStats::Result structure which will be used for thresholding. This is required if a threshold value is given.

threshold: double &optional

The threshold value for calling the threshold_crossed callback.

threshold_series: vector &optional

A series of thresholds for calling the threshold_crossed callback.

threshold_crossed: function (key: SumStats::Key, result: SumStats::Result) : void &optional

A callback that is called when a threshold is crossed.

epoch_result: function (ts: time, key: SumStats::Key, result: SumStats::Result) : void &optional

A callback that receives each of the results at the end of the analysis epoch. The function will be called once for each key.

epoch_finished: function (ts: time) : void &optional

A callback that will be called when a single collection interval is completed. The ts value will be the time of when the collection started.

SumStats represent an aggregation of reducers along with mechanisms to handle various situations like the epoch ending or thresholds being crossed.

It’s best to not access any global state outside of the variables given to the callbacks because there is no assurance provided as to where the callbacks will be executed on clusters.

Functions

SumStats::create
Type:function (ss: SumStats::SumStat) : void

Create a summary statistic.

SumStats::key2str
Type:function (key: SumStats::Key) : string

Helper function to represent a SumStats::Key value as a simple string.

Key:The metric key that is to be converted into a string.
Returns:A string representation of the metric key.
SumStats::observe
Type:function (id: string, key: SumStats::Key, obs: SumStats::Observation) : void

Add data into an observation stream. This should be called when a script has measured some point value.

Id:The observation stream identifier that the data point represents.
Key:The key that the value is related to.
Obs:The data point to send into the stream.
SumStats::request_key
Type:function (ss_name: string, key: SumStats::Key) : SumStats::Result

Dynamically request a sumstat key. This function should be used sparingly and not as a replacement for the callbacks from the SumStats::SumStat record. The function is only available for use within “when” statements as an asynchronous function.

Ss_name:SumStat name.
Key:The SumStat key being requested.
Returns:The result for the requested sumstat key.
Copyright 2013, The Bro Project. Last updated on June 15, 2015. Created using Sphinx 1.2.2.