Bro’s default ASCII log format is not exactly the most efficient way for searching large volumes of data. ElasticSearch is a new data storage technology for dealing with tons of data. It’s also a search engine built on top of Apache’s Lucene project. It scales very well, both for distributed indexing and distributed searching.
Contents
This writer plugin is still in testing and is not yet recommended for production use! The approach to how logs are handled in the plugin is “fire and forget” at this time, there is no error handling if the server fails to respond successfully to the insertion request.
Download the latest version from: http://www.elasticsearch.org/download/. Once extracted, start ElasticSearch with:
# ./bin/elasticsearch
For more detailed information, refer to the ElasticSearch installation documentation: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup.html
First, ensure that you have libcurl installed then run configure:
# ./configure
[...]
====================| Bro Build Summary |=====================
[...]
cURL: true
[...]
ElasticSearch: true
[...]
================================================================
The easiest way to enable ElasticSearch output is to load the tuning/logs-to-elasticsearch.bro script. If you are using BroControl, the following line in local.bro will enable it:
@load tuning/logs-to-elasticsearch
With that, Bro will now write most of its logs into ElasticSearch in addition to maintaining the Ascii logs like it would do by default. That script has some tunable options for choosing which logs to send to ElasticSearch, refer to the autogenerated script documentation for those options.
There is an interface being written specifically to integrate with the data that Bro outputs into ElasticSearch named Brownian. It can be found here:
https://github.com/grigorescu/Brownian
A common problem encountered with ElasticSearch is too many files being held open. The ElasticSearch website has some suggestions on how to increase the open file limit.
Lots.