SBM Labs

Cyber Security Collaboration Project

Centralized logs collection and analysis. Elasticsearch indices management.
16 May 2015

We have reviewed Centralized logs collection and analysis. Elasticsearch integration with correlation engine earlier. Let's look at how we can manage and monitor Elasticsearch indices.

Assume that we have several locations: loc1, loc2 and loc3. The following nodes are installed in each location:

- Client nodes (Logstash) for receiving, filtering, normalization and transmission data to cluster.
- Data nodes (Elasticsearch) for storing data (indices) that were received from client nodes.
- Master nodes (Elasticsearch) for cluster management.

We use default cluster configuration. Each index creates daily, has 5 shards and 1 replica.

# Set the number of shards (splits) of an index (5 by default):
#index.number_of_shards: 5

# Set the number of replicas (additional copies) of an index (1 by default):
#index.number_of_replicas: 1

1. Assume that network channel to loc3 is unstable and has a low bandwidth. We are going to exclude indices storing from loc1 and loc2 on data nodes in loc 3 and other way round. This task can be solved as follows:

1.1. Enable cluster allocation.

# curl -XPUT localhost:9200/_cluster/settings -d '{"persistent" : {"cluster.routing.allocation.enable" : "all"}}'

1.2. Set location tag for each data node.

Edit configuration file elasticsearch.yml.
# vi /etc/elasticsearch/elasticsearch.yml

Add the following line.
node.tag: loc1

Restart Elasticsearch.
# systemctl restart elasticsearch.service

1.3. Set location for indices.

# curl -XPUT localhost:9200/INDEX_FROM_LOC1/_settings -d '{"index.routing.allocation.include.tag" : "loc1,loc2"}'
# curl -XPUT localhost:9200/INDEX_FROM_LOC2/_settings -d '{"index.routing.allocation.include.tag" : "loc1,loc2"}'
# curl -XPUT localhost:9200/INDEX_FROM_LOC3/_settings -d '{"index.routing.allocation.include.tag" : "loc3"}'

Also for this task we can use tool elasticsearch-curator.

# yum install python-pip
# pip install elasticsearch-curator
# curator allocation --rule tag=loc1 indices --prefix INDEX_FROM_LOC1

2. Storing large number of open indices creates a load on cluster. We can close, for example, indices older then 7 days in order to reduce it.

# curator close indices --older-than 7 --time-unit days --timestring '%Y.%m.%d' --prefix INDEX_NAME

3. Storing large number of indices require a lot of disk space. We can delete, for example, indices older then 14 days.

# curator delete indices --older-than 14 --time-unit days --timestring '%Y.%m.%d' --prefix INDEX_NAME

You can use Cron in order to automate these tasks.

4. There is plugin Marvel for cluster and indices monitoring. This plugin must be installed on each cluster node.

# cd /usr/share/elasticsearch/
# bin/plugin -i elasticsearch/marvel/latest

Centralized logs collection and analysis. Elasticsearch integration with correlation engine.
Centralized logs collection and analysis. Elasticsearch cluster configuration.
Centralized logs collection and analysis. Elasticsearch, Logstash and Kibana. Introduction.
OSSEC. Use Cases.
OSSEC. Introduction.