SBM Labs

Cyber Security Collaboration Project

Centralized logs collection and analysis. Elasticsearch cluster configuration.
24 Apr 2015

We have discussed Centralized logs collection and analysis. Elasticsearch, Logstash and Kibana. Introduction. in the last article. Let's review cluster configuration now.

For example, we have servers that are located at three geographically distant datacenters. We are going to perform centralized Linux logs collection using Syslog protocol.

Now we will discuss design of Elasticsearch cluster. We will install client node for logs normalization and transmission to cluster, data node for storing indexed data and master node for cluster management at each location. Also we will install additional client node at one location for events analysis. So it will be ten hosts in our cluster.

We will use OS Centos 7 for our cluster hosts. The names of the hosts will display their location and purpose. For example, loc1-logstash-01, loc1-data-01, loc1-master-01, loc1-kibana-01.

1. Elasticsearch installation.

We are going to install java and elasticsearch on data nodes (loc1-data-01, loc2-data-01, loc3-data-01), master nodes (loc1-master-01, loc2-master-01, loc3-master-01) and client node (loc1-kibana-01).

# yum install java-1.7.0-openjdk

# java -version
java version "1.7.0_75"
OpenJDK Runtime Environment (rhel-2.5.4.2.el7_0-x86_64 u75-b13)
OpenJDK 64-Bit Server VM (build 24.75-b04, mixed mode)

# rpm --import http://packages.elasticsearch.org/GPG-KEY-elasticsearch

vi /etc/yum.repos.d/elasticsearch.repo
[elasticsearch-1.5]
name=Elasticsearch repository for 1.5.x packages
baseurl=http://packages.elasticsearch.org/elasticsearch/1.5/centos
gpgcheck=1
gpgkey=http://packages.elasticsearch.org/GPG-KEY-elasticsearch
enabled=1

# yum install elasticsearch

2. Elasticsearch configuration.

By default after installation Elasticsearch service will act as data and master node (node.master: true and node.data: true). So we can use one server in order to install Elasticsearch, Logstash and KibanŠ°. As a result we will receive solution for small-scale environment (company). In this article we configure Elasticsearch using different node types for each cluster host in order to understand how it is possible to scale ELK stack.

vi /etc/elasticsearch/elasticsearch.yml

Elasticsearch master node configuration (loc1-master-01, loc2-master-01, loc3-master-01).

cluster.name: elk
node.name: "loc1-master-01"
node.master: true
node.data: false
http.enabled: false

discovery.zen.minimum_master_nodes: 2
discovery.zen.ping.timeout: 3s
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["loc1-master-01", "loc2-master-01", "loc3-master-01"]

You sholud pay your attention at number of minimum master nodes. At any time it must be only one active master node in the cluster. Otherwise cluster split-brain can occur. Active master node should be elected by quorum (n/2+1). In our case it's discovery.zen.minimum_master_nodes = (3/2+1) = 2.

Elasticsearch data node configuration (loc1-data-01, loc2-data-01, loc3-data-01).

cluster.name: elk
node.name: "loc1-data-01"
node.master: false
node.data: true
http.enabled: false

discovery.zen.minimum_master_nodes: 2
discovery.zen.ping.timeout: 3s
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["loc1-master-01", "loc2-master-01", "loc3-master-01"]

Elasticsearch client node configuration (loc1-kibana-01).

cluster.name: elk
node.name: "loc1-data-01"
node.master: false
node.data: false

discovery.zen.minimum_master_nodes: 2
discovery.zen.ping.timeout: 3s
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["loc1-master-01", "loc2-master-01", "loc3-master-01"]

Port TCP/9300 is used for cluster communication.

It should be enabled Elasticsearch HTTP protocol (TCP/9200) on the client node loc1-kibana-01 in order to get access to indexed data. This configuration is by default. For other nodes we can disable it - http.enabled: false.

3. Start Elasticsearch.

# systemctl enable elasticsearch.service
# systemctl start elasticsearch.service

Check cluster health on the client node loc1-kibana-01:

curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'
{
"cluster_name" : "elk",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 7,
"number_of_data_nodes" : 3,
...
}

Path to Elasticsearch log folder is /var/log/elasticsearch/.

4. Kibana installation.

All actions must be done on the client node loc1-kibana-01.

# wget https://download.elasticsearch.org/kibana/kibana/kibana-4.0.1-linux-x64.tar.gz

# tar xvf kibana-4.0.1-linux-x64.tar.gz

# mkdir -p /opt/kibana

cp -R ~/kibana-4.0.1-linux-x64/* /opt/kibana/

vi /etc/systemd/system/kibana4.service

[Service]
ExecStart=/opt/kibana/bin/kibana
Restart=always
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=kibana4
User=root
Group=root
Environment=NODE_ENV=production

[Install]
WantedBy=multi-user.target

5. Start Kibana.

systemctl enable kibana4
systemctl start kibana4

Path to Kibana log file is /var/log/messages.

Path to Kibana configuration file is /opt/kibana/config/kibana.yml. Kibana connects to localhost Elasticsearch client node by port TCP/9200 and gets access to cluster data. Kibana web interface is available by port TCP/5601.

6. Logstash installation.

You should install java and Logstash on the hosts loc1-logstash-01, loc2-logstash-01, loc3-logstash-01.

# yum install java-1.7.0-openjdk

# java -version
java version "1.7.0_75"
OpenJDK Runtime Environment (rhel-2.5.4.2.el7_0-x86_64 u75-b13)
OpenJDK 64-Bit Server VM (build 24.75-b04, mixed mode)

# rpm --import http://packages.elasticsearch.org/GPG-KEY-elasticsearch

vi /etc/yum.repos.d/logstash.repo
[logstash-1.5]
name=logstash repository for 1.5.x packages
baseurl=http://packages.elasticsearch.org/logstash/1.5/centos
gpgcheck=1
gpgkey=http://packages.elasticsearch.org/GPG-KEY-elasticsearch
enabled=1

# yum install logstash

7. Logstash configuration.

Configuration that listed below allows to use Logstash as cluster client node.

# vi /etc/init.d/logstash

LS_JAVA_OPTS="-Djava.io.tmpdir=${LS_HOME} -Des.config=/etc/logstash/elasticsearch.yml"

# vi /etc/logstash/elasticsearch.yml

cluster.name: elk
discovery.zen.ping.unicast.hosts: ["loc1-master-01", "loc2-master-01", "loc3-master-01"]

Let's configure Logstash to receive syslog messages by port 5514 and send it to cluster. Default index [logstash-]YYYY.MM.DD will be assigned to all events.

vi /etc/logstash/conf.d/custom.conf

input {
syslog {
port => 5514
}
}

output {
elasticsearch { }
}

Path to Logstash log folder is /var/log/logstash/.

8. Start logstash.

# /etc/init.d/logstash start

Check cluster health on the client node loc1-kibana-01:

curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'
{
"cluster_name" : "elk",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 10,
"number_of_data_nodes" : 3,
...
}

Base cluster configuration is completed. Now we need to configure rsyslog on servers at each location in order to forward security events to Logstash. For example, authpriv.* @loc1-logstash-01:5514.

Then go to Kibana web interface http://loc1-kibana-01:5601 and set default index name. You can add index only for events that are in cluster.

After it go to section Discover, choose Time Filter = 15 minutes and Refresh Interval = 5 seconds. You will see events with default index [logstash-]YYYY.MM.DD.

In the next article we will review ELK stack integration with correlation engine.

Centralized logs collection and analysis. Elasticsearch indices management.
Centralized logs collection and analysis. Elasticsearch integration with correlation engine.
Centralized logs collection and analysis. Elasticsearch, Logstash and Kibana. Introduction.
OSSEC. Use Cases.
OSSEC. Introduction.