Ansible ELK
ELK provisioning through Ansible.
Elasticsearch – It stores incoming logs from Logstash and provides the ability to search the logs/data in real-time.
Logstash – It does the processing (Collect, enrich and send it to Elasticsearch) of incoming logs sent by beats (forwarder).
Kibana – provides visualization of logs.
Pre-requisites
The amount of CPU, RAM, and storage that your Elastic Stack server will require depends on the volume of logs that we intend to gather commonly we are using,
RAM: 4GB
CPU: 2
As Usual, procedurally install Ansible on the local machine. Host details to be entered as,
vi /etc/ansible/hosts
Ping all the nodes,
[root@localhost ~]# ansible all –m ping
playbooks GitHub link,
https://github.com/Serlya/ELK-multOS
set a host file to connect all nodes.
In this playbook, we were created Elasticsearch Logstash Kibana through ansible playbooks.
First, we created a playbook for ubuntu,
ELK is java based web interface,
- name: Add the Java PPA repo
apt_repository:
repo: ppa:webupd8team/java
in this we adding the java repo.
Next, we accepting the license,
- name: Automatically accept the Oracle license
shell: echo debconf shared/accepted-oracle-license-v1-1 select true | sudo debconf-set-selections
download & install the java,
- name: Install Java 8
apt:
name: openjdk-8-jre-headless
state: present
update_cache: yes
it automatically sets the java path,
and now adding the Elasticsearch apt-key,
- name: Add Elasticsearch apt key
apt_key:
url: "https://packages.elastic.co/GPG-KEY-elasticsearch"
state: present
#Add the Elasticsearch apt repo. For versions 6 of the stack - use '6.x-prerelease':
- name: Adding Elasticsearch repo
apt_repository:
repo: deb https://artifacts.elastic.co/packages/6.x/apt stable main
state: present
# Installing Elasticsearch
- name: Install Elasticsearch
apt:
name: elasticsearch
state: latest
update_cache: yes
# Update Elasticsearch config file to allow access (to secure Elasticsearch, bind to 'localhost').
In ansible, there is a linein file command to used to manage lines in text files,
- name: Updating the config file to allow outside access
lineinfile:
destfile: /etc/elasticsearch/elasticsearch.yml
regexp: 'network.host:'
line: 'network.host: 0.0.0.0'
# Update Elasticsearch port in config file
- name: Updating the port in config file
lineinfile:
destfile: /etc/elasticsearch/elasticsearch.yml
regexp: 'http.port:'
line: 'http.port: 9200'
# Start Elasticsearch service,
- name: Starting Elasticsearch
service:
name: elasticsearch
state: started
# install logstash
- name: Install Logstash with apt
apt:
name: logstash
state: latest
# copy the configuration files for Logstash,
In this configuration files, location is master, so we are going to copy to the node,
- name: copy files
copy: src=/root/ansible-elk-playbook/02-beats-input.conf dest=/etc/logstash/conf.d/02-beats-input.conf
- name: copy filter files
copy: src=/root/ansible-elk-playbook/10-syslog-filter.conf dest=/etc/logstash/conf.d/10-syslog-filter.conf
- name: copy output file
copy: src=/root/ansible-elk-playbook/30-elasticsearch-output.conf dest=/etc/logstash/conf.d/30-elasticsearch-output.conf
#start logstash service
- name: Starting logstash
service:
name: logstash
state: started
# Install Kibana
- name: Install Kibana with apt
apt:
name: kibana
state: latest
update_cache: yes
# Configurations,
- name: Updating the config file to allow outside access
lineinfile:
destfile: /etc/kibana/kibana.yml
regexp: 'server.host:'
line: 'server.host: 0.0.0.0'
- name: Defining server port
lineinfile:
destfile: /etc/kibana/kibana.yml
regexp: 'server.port:'
line: 'server.port: 5601'
- name: Defining Elasticsearch URL
lineinfile:
destfile: /etc/kibana/kibana.yml
regexp: 'elasticsearch.url:'
line: 'elasticsearch.url: "http://localhost:9200"'
Starts the kibana service,
- name: Starting Kibana
service:
name: kibana
state: started
playbook for centos node,
in this playbook for elk provision in centos node,
- name: Add Elasticsearch key
rpm_key:
key: "https://packages.elastic.co/GPG-KEY-elasticsearch"
state: present
adding the elk repos,
- name: copy files
copy: src=/root/sunita/elk.repo dest=/etc/yum.repos.d
installing the elastic search,
-name: install elasticsearch
yum:
name: elasticsearch
state: latest
update_cache: yes
Update Elasticsearch config file to allow access
- name: Updating the config file to allow outside access
lineinfile:
destfile: /etc/elasticsearch/elasticsearch.yml
regexp: 'network.host:'
line: 'network.host: 0.0.0.0'
- name: Updating the port in config file
lineinfile:
destfile: /etc/elasticsearch/elasticsearch.yml
regexp: 'http.port:'
line: 'http.port: 9200'
this task to start the elasticsearch service,
- name: Starting Elasticsearch
service:
name: elasticsearch
state: started
install the kibana with yum,
- name: Install Kibana with yum
yum:
name: kibana
state: latest
update_cache: yes
configurations for kibana,
- name: Updating the config file to allow outside access
lineinfile:
destfile: /etc/kibana/kibana.yml
regexp: 'server.host:'
line: 'server.host: 0.0.0.0'
- name: Defining server port
lineinfile:
destfile: /etc/kibana/kibana.yml
regexp: 'server.port:'
line: 'server.port: 5601'
- name: Defining Elasticsearch URL
lineinfile:
destfile: /etc/kibana/kibana.yml
regexp: 'elasticsearch.url:'
line: 'elasticsearch.url: "http://localhost:9200"'
task for starts the kibana service,
- name: Starting Kibana
service:
name: kibana
state: started
next playbook elk provision for windows,
first, install the elasticsearch for windows through chocolatey, there are two ways to download and install the packages,
if you are download from URL it's not working properly better to download from chocolatey with versions,
while installing the packages through chocolatey versions for elasticsearch logstash & kibana must be the same.
- name: elasticsearch
win_chocolatey:
name: elasticsearch
version: 6.2.4
setting configuration files for elasticsearch,
- name: set path for elasticsearch
win_lineinfile:
path: C:\ProgramData\chocolatey\lib\elasticsearch\tools\elasticsearch-6.2.4\config\elasticsearch.yml
regexp: 'path.data:'
line: 'path.data:C:\ProgramData\chocolatey\lib\elasticsearch\tools\elasticsearch-6.2.4\logs\data'
- name: set log path elasticsearch
win_lineinfile:
path: C:\ProgramData\chocolatey\lib\elasticsearch\tools\elasticsearch-6.2.4\config\elasticsearch.yml
regexp: 'path.logs:'
line: 'path.logs:C:\ProgramData\chocolatey\lib\elasticsearch\tools\elasticsearch-6.2.4\logs'
- name: memory setup
win_lineinfile:
path: C:\ProgramData\chocolatey\lib\elasticsearch\tools\elasticsearch-6.2.4\config\elasticsearch.yml
regexp: 'bootstrap.memory_lock: true'
line: 'bootstrap.memory_lock: true'
- name: Updating the config file to allow outside access
win_lineinfile:
path: C:\ProgramData\chocolatey\lib\elasticsearch\tools\elasticsearch-6.2.4\config\elasticsearch.yml
regexp: 'network.host:'
line: 'network.host: 0.0.0.0'
- name: Updating the port in config file
win_lineinfile:
path: C:\ProgramData\chocolatey\lib\elasticsearch\tools\elasticsearch- 6.2.4\config\elasticsearch.yml
regexp: 'http.port:'
line: 'http.port: 9200'
starts the elasticsearch service,
- name: elasticsearch start
win_command: net start elasticsearch-service-x64
install the kibana
- name: kibana
win_chocolatey:
name: kibana
version: 6.2.4
configuring the kibana config files to access,
- name: Defining server port
win_lineinfile:
path: C:\ProgramData\chocolatey\lib\kibana\tools\kibana-6.2.4-windows-x86_64\config\kibana.yml
regexp: 'server.port:'
line: 'server.port: 5601'
- name: Updating the config file to allow outside access
win_lineinfile:
path: C:\ProgramData\chocolatey\lib\kibana\tools\kibana-6.2.4-windows-x86_64\config\kibana.yml
regexp: 'server.host:'
line: 'server.host: 192.168.1.18'
- name: Defining Elasticsearch URL
win_lineinfile:
path: C:\ProgramData\chocolatey\lib\kibana\tools\kibana-6.2.4-windows-x86_64\config\kibana.yml
regexp: 'elasticsearch.url:'
line: 'elasticsearch.url: "http://localhost:9200"'
start the kibana service if error occurs it skips and will go to next task,
- name: kibana service
win_command: powershell.exe C:\ProgramData\chocolatey\lib\kibana\tools\kibana-6.2.4-windows-x86_64\bin\kibana.bat
ignore_errors: yes
install the logstash from logstash,
- name: logstash
win_chocolatey:
name: logstash
version: 6.2.4
edit the config files,
edit the config file in master and copy that files into windows
- name: copy files
win_copy:
src: '/home/logstash.conf'
dest: 'C:\ProgramData\chocolatey\lib\logstash\tools\logstash-6.2.4\bin\logstash.conf'
[root@localhost ~]#cat logstash.conf
input {
beats {
port => 5044
}
}
output {
elasticsearch {
hosts => "localhost:9200"
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}
Start the Logstash service,
-name: Logstash service
win_command: powershell.exe C:\ProgramData\chocolatey\lib\logstash\tools\logstash-6.2.4\bin\logstash.bat -f Logstash.conf &
name: Logstash-6.4
path: C:\ProgramData\chocolatey\lib\logstash\tools\logstash-6.2.4\bin\logstash.bat -f Logstash.conf
start_mode: auto
finally to run this playbook,
[root@localhost ~]# nohup ansible-playbook single.yml & > /tmp/nohup.out 2>&1 &
Nohup is short for “No Hangups.” It’s not a command that you run by itself. Nohup is a supplemental command that tells the Linux system not to stop another command once it has started. That means it’ll keep running until it’s done, even if the user that started it logs out.
we need to start the services in the background that's why we used the nohup command,
Check the node IP with elk ports.
Relevant Blogs:
Recent Comments
No comments
Leave a Comment
We will be happy to hear what you think about this post