ELK Stack Architecture Deep-Dive

Setting up Elasticsearch, Logstash, and Kibana

Photo by Nick Fewings on Unsplash

In this topic, we will discuss ELK stack architecture: Elasticsearch, Logstash, and Kibana. It is an open-source tool (although some weird changes going on with licensing). It is used for LOG monitoring and analytics.

Elastic (ELK) Stack:

An elastic stack is a group of Open Source products from Elastic designed to help users to take data from any type of source and in any format and search, analyze, and visualize that data in real-time. It uses Logstash for log aggregation, Elasticsearch for searching, and Kibana for visualizing and analyzing data.

• ElasticSearch: (Database) Store, Search and Analyse
• Logstash: (Log collection engine) Collect Logs and events data, parse and Transform
• Kibana: (VI tool) Explore, Visualize and Share, eliminating the need for complex SQL query execution
• Beats: is a data shipper which collects the data at the client and ship it either to elasticsearch or logstash

Beats(agent) collects the logs and send data to Logstash, Logstash will do a filter, parse and transform into meaning full data and store it into Elasticsearch and then make it available in a fast and reliable way and Kibana uses a…