This blog will explore a step-by-step guide on deploying the ELK Stack on Kubernetes. It’s a powerful combination of Elasticsearch, Logstash, and Kibana, enabling scalable search, analytics, and log processing for data-driven applications. By the end of this guide, you'll have a fully functional ELK stack setup ready to manage and analyze your logs effectively.
So let’s figure out what exactly ELK stack is
The ELK stack is an abbreviation for Elasticsearch, Logstash, and Kibana, which offers the following capabilities:
- Elasticsearch: a scalable search and analytics engine with a log analytics tool and application-formed database, perfect for data-driven applications.
- Logstash: a log-processing tool that collects logs from various sources, parses them, and sends them to Elasticsearch for storage and analysis.
- Kibana: A powerful visualization tool that allows you to explore and analyze the data stored in Elasticsearch using interactive charts, graphs, and dashboards.
The infrastructure of Elasticsearch
Before we dive into deploying the ELK Stack, let's first understand the critical components of Elasticsearch's infrastructure:
- Nodes: elasticsearch runs on dedicated servers called nodes, which operate as binaries for search and analytics tasks.
- Shards: the database space is logically divided into shards, enabling faster data accessibility and distribution.
- Indices: elasticsearch organizes the stored data into indices, facilitating efficient data management.
Configuring the ELK Stack:
You'll need a Kubernetes cluster to deploy the ELK Stack on Kubernetes. If you already have one, you can proceed with the deployment. Alternatively, you can use the provided GitHub repository with Terraform files to set up a Kubernetes cluster.
Utilizing Helm charts, we can efficiently deploy Elasticsearch. Modify the values file to match your specific requirements, such as adjusting the number of replicas or turning certain features on/off. Download them from Artifactory Hub.
clusterName: "itsyndicateblog" replicas: 1 minimumMasterNodes: 1 createCert: true secret: enabled: true password: "" # generated randomly if not defined image: "docker.elastic.co/elasticsearch/elasticsearch" imageTag: "8.5.1" resources: requests: cpu: "200m" memory: "500Mi" limits: cpu: "300m" memory: "1Gi" ingress: enabled: false # enable ingress only if you need external access to elasticsearch cluster hosts: - host: elastic.itsyndicate.org paths: - path: /
Once you've customized the values, use the Helm chart to install Elasticsearch:
helm install elasticsearch -f elasticsearch-values.yaml <chart-name>
Note: Ensure you have configured the drivers (EBS or EFS) for persistent volumes.
Kibana deployment is straightforward using Helm charts. In the values file, specify the URL and port of the Elasticsearch service:
elasticsearchHosts: "https://elasticsearch-master:9200" enterpriseSearch: host: "https://elasticsearch-master:9200"
helm install kibana -f kibana-values.yaml <chart-name>
Check if Kibana is installed correctly, port forward the container’s port to the local network (I am using K8s Lens)
To manage logs effectively, we use Logstash and Filebeat. Filebeat collects records from various sources and Logstash processes and sends them to Elasticsearch.
- Clone repository with configs: https://github.com/inemyrovsk/tf-modules/tree/master/eks/manifests/logstash-k8s
- Move to tf-modules/eks/manifests/logstash-k8s
- Edit configmap.yaml file
- add elasticsearch host, user and password(you can take them from “Secrets” Kubernetes resource)
- Apply templates:
kubectl apply -f logstash-k8s -n $CHANGE_TO_ELASTIC_NS
- Ensure Filebeat's configuration points to the correct log files on your nodes. Usually, in EKS, it’s the /var/log/containers folder. To check it, log in to one of your nodes and move to the /var/log/containers directory; if there are no files, try to change the directory.
- In case everything is correct, apply Kubernetes templates:
kubectl apply -f filebeat-k8s
- Enter the eks/manifests folder from cloned repository.
- Execute command:
kubeclt apply -f app -n default
After installation is complete, revisit Kibana and create an elasticsearch index.
Creating an index:
Create logstash index pattern: logstash-[namespace]*
Now you should see logs from the deployed application. If not, make some requests to this app and try to troubleshoot the issue; refer to the video guide in case help is required.
You've successfully deployed the ELK Stack on Kubernetes, empowering your applications with robust log analysis and data-driven insights. Elasticsearch, Logstash, and Kibana seamlessly handle large data streams and provide meaningful visualizations.
Now that you have a robust logging solution, you can efficiently manage your logs and gain valuable insights. Happy analyzing!
Thank you for reading this guide on deploying the ELK Stack. Feel free to reach out if you have any questions or require further assistance. Happy coding!
Discover how our services can benefit your business. Leave your contact information and our team will reach out to provide you with detailed information tailored to your specific needs. Take the next step towards achieving your business goals.
Spoiler alert: understand that could be a game-changer for your IT ops!
Enhance your project's security with an innovative GCP service account key rotation mechanism using Kubernetes and Python
Infrastructure management requires consideration of many things, such as application performance, observability, reliability, and disaster recovery, to name a few.