Introduction

If you're using the EFK stack (Elasticsearch, FluentBit, Kibana) in a production Kubernetes environment on AWS, you need a reliable and scalable storage backend for Elasticsearch.

Since Elasticsearch is a search engine, not a traditional database, it does not require an external database for logging. Instead, it stores logs in its own Lucene-based document store. However, in AWS production environments, you should use AWS OpenSearch Service (Managed Elasticsearch) to store logs reliably.

This guide will cover:
✅ How to deploy Elasticsearch (AWS OpenSearch) in a Kubernetes production environment
✅ How to integrate FluentBit to send logs to AWS OpenSearch
✅ How to set up Kibana for visualization
✅ How to ensure security, scalability, and performance

Step 1: Set Up AWS OpenSearch (Managed Elasticsearch)

Since running self-hosted Elasticsearch in Kubernetes can be resource-intensive, we use AWS OpenSearch (Managed Elasticsearch) for a scalable, production-ready solution.

1.1 Create an AWS OpenSearch Domain

Log in to your AWS Console → Navigate to AWS OpenSearch Service.
Click on Create Domain.
Choose Deployment Type:
- ✅ Production (Recommended)
- ❌ Development and Testing (For non-production use cases)
Choose an Elasticsearch version (7.x or later, compatible with FluentBit).
Select Instance Type:
- Small workloads: t3.small.search
- Production: r5.large.search or r5.xlarge.search
Enable Dedicated Master Nodes to prevent failures.
Set EBS Storage (Recommended at least 100GB for logs).

1.2 Configure Network and Security

Choose VPC access (Recommended for private access).
Allow traffic from your Kubernetes cluster’s security group.
Enable IAM-based access control for authentication.

1.3 Set Up IAM Policies for FluentBit

To allow FluentBit to send logs to OpenSearch, create an IAM policy with permissions:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "es:ESHttpPut",
                "es:ESHttpPost"
            ],
            "Resource": "arn:aws:es:us-east-1:123456789012:domain/your-opensearch-domain/*"
        }
    ]
}

Attach this policy to the IAM Role used by FluentBit.

Step 2: Deploy FluentBit in Kubernetes

2.1 Install FluentBit as a DaemonSet

FluentBit collects logs from Kubernetes nodes and forwards them to AWS OpenSearch.

Create FluentBit ConfigMap

Create a ConfigMap for FluentBit configuration:

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: kube-system
data:
  fluent-bit.conf: |
    [SERVICE]
        Flush        5
        Log_Level    info
        Parsers_File parsers.conf

    [INPUT]
        Name              tail
        Path              /var/log/containers/*.log
        Tag               kube.*
        Parser            docker

    [FILTER]
        Name             kubernetes
        Match            kube.*
        Kube_URL         https://kubernetes.default.svc:443
        Merge_Log        On

    [OUTPUT]
        Name  es
        Match *
        Host  your-opensearch-endpoint
        Port  443
        AWS_Auth On
        Region us-east-1
        Index kubernetes-logs
        Type  _doc

Apply the ConfigMap

kubectl apply -f fluent-bit-config.yaml

2.2 Deploy FluentBit DaemonSet

Now deploy FluentBit as a DaemonSet to collect logs from all Kubernetes nodes.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluent-bit
  namespace: kube-system
spec:
  selector:
    matchLabels:
      name: fluent-bit
  template:
    metadata:
      labels:
        name: fluent-bit
    spec:
      containers:
      - name: fluent-bit
        image: fluent/fluent-bit:latest
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: config-volume
          mountPath: /fluent-bit/etc
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: config-volume
        configMap:
          name: fluent-bit-config

Apply the DaemonSet

kubectl apply -f fluent-bit-daemonset.yaml

Step 3: Deploy Kibana for Log Visualization

AWS OpenSearch comes with Kibana pre-installed.

3.1 Access Kibana

Navigate to AWS OpenSearch Dashboard.
Click on OpenSearch Dashboards (Kibana UI).
Log in using IAM Role authentication or OpenSearch authentication.

3.2 Configure Index Pattern

In Kibana, go to Stack Management → Index Patterns.
Click Create Index Pattern.
Enter kubernetes-logs-* as the pattern (same as in FluentBit config).
Select @timestamp as the primary time field.

3.3 Visualize Logs

Use Discover in Kibana to view logs from Kubernetes.
Apply filters based on namespace, pod name, or error levels.
Create dashboards for real-time log monitoring.

Step 4: Optimize and Secure the Setup

4.1 Enable Index Lifecycle Management

To prevent high storage costs, configure log retention policies:

{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_size": "50gb",
            "max_age": "30d"
          }
        }
      },
      "delete": {
        "min_age": "90d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

4.2 Scale OpenSearch for High Log Volume

Use r5.large.search or r5.xlarge.search instances.
Enable UltraWarm storage for historical logs.
Use dedicated master nodes for stability.

4.3 Secure OpenSearch and FluentBit

Enable IAM authentication for FluentBit.
Restrict access using VPC-based security groups.
Encrypt logs with AWS KMS encryption.

Final Architecture in Production

[FluentBit DaemonSet] → [AWS OpenSearch (Managed Elasticsearch)] → [Kibana Dashboard]

FluentBit collects logs from Kubernetes nodes.
Logs are sent to AWS OpenSearch securely.
Kibana queries OpenSearch to visualize logs.

Conclusion

By integrating AWS OpenSearch (Elasticsearch) with FluentBit and Kibana, you achieve a highly scalable, production-ready logging setup for Kubernetes.

Benefits of This Setup:

Fully managed Elasticsearch (AWS OpenSearch) for scalability.
Lightweight FluentBit agent for efficient log forwarding.
Kibana for real-time log monitoring and alerting.
Secure IAM-based access to AWS OpenSearch.

This approach ensures cost-effectiveness, scalability, and security, making it ideal for production Kubernetes logging.

Connecting Elasticsearch to a Database in AWS for Production Logging with EFK (Elasticsearch, FluentBit, Kibana) Stack

Table of contents