In containerized and Kubernetes environments, applications run in pods that can be short-lived and scaled across many nodes. To get centralized, searchable logs you need a clear logging architecture: where logs are written, how they are collected, and how they reach a backend (e.g. Elasticsearch, Loki, or a cloud logging service). This post describes a full architecture using an application, a log sidecar, and a log daemon, with examples and when to use each.

---

Why centralized logging?

• Containers are ephemeral — When a pod is restarted or rescheduled, local log files (and often stdout) are lost unless something has already shipped them.
• Many pods, many nodes — You need one place to search and correlate logs from all replicas and nodes.
• Apps write in different ways — Some write to stdout/stderr, others to files (e.g. /var/log/app/app.log). The collection strategy must match.

A typical target is: application → collector(s) → log backend (e.g. Elasticsearch, Loki, S3, CloudWatch). The “collector” layer can be a sidecar (per-pod), a daemon (per-node), or both.

---

Architecture overview

The diagram below shows the three main pieces: the application (in a pod), an optional log sidecar in the same pod, and a log daemon (DaemonSet) on each node. Logs can go app → sidecar → daemon → backend, or app → daemon → backend if you skip the sidecar.

+------------------+     +------------------+     +------------------+
|  Log backend     |     |  Log backend     |     |  Log backend     |
|  (Elasticsearch, |     |  (Loki, S3,      |     |  (CloudWatch,    |
|   OpenSearch)    |     |   etc.)          |     |   etc.)          |
+--------^---------+     +--------^---------+     +--------^---------+
         |                       |                       |
         |  HTTP / TCP / etc.    |                       |
         |                       |                       |
+--------+-----------------------+-----------------------+------------+
|                         Kubernetes cluster                          |
|  +---------------- Node 1 ---------------+  (same on Node 2, ...)   |
|  |  Log daemon (DaemonSet)               |                          |
|  |  e.g. Fluent Bit, Filebeat            |                          |
|  |  - Reads container logs from node     |                          |
|  |  - Or receives from sidecars          |                          |
|  |  - Buffers, parses, forwards          |                          |
|  +------------------^--------------------+                          |
|                     |                                               |
|  +------------------+--------------------------------------------+  |
|  |  Pod: my-app                                                  |  |
|  |  +----------------+  +----------------+  shared volume (logs) |  |
|  |  | app container   |  | log sidecar    |  /var/log/app        |  |
|  |  | writes to       |  | tails files,   |  <--------+          |  |
|  |  | /var/log/app/   |  | forwards to    |           |          |  |
|  |  | or stdout       |  | daemon/backend |  ---------+          |  |
|  |  +----------------+   +----------------+                      |  |
|  +---------------------------------------------------------------+  |
+---------------------------------------------------------------------+

• Application: Writes logs to stdout/stderr (handled by the container runtime and visible to the daemon) or to files in a volume (often need a sidecar to ship them).
• Log sidecar: Runs in the same pod, reads log files from a shared volume, and forwards them (e.g. to the log daemon or directly to the backend). Used when the app writes to files.
• Log daemon: Runs on every node (DaemonSet), collects logs from the node (e.g. container stdout/stderr under /var/log/pods) and/or from sidecars, then forwards to the backend.

---

Application: where logs come from

Style	Where logs go	Who can collect
Stdout / stderr	Container runtime writes to files on the node (e.g. under /var/log/pods/...)	Log daemon can read these files; no sidecar needed.
File in container	e.g. /var/log/app/app.log inside the container	Not visible to the node; need a sidecar or an agent that runs inside the pod to read and ship.
File on shared volume	e.g. emptyDir or PVC mounted at /var/log/app; app and sidecar both see it	Sidecar tails the file(s) and forwards; daemon can also read if the volume is node-local and exposed (less common).

Recommendation: Prefer stdout/stderr so the log daemon can collect without a sidecar. Use a sidecar when the app must write to files (e.g. legacy app, or rotation/format that is file-based).

---

Log sidecar pattern

A log sidecar is an extra container in the same pod whose job is to read log data (usually from a shared volume) and send it to a log daemon or backend.

When to use:

• App writes to log files (not just stdout).
• You want parsing, filtering, or multiline handling in the pod before sending.
• You want to ship to a different destination per app (e.g. different index or stream).

Flow:

1. App and sidecar share a volume (e.g. emptyDir) mounted at /var/log/app.
2. App writes logs to e.g. /var/log/app/app.log.
3. Sidecar runs a log shipper (e.g. Fluent Bit, Filebeat, or a small tailer script) that tails the file(s) and forwards to the daemon (e.g. over HTTP or TCP) or directly to the backend.

Important: The sidecar uses extra CPU/memory per pod and duplicates the shipper on every replica. Use it only when the daemon cannot read the logs (e.g. file-only app).

---

Log daemon (DaemonSet) pattern

A log daemon runs one pod per node (Kubernetes DaemonSet). It reads logs that are on the node (e.g. container log files created by the runtime) and optionally receives streams from sidecars, then forwards to the backend.

When to use:

• You want one collector per node instead of per pod (fewer moving parts, less resource use).
• Apps write to stdout/stderr — the runtime writes these to files under /var/log/pods/ (or similar); the daemon reads them.
• You want node-level metadata (node name, pod UID) attached to every log line.

Flow:

1. Container runtime writes stdout/stderr of each container to a file on the node (e.g. /var/log/pods/<namespace>_<pod>_<uid>/<container>.log).
2. The log daemon (e.g. Fluent Bit, Filebeat) is configured to tail these paths (often via a symlink or directory scan).
3. The daemon enriches with metadata (namespace, pod name, container name, labels), optionally parses/filters, and sends to the backend.

Advantage: No change to application pods; works for any container that logs to stdout/stderr.

---

Full architecture: sidecar + daemon

When the app writes to files, you combine both:

1. App writes to a shared volume (e.g. /var/log/app).
2. Sidecar in the same pod tails those files and sends to the log daemon (e.g. Fluent Bit on the node listening on a port, or a unix socket mounted from the node).
3. Log daemon receives from sidecars and/or reads container stdout/stderr from the node; forwards everything to the backend.

So: app (files) → sidecar → daemon → backend, and app (stdout) → daemon → backend. One backend, one daemon per node, sidecars only where needed.

---

Example 1: Pod with log sidecar (shared volume)

App writes to /var/log/app/app.log; sidecar runs Fluent Bit and forwards to the node’s log daemon (or directly to backend).

apiVersion: v1
kind: Pod
metadata:
  name: my-app
  labels:
    app: my-app
spec:
  containers:
    # Main application: writes to shared volume
    - name: app
      image: my-registry.io/my-app:latest
      volumeMounts:
        - name: app-logs
          mountPath: /var/log/app
      # App is configured to write logs to /var/log/app/app.log

    # Log sidecar: tails /var/log/app and forwards
    - name: log-sidecar
      image: fluent/fluent-bit:2.2
      volumeMounts:
        - name: app-logs
          mountPath: /var/log/app
          readOnly: true
      env:
        - name: FLUENT_BIT_DAEMON_HOST
          valueFrom:
            fieldRef:
              fieldPath: status.hostIP
      # Fluent Bit config: read from /var/log/app/*.log, forward to daemon
  volumes:
    - name: app-logs
      emptyDir: {}

The sidecar’s Fluent Bit config would tail /var/log/app/*.log and output to the daemon (e.g. Host ${FLUENT_BIT_DAEMON_HOST} and a port). The daemon runs on the node (see next example).

---

Example 2: Log daemon (DaemonSet)

One Fluent Bit pod per node: read container logs from the node and optionally accept TCP/HTTP from sidecars; forward to Elasticsearch (or another backend).

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluent-bit
  namespace: logging
spec:
  selector:
    matchLabels:
      app: fluent-bit
  template:
    metadata:
      labels:
        app: fluent-bit
    spec:
      serviceAccountName: fluent-bit
      containers:
        - name: fluent-bit
          image: fluent/fluent-bit:2.2
          resources:
            requests:
              memory: "64Mi"
              cpu: "50m"
            limits:
              memory: "128Mi"
              cpu: "200m"
          volumeMounts:
            # Container logs on the node (path depends on runtime and OS)
            - name: varlog
              mountPath: /var/log
              readOnly: true
            - name: varlibdockercontainers
              mountPath: /var/lib/docker/containers
              readOnly: true
            - name: fluent-bit-config
              mountPath: /fluent-bit/etc/fluent-bit.conf
              subPath: fluent-bit.conf
      volumes:
        - name: varlog
          hostPath:
            path: /var/log
        - name: varlibdockercontainers
          hostPath:
            path: /var/lib/docker/containers
        - name: fluent-bit-config
          configMap:
            name: fluent-bit-config
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: logging
data:
  fluent-bit.conf: |
    [SERVICE]
        Flush         5
        Log_Level     info

    [INPUT]
        Name              tail
        Path              /var/log/containers/*.log
        Parser            docker
        Tag               kube.*
        Refresh_Interval  10

    [FILTER]
        Name                kubernetes
        Match               kube.*
        Kube_URL            https://kubernetes.default.svc:443
        Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
        Merge_Log           On
        Keep_Log            Off
        Labels              On
        Annotations         Off

    [OUTPUT]
        Name  es
        Match *
        Host  elasticsearch.logging.svc.cluster.local
        Port  9200
        Logstash_Format On
        Logstash_Prefix logs

Explanation:

• INPUT (tail): Reads container log files under /var/log/containers/*.log (typical path when the kubelet writes container logs; adjust for your runtime).
• FILTER (kubernetes): Enriches with pod/namespace/labels so you can filter by app in the backend.
• OUTPUT (es): Sends to Elasticsearch in the cluster; replace with Loki, HTTP, or S3 as needed.

On EKS/GKE/AKS, the exact node log path may differ (e.g. /var/log/pods); use the path your runtime uses and mount it into the daemon.

---

Example 3: Sidecar sending to the daemon

The sidecar must know where the daemon is. Options:

1. Host network (daemon): DaemonSet runs with hostNetwork: true and listens on a fixed port (e.g. 24224). Sidecar uses status.hostIP (from downward API) and that port to reach the daemon on the same node.
2. Node-local Service: A Service with externalTrafficPolicy: Local or a DaemonSet that exposes a Service so that traffic to the “daemon” goes to the pod on the same node. Sidecar uses that Service name and port.

Minimal idea: in the sidecar’s Fluent Bit output, set:

[OUTPUT]
    Name          forward
    Match         *
    Host          ${FLUENT_BIT_DAEMON_HOST}
    Port          24224

and pass FLUENT_BIT_DAEMON_HOST from status.hostIP (as in Example 1). The daemon’s Fluent Bit would have an INPUT forward listening on 24224 to receive from sidecars.

---

When to use which

Scenario	Use
App logs to stdout/stderr only	Log daemon only. No sidecar.
App logs to files only	Sidecar + daemon (or sidecar → backend if you prefer not to run a daemon).
Mix of stdout and file logs	Daemon for stdout; sidecar for file logs; both can send to the same backend.
Minimal resource use, simple stack	Daemon only, and standardize on stdout.
Per-app parsing or routing	Sidecar that parses and forwards to daemon or backend.

---

Summary

Component	Role
Application	Writes to stdout/stderr (preferred) or to files on a shared volume.
Log sidecar	Runs in the same pod; tails log files from shared volume and forwards to daemon or backend. Use when app writes to files.
Log daemon (DaemonSet)	One per node; reads container logs from the node (stdout/stderr) and/or receives from sidecars; enriches and forwards to backend.
Backend	Elasticsearch, Loki, S3, CloudWatch, etc. — one place to search and store logs.

Prefer stdout + log daemon for simplicity; add a log sidecar only when the application writes to files. The full architecture (app + sidecar + daemon + backend) gives you a single pipeline for both file-based and stdout-based logs with minimal app changes.