Docker Container & Kubernetes Node Health Monitoring for SREs

Docker Container & Kubernetes Node Health Monitoring for SREs: Metrics, Alerts, Diagnostics, and Runbooks

Running containerized workloads in production introduces layered failure modes across container runtimes, Linux kernel cgroups, kubelet daemons, and software-defined overlay networks. When a Kubernetes node becomes unresponsive or a container enters a crash loop, the operational challenge is isolating whether the root cause originates in application code, memory leaks, Linux kernel resource exhaustion, or storage subsystem saturation.

Site Reliability Engineers (SREs) establish distinct observability boundaries between application health, container lifecycle state, node infrastructure stability, and cluster control-plane responsiveness. This guide details container and node health metrics, Prometheus PromQL alert definitions, Linux kernel diagnostic workflows, and production evacuation runbooks.

1. Container & Kubernetes Node Telemetry Architecture

To prevent blind spots, telemetry pipelines must ingest data from every layer of the container orchestration stack:

┌─────────────────────────────────────────────────────────────┐
│                    Application / Workload                   │
│   (HTTP Status, Latency, Error Rates, OpenTelemetry Spans)  │
└──────────────────────────────┬──────────────────────────────┘
                               │
┌──────────────────────────────▼──────────────────────────────┐
│                    Pod & Container Boundary                 │
│      (cAdvisor, cgroups v2, Container Health Checks, OOM)   │
└──────────────────────────────┬──────────────────────────────┘
                               │
┌──────────────────────────────▼──────────────────────────────┐
│                    Node Operating System                    │
│   (kubelet, containerd / CRI, node-exporter, Linux Kernel)  │
└──────────────────────────────┬──────────────────────────────┘
                               │
┌──────────────────────────────▼──────────────────────────────┐
│                    Kubernetes Control Plane                 │
│   (kube-apiserver, kube-scheduler, kube-controller-manager) │
└─────────────────────────────────────────────────────────────┘

The telemetry pipeline flows as follows:

1. cAdvisor (embedded in kubelet) collects container-level resource statistics (CPU CFS quotas, memory working set, network interfaces, filesystem usage).
2. node-exporter collects host-level metrics (system load, memory pressure, disk I/O latency, TCP retransmissions, socket allocations).
3. kube-state-metrics translates Kubernetes API objects (node conditions, pod phases, replica counts, daemonset status) into Prometheus metrics.
4. Prometheus / VictoriaMetrics scrapes telemetry endpoints on fixed intervals and evaluates SLO alert rules.

2. Docker Container Health: What SREs Must Measure

A container exists in one of six core lifecycle states: created, running, paused, restarting, exited, or dead. Monitoring container health requires tracking both runtime state transitions and kernel-enforced resource constraints:

A. CPU Throttling & CFS Quotas

Linux cgroups enforce CPU limits using the Completely Fair Scheduler (CFS). When a container exhausts its allocated cpu.cfs_quota_us within a given cpu.cfs_period_us, the kernel throttles execution threads, introducing massive tail latency spikes without showing 100% CPU utilization.

B. Memory RSS vs. Working Set & OOM Kills

Kubernetes bases OOM (Out Of Memory) eviction decisions on the memory working set rather than Resident Set Size (RSS):

[\text{Memory Working Set} = \text{Memory Usage} - \text{Inactive File Cache}]

If the working set exceeds the cgroup memory.max or Kubernetes limits.memory, the Linux kernel OOM killer terminates the container process (exit code 137).

C. Filesystem & Inode Saturation

Unbounded logging to container stdout/stderr or ephemeral /tmp writes consume the container writable layer. If disk space or inodes reach 100%, file write calls fail immediately, corrupting runtime states.

3. Docker CLI Diagnostics & Runtime Inspection

When troubleshooting Docker container failures directly on a host, execute these diagnostic commands:

# 1. List container status, uptime, and exit codes
docker ps -a --format 'table {{.ID}}\t{{.Names}}\t{{.Status}}\t{{.State}}'

# 2. Inspect real-time CPU, memory, network I/O, and block I/O
docker stats --no-stream

# 3. Retrieve container health-check history and failure reasons
docker inspect --format '{{json .State.Health}}' <container_id> | jq .

# 4. Check OOMKilled status and container termination details
docker inspect --format 'OOMKilled={{.State.OOMKilled}} ExitCode={{.State.ExitCode}} Error={{.State.Error}}' <container_id>

# 5. Review container storage consumption across layers
docker system df -v

# 6. Stream live runtime container events
docker events --since 30m --filter type=container

# 7. Check kernel logs for out-of-memory terminations
dmesg -T | grep -Ei 'oom|killed process|out of memory'

4. Kubernetes Node Health Model & Conditions

A Kubernetes node reports its operational status through the NodeStatus API object. Node conditions represent binary health states:

Node Leases and Heartbeats

Every node maintains a Lease object in the kube-node-lease namespace. The kubelet renews this lease every (10\text{ seconds}) (node-status-update-frequency). If the control plane does not receive a renewal within (40\text{ seconds}) (node-monitor-grace-period), the node controller marks the node condition as Ready=Unknown and applies the node.kubernetes.io/unreachable taint.

# Check node conditions, taints, and allocatable capacity
kubectl describe node <node_name>

# Inspect active node leases
kubectl get lease -n kube-node-lease

# View recent cluster events sorted by timestamp
kubectl get events -A --sort-by=.lastTimestamp

5. Node Resource Saturation Threshold Matrix

SRE teams establish threshold matrices to differentiate operational warnings from critical incidents requiring node drainage:

6. Container Metrics & PromQL Query Library

Monitor container performance and throttling using Prometheus cAdvisor metrics:

Container CPU Utilization Rate

sum by (namespace, pod, container) (
  rate(container_cpu_usage_seconds_total{container!=""}[5m])
)

Container CPU Throttling Ratio (%)

100 * sum by (namespace, pod, container) (
  rate(container_cpu_cfs_throttled_periods_total{container!=""}[5m])
) / sum by (namespace, pod, container) (
  rate(container_cpu_cfs_periods_total{container!=""}[5m])
)

Container Memory Working Set vs. Memory Limit (%)

100 * (
  container_memory_working_set_bytes{container!=""}
  /
  container_spec_memory_limit_bytes{container!=""} > 0
)

Container OOM Kill Detection Rate

sum by (namespace, pod, container) (
  increase(container_oom_events_total[5m])
) > 0

7. Node-Level Metrics & PromQL Calculations

Track underlying virtual machine or bare-metal host health using node-exporter:

Host CPU Utilization (%)

Host CPU utilization measures non-idle processor capacity:

[\text{CPU Utilization} = 100 \times \left(1 - \frac{\text{Rate of Idle CPU Seconds}}{\text{Rate of Total CPU Seconds}}\right)]

100 * (1 - avg by (instance) (
  rate(node_cpu_seconds_total{mode="idle"}[5m])
))

Host Memory Availability (%)

100 * (
  node_memory_MemAvailable_bytes
  /
  node_memory_MemTotal_bytes
)

Host Root Disk Free (%)

100 * (
  node_filesystem_avail_bytes{mountpoint="/"}
  /
  node_filesystem_size_bytes{mountpoint="/"}
)

8. Linux Kernel Pressure Stall Information (PSI)

Linux kernel Pressure Stall Information (PSI) measures resource starvation before traditional saturation metrics spike:

# 1. Check CPU pressure stall metrics (some vs full)
cat /proc/pressure/cpu

# 2. Check Memory pressure stall metrics
cat /proc/pressure/memory

# 3. Check Disk I/O pressure stall metrics
cat /proc/pressure/io

# 4. Check allocated vs maximum file descriptors
cat /proc/sys/fs/file-nr

# 5. Check connection tracking table utilization
conntrack -C
cat /proc/sys/net/netfilter/nf_conntrack_max

Sample output from /proc/pressure/memory:

some avg10=2.45 avg60=1.12 avg300=0.40 total=18492042
full avg10=0.82 avg60=0.31 avg300=0.08 total=4920192

• some: Percentage of time during which at least one task was stalled on memory allocation.
• full: Percentage of time during which all non-idle tasks were simultaneously stalled on memory.

9. Kubernetes Networking & DNS Health Diagnostics

Containerized networking failures occur across virtual ethernet pairs (veth), CNI bridge interfaces, or CoreDNS resolvers:

# 1. Check socket statistics and active TCP connections
ss -s

# 2. Inspect CoreDNS pod status and logs in kube-system
kubectl -n kube-system get pods -l k8s-app=kube-dns
kubectl -n kube-system logs -l k8s-app=kube-dns --tail=100

# 3. Test in-cluster DNS resolution from an ephemeral debug pod
kubectl run dns-test --rm -it --image=busybox:1.36 --restart=Never -- nslookup kubernetes.default.svc.cluster.local

# 4. Trace packet drops on a specific node interface
tcpdump -ni eth0 'tcp[tcpflags] & (tcp-rst) != 0' -c 50

10. Kubernetes Probe Lifecycle: Startup, Liveness, and Readiness

Misconfigured container probes trigger cascading cluster outages during rolling deployments:

SRE Rule: Never include downstream dependencies (database queries, external third-party APIs) in liveness probes. If a database degrades, restarting application containers multiplies connection storms and worsens recovery time.

11. Production Prometheus Alert Rules

Deploy production Alertmanager rules to detect node degradation before user outages occur:

groups:
  - name: kubernetes_node_alerts
    rules:
      - alert: KubernetesNodeNotReady
        expr: kube_node_status_condition{condition="Ready", status="true"} == 0
        for: 5m
        labels:
          severity: critical
          tier: infrastructure
        annotations:
          summary: "Node {{ $labels.node }} is NotReady"
          description: "Node {{ $labels.node }} has been in NotReady state for more than 5 minutes."

      - alert: KubernetesNodeDiskPressure
        expr: kube_node_status_condition{condition="DiskPressure", status="true"} == 1
        for: 3m
        labels:
          severity: warning
          tier: infrastructure
        annotations:
          summary: "Node {{ $labels.node }} is under DiskPressure"
          description: "Available disk or inodes on node {{ $labels.node }} breached eviction threshold."

      - alert: ContainerHighCPUThrottling
        expr: |
          100 * sum by (namespace, pod, container) (
            rate(container_cpu_cfs_throttled_periods_total{container!=""}[5m])
          ) / sum by (namespace, pod, container) (
            rate(container_cpu_cfs_periods_total{container!=""}[5m])
          ) > 25
        for: 10m
        labels:
          severity: warning
          tier: application
        annotations:
          summary: "Container {{ $labels.container }} CPU throttled > 25%"
          description: "Pod {{ $labels.pod }} in namespace {{ $labels.namespace }} is experiencing severe CPU throttling."

12. Connecting Node Health to Service Level Objectives (SLOs)

A single node reaching 90% memory utilization does not automatically mean users are experiencing errors. SRE teams measure container health against service availability budgets:

[E = 1 - S]

Where (S) represents your availability target. For a (99.95%) SLO, the total allowed error budget (E = 0.05%) ((21.9\text{ minutes}) per month).

Calculate your infrastructure error budget before tuning alert thresholds. Use our SLA Calculator to convert target availability percentages into allowable downtime budgets, and model outage impacts with the Downtime Calculator.

13. SRE 10-Step Node Troubleshooting Runbook

When a node alert fires, execute this structured diagnostic runbook:

1. Verify whether the alert reflects an isolated node failure or a cluster-wide control plane outage.
2. Inspect node status, conditions, and active taints using kubectl describe node <node_name>.
3. Check recent system events using kubectl get events -A --sort-by=.lastTimestamp.
4. Examine kubelet and container runtime daemon logs:

   journalctl -u kubelet --since "30m ago" --no-pager
   journalctl -u containerd --since "30m ago" --no-pager
   

5. Measure kernel Pressure Stall Information (PSI) via /proc/pressure/{cpu,memory,io}.
6. Correlate container exit codes (137 = OOM, 143 = SIGTERM, 1 = Application Error).
7. Evaluate DNS resolution latency and CoreDNS error logs.
8. Cordon the unhealthy node to prevent new pod scheduling:

   kubectl cordon <node_name>
   

9. Drain non-DaemonSet workloads safely to healthy cluster nodes:

   kubectl drain <node_name> --ignore-daemonsets --delete-emptydir-data --grace-period=60 --timeout=10m
   

10. Reboot or replace the underlying compute instance if hardware or kernel corruption persists.

14. 18-Point SRE Production Monitoring Checklist

Maintain this operational checklist across every container cluster:

• Node Ready condition and kubelet lease renewals monitored with < 5m alert triggers.
• MemoryPressure, DiskPressure, and PIDPressure node conditions alert on True.
• Container memory working set tracked against configured pod memory limits.
• Container CPU CFS throttling ratio monitored continuously via cAdvisor.
• OOM kill events (container_oom_events_total) alerted in real time.
• Host disk space and inode utilization monitored with < 85% warning thresholds.
• Linux kernel PSI metrics (/proc/pressure/memory) ingested to detect allocation stalls.
• TCP retransmission rates and dropped packet counters tracked via node-exporter.
• CoreDNS query latency and NXDOMAIN/SERVFAIL error rates alerted.
• Readiness probes configured without hard external database dependencies.
• Startup probes configured for slow-initializing workloads to prevent restart loops.
• Pod Disruption Budgets (PDB) configured for all production deployments.
• DaemonSets configured with resource requests to avoid node starvation.
• Kubelet client and server certificates monitored for expiration (< 30 days).
• Conntrack table utilization monitored against /proc/sys/net/netfilter/nf_conntrack_max.
• Node cordon and graceful drain runbooks automated and tested.
• Error budgets calculated using the SLA Calculator before tuning page thresholds.
• External synthetic health checks configured in Pingzo to verify end-user ingress reachability independent of internal cluster telemetry.