Linux Systemd Service Restarts, OOM Kills, and Journalctl Debugging: SRE Triage Guide

A critical backend API service, worker daemon, or database replica suddenly disappears from the network. Load balancers report HTTP 502 Bad Gateway errors, ongoing TCP sessions receive immediate connection resets (RST), and your on-call dashboard lights up.

You SSH into the host and run systemctl status myapp.service. The service appears running with a green active (running) indicator, but the Main PID has changed, the service has an uptime of only $14\text{ seconds}$, and NRestarts is climbing:

● myapp.service - Core API Daemon
     Loaded: loaded (/etc/systemd/system/myapp.service; enabled; preset: enabled)
     Active: active (running) since Tue 2026-09-15 04:12:02 UTC; 14s ago
   Main PID: 48291 (myapp)
      Tasks: 18 (limit: 4915)
     Memory: 1.4G (max: 2.0G)
        CPU: 12.402s
     CGroup: /system.slice/myapp.service
             └─48291 /usr/local/bin/myapp --config /etc/myapp/config.yaml

Sep 15 04:11:48 node-04 systemd[1]: myapp.service: Main process exited, code=killed, status=9/KILL
Sep 15 04:11:48 node-04 systemd[1]: myapp.service: Failed with result 'oom-kill'.

Why did the process die silently without generating an application stack trace?

When a process exceeds its Linux Control Group memory limit (MemoryMax=) or the host experiences global physical memory exhaustion, the Linux kernel's Out-Of-Memory (OOM) Killer terminates the process with an uncatchable SIGKILL (signal 9). Because SIGKILL bypasses all application-level exception handlers and graceful shutdown routines, systemd catches the exit code (137) and immediately restarts the service—triggering an endless crash-restart storm.

In this principal SRE guide, we dissect the systemd service lifecycle, decode cgroup v2 memory telemetry (memory.events and PSI pressure), query journalctl forensic pipelines, and configure resilient restart-rate limits.

1. The Linux Service Failure Model: systemd, cgroups, and PID 1

To diagnose unexpected service restarts, you must understand how systemd (PID 1) supervises processes and tracks failures through the Linux cgroups hierarchy.

                 THE SYSTEMD & CGROUP SUPERVISION LIFECYCLE

   systemd (PID 1)
   ┌─────────────────────────────────────────────────────────────┐
   │ 1. Spawns Worker Process via fork() -> execve()             │
   │ 2. Places PID into dedicated cgroup v2 slice:               │
   │    /sys/fs/cgroup/system.slice/myapp.service/               │
   │ 3. Enforces Memory Boundaries (MemoryMax=2G, MemoryHigh=1.8G)│
   └──────────────────────────────┬──────────────────────────────┘
                                  │ (Process Runs and Allocates RAM)
                                  ▼
   Kernel Memory Management Subsystem
   ┌─────────────────────────────────────────────────────────────┐
   │ - Anonymous RSS + Page Cache exceeds MemoryMax (2.0 GB)     │
   │ - Kernel reclaims page cache -> Still over limit!           │
   │ - memcg OOM Killer fires: Selects PID 48210                 │
   │ - Sends SIGKILL (Signal 9) directly to process thread       │
   └──────────────────────────────┬──────────────────────────────┘
                                  │ (Process dies immediately without cleanup)
                                  ▼
   systemd (PID 1) Event Loop
   ┌─────────────────────────────────────────────────────────────┐
   │ - Receives SIGCHLD from Kernel                              │
   │ - Records: ExecMainCode=killed, ExecMainStatus=9 (SIGKILL)  │
   │ - Evaluates Restart Policy (Restart=on-failure)             │
   │ - Starts replacement process after RestartSec interval      │
   └─────────────────────────────────────────────────────────────┘

Exit Code 137: The Mathematical Signature of SIGKILL

When inspecting systemd logs or container termination states, exit code 137 is universally reported.

In UNIX/POSIX process standards, when a process is killed by an asynchronous signal, its exit status is computed as:

$$ ext{Exit Status} = 128 + ext{Signal Number}$$

Since SIGKILL is signal number 9: $$ ext{Exit Status} = 128 + 9 = mathbf{137}$$

Similarly, if systemd terminates a misbehaving process with SIGTERM (signal 15) during a shutdown timeout, the resulting exit status is $128 + 15 = mathbf{143}$.

2. Global OOM vs Control Group (memcg) OOM

A critical SRE distinction is knowing whether a service was killed due to Host-Level RAM Exhaustion or Service-Level cgroup Quota Enforcement:

3. Deep cgroup v2 Memory Diagnostics

Modern Linux distributions (Ubuntu 22.04+, Debian 12+, RHEL 9+, Fedora, Rocky Linux) use the unified cgroup v2 hierarchy.

Instead of guessing memory usage, inspect the exact kernel telemetry tracked for your service:

# 1. Retrieve the active cgroup path for the service
CGROUP_PATH=$(systemctl show -p ControlGroup --value myapp.service)
BASE_DIR="/sys/fs/cgroup${CGROUP_PATH}"

echo "======================================================================"
echo " 1. ACTIVE CGROUP MEMORY TELEMETRY: $BASE_DIR"
echo "======================================================================"
echo "Current Usage (bytes): $(cat "$BASE_DIR/memory.current")"
echo "Peak Usage (bytes):    $(cat "$BASE_DIR/memory.peak")"
echo "Hard Max Limit (bytes):$(cat "$BASE_DIR/memory.max")"
echo "High Throttle Limit:   $(cat "$BASE_DIR/memory.high")"

echo -e "
======================================================================"
echo " 2. CGROUP OOM EVENT COUNTERS (memory.events)"
echo "======================================================================"
cat "$BASE_DIR/memory.events"

Decoding memory.events Output

low 0
high 1420
max 48
oom 12
oom_kill 12
oom_group_kill 0

• high (1420): The process exceeded MemoryHigh= 1,420 times, causing the kernel to throttle execution and reclaim page cache.
• max (48): The process hit the hard MemoryMax= boundary 48 times.
• oom (12): The kernel triggered the memory cgroup Out-Of-Memory subsystem 12 times.
• oom_kill (12): The kernel explicitly dispatched SIGKILL to terminate processes inside this service 12 times.

4. Pressure Stall Information (PSI): Detecting Memory Choke Points

Linux Pressure Stall Information (PSI) measures the real-world performance impact of resource shortages. Rather than monitoring simple percentage utilization, PSI measures the percentage of CPU wall-clock time that threads spend waiting for memory pages:

cat /sys/fs/cgroup${CGROUP_PATH}/memory.pressure

Sample Output:

some avg10=14.20 avg60=8.40 avg300=2.10 total=48920194
full avg10=4.80  avg60=1.20 avg300=0.40 total=12048910

• some: Percentage of time during which at least one thread was stalled waiting for memory reclaim or swap-in.
• full: Percentage of time during which all threads in the service were completely stalled.
• SRE Threshold: An avg10 PSI score above $10.0%$ indicates acute memory thrashing, preceding an OOM kill by seconds.

5. Master Journalctl Forensic Queries for Incidents

When troubleshooting crash-restarts, execute these targeted journalctl pipelines to reconstruct the timeline:

# 1. Isolate all kernel OOM events across the system
sudo journalctl -k -b --no-pager -g 'oom|out of memory|killed process|memory cgroup'

# 2. Inspect the precise exit reason and signal for a specific service
sudo journalctl -u myapp.service --since "1 hour ago" -o short-precise --no-pager

# 3. Query systemd unit properties and restart statistics
systemctl show myapp.service \
  -p ActiveState \
  -p SubState \
  -p MainPID \
  -p NRestarts \
  -p ExecMainCode \
  -p ExecMainStatus \
  -p Result

Correlating the Evidence Chain

A confirmed cgroup OOM incident exhibits this exact 3-point correlation:

1. journalctl -u myapp.service: Emits Failed with result 'oom-kill' and code=killed, status=9/KILL.
2. journalctl -k (Kernel Buffer): Emits Memory cgroup out of memory: Killed process 48210 (myapp) total-vm:2840MB, anon-rss:1980MB.
3. /sys/fs/cgroup/.../memory.events: oom_kill counter increments by $1$.

6. Preventing Thundering-Herd Restart Storms

If a service crashes repeatedly due to an unhandled exception or immediate OOM kill, systemd's default Restart=always policy will restart the process hundreds of times per minute.

This causes:

• CPU & Disk I/O Saturation: Rapid process initialization thrashing the CPU.
• Log Flooding: Exhausting journald disk buffers and obscuring original error records.
• Cascading Upstream Outages: Overwhelming databases with bursts of failed connection attempts.

                 THUNDERING-HERD RESTART CONTAINMENT

  Process Crashes (OOM / Error)
             │
             ▼
  systemd checks StartLimitBurst (e.g., 5 restarts in 5 minutes)
             │
      ┌──────┴───────────────────────────┐
      ▼                                  ▼
  Within Limit (< 5 restarts)       Exceeded Limit (>= 5 restarts)
      │                                  │
      ▼                                  ▼
  Wait RestartSec=10s               🛑 SYSTEMD ENFORCES BACKOFF FREEZE
  Restart Process Cleanly           Service state -> failed (Result: start-limit-hit)
                                    Alerts trigger; host CPU protected!

The Hardened Systemd Service Unit File

Apply this battle-tested unit template (/etc/systemd/system/myapp.service) to enforce memory ceilings, rate limits, and failure escalation:

[Unit]
Description=High-Throughput Production API
After=network.target network-online.target
Wants=network-online.target
# Escalate to on-call alert unit if service enters permanent failure state
OnFailure=service-alert-handler@%n.service

[Service]
Type=exec
User=appuser
Group=appuser
WorkingDirectory=/opt/myapp
ExecStart=/usr/local/bin/myapp --config /etc/myapp/config.yaml

# 1. Restart Policy & Interval Backoff
Restart=on-failure
RestartSec=10s

# 2. Restart Rate Limiting (Prevent infinite crash storms)
StartLimitIntervalSec=300s
StartLimitBurst=5

# 3. cgroup v2 Memory Hard & Soft Ceilings
MemoryAccounting=yes
MemoryHigh=1800M
MemoryMax=2048M
OOMPolicy=stop

# 4. Graceful Shutdown & Timeout Management
TimeoutStartSec=30s
TimeoutStopSec=45s
KillMode=mixed
KillSignal=SIGTERM

[Install]
WantedBy=multi-user.target

7. Mathematical Model: SRE Memory Headroom

To prevent surprise OOM kills, size service memory limits based on peak working set and allocation velocity:

$$ ext{Memory Headroom (%)} = rac{ ext{MemoryMax} - ext{Working Set Peak}}{ ext{MemoryMax}} imes 100$$

$$ ext{Time to OOM } (T_{ ext{OOM}}) = rac{ ext{MemoryMax} - ext{MemoryCurrent}}{rac{d( ext{Memory})}{dt}}$$

Practical SRE Capacity Rules:

• Steady-State Headroom: Target $ge 25%$ memory headroom during peak traffic.
• MemoryHigh Buffer: Set MemoryHigh= at approximately $80% - 85%$ of MemoryMax=. This allows the kernel to throttle allocations and aggressively flush page caches before reaching the catastrophic MemoryMax SIGKILL threshold.

8. SRE Troubleshooting & Decision Workflow

                 SYSTEMD RESTART TRIAGE DECISION TREE

  Service restarted unexpectedly (PID changed)
                    │
                    ▼
  Check: systemctl show myapp -p ExecMainStatus -p Result
                    │
         ┌──────────┴────────────────────────┐
         ▼                                   ▼
  Result = 'oom-kill' (Status = 9)     Result = 'exit-code' (Status != 0)
         │                                   │
  Check memory.events & dmesg:         Inspect Application Logs:
  - Did it hit MemoryMax (memcg)?      journalctl -u myapp.service -e
  - Did Host run out of RAM?                 │
         │                             Fix code bug, unhandled exception,
  Increase MemoryMax= or fix leak      or missing configuration file.

9. SRE Production Checklist for Systemd Services

• Explicit MemoryMax Configured: All critical daemons have cgroup memory limits preventing runaway memory leaks from destabilizing the host.
• Rate Limits Active: StartLimitIntervalSec=300s and StartLimitBurst=5 prevent continuous restart storms.
• Structured Logging: Services write to stdout/stderr with ISO-8601 timestamps, parsed natively by journald.
• Graceful Timeout Windows: TimeoutStopSec=45s gives workers sufficient time to drain connections before receiving SIGKILL.
• Dead Man's Snitch / Health Checks: Services emit heartbeat pings on regular intervals to detect silent process hangs.

Conclusion & Next Steps

A systemd service restart is not an isolated event—it is the final step in an execution chain involving kernel memory managers, cgroup controllers, and POSIX signal handlers.

By inspecting memory.events, decoding exit status 137, leveraging targeted journalctl queries, and enforcing strict restart rate limits, you can quickly diagnose root causes and keep your production daemons resilient.

Monitor Daemon Health & Endpoint Uptime with Pingzoapp

When systemd services crash or enter restart loops, external synthetic probes are the fastest way to detect user impact.

With Pingzoapp, you get:

• Instant Heartbeat & Daemon Monitoring: Configure reverse ping checks for background workers and systemd services. If a service stops checking in, get alerted in under 30 seconds.
• SLA & Uptime Calculators: Translate restart downtime into error budget impact using our free SLA Calculator.
• Multi-Channel On-Call Alerts: Real-time escalations via WhatsApp, Telegram, SMS, Slack, and Discord when services fail.

👉 Start Monitoring Free with Pingzoapp and protect your infrastructure against silent service crashes.