Setting and Calculating Service Level Objectives

Setting and Calculating Service Level Objectives

For modern engineering teams, achieving 100% website uptime is an unrealistic and economically prohibitive goal. High reliability requires significant cloud infrastructure overhead and slows down software release velocity, while low reliability drives customer churn.

To balance feature delivery speed with system stability, site reliability engineers (SREs) establish a structured framework using Service Level Indicators (SLIs), Service Level Objectives (SLOs), and Error Budgets. This guide walks you through defining SLIs, calculating monthly error budgets, configuring PromQL rules, and executing diagnostic troubleshooting when budget burns accelerate.

1. SLA vs. SLO vs. SLI: The Core Differences

To manage platform reliability, you must distinguish between contract commitments, engineering targets, and raw telemetry:

2. Mathematical Models for Reliability Planning

Platform availability is calculated by tracking successful user-facing events over a specific window:

[\text{Availability} = \frac{\text{Good Events}}{\text{Total Valid Events}} \cdot 100]

The Error Budget

The error budget represents the allowed fraction of failures before reliability targets are violated. It is defined as:

[\text{Error Budget} = 1 - \text{SLO Target}]

For a platform processing (100,000,000) API requests per month with a (99.9%) SLO target, the error budget allows up to (100,000) failed requests before deployments must be frozen to prioritize reliability tasks:

[\text{Allowed Failures} = 100,000,000 \cdot (1 - 0.999) = 100,000\text{ requests}]

Trailing Downtime Allowances

This table shows how SLO targets convert into allowed downtime across common rolling measurement windows:

3. Prometheus Metric Implementation

SRE teams use Prometheus recording rules to track availability and compute percentile latency profiles.

A. PromQL Rule for Request Success Rates

Use this recording rule configuration to calculate the ratio of successful requests (excluding server-side HTTP 5xx errors) to total valid requests:

groups:
  - name: gateway-slo-rules
    rules:
      - record: job:http_requests:availability_rate5m
        expr: >-
          sum(rate(http_requests_total{status!~"5.."}[5m]))
          /
          sum(rate(http_requests_total[5m]))

B. PromQL Rule for Latency Percentiles

To check if (95%) of your requests complete within your target latency threshold, query your histogram buckets:

histogram_quantile(
  0.95,
  sum(rate(http_request_duration_seconds_bucket[5m])) by (le)
)

4. Querying SLO Logs with SQL and Shell Scripts

During incident triaging or historical audits, you can compute availability metrics directly from raw server logs or transaction databases.

A. Command-Line Log Parser

Run awk over your Nginx access logs to quickly extract the success-to-failure ratio of raw requests:

awk '$9 ~ /^[23]/ {success++} {total++} END {if (total > 0) print "Availability Rate: " (success/total)*100 "%"; else print "No requests found"}' /var/log/nginx/access.log

B. PostgreSQL SLO Query

Query your database event table to isolate daily route compliance and calculate error budget consumption:

SELECT 
  date_trunc('day', created_at) AS transaction_day,
  count(id) AS total_requests,
  sum(CASE WHEN status_code >= 500 THEN 1 ELSE 0 END) AS failed_requests,
  (1.0 - (sum(CASE WHEN status_code >= 500 THEN 1 ELSE 0 END)::numeric / count(id))) * 100 AS availability_pct
FROM application_logs
WHERE created_at >= now() - INTERVAL '30 days'
GROUP BY 1
ORDER BY 1 DESC;

[!TIP] Tip (SLO Audit): Use the SLA Calculator to convert your target availability percentage into exact monthly downtime allowances and budget boundaries. Use these calculations to align engineering velocity blocks with actual business requirements.

5. Troubleshooting SLO and Budget Breaches

If your metrics indicate rapid error budget consumption, execute this step-by-step diagnostic workflow:

1. Decompose the error source: Check whether the budget burn is caused by a latency regression (slow TTFB) or a sudden spike in HTTP 5xx server errors.
2. Isolate the denominator: Confirm that invalid traffic (such as client-side HTTP 400 validations or bot scans) is correctly excluded from the SLI calculation to avoid false alerts.
3. Trace request execution layers: Run network verification checks to isolate transport latencies from backend execution time:

   curl -o /dev/null -s -w 'DNS: %{time_namelookup}s TCP: %{time_connect}s TLS: %{time_appconnect}s TTFB: %{time_starttransfer}s\n' https://pingzoapp.com/
   

4. Analyze database metrics: Check for connection pool exhaustion, slow query lock contentions, or high replication lag on database replicas.
5. Examine downstream dependencies: Review API timeouts and error rates from external payment gateways, authentication providers, or third-party web services.
6. Verify cache performance: Check if origin server latency rose due to a sudden drop in CDN cache-hit ratios.
7. Identify deployment correlations: Cross-reference the timestamp of the error budget burn with your CI/CD deployment timeline or configuration updates.
8. Evaluate regional variances: Segment SLI telemetry by geographic region, ISP, and browser client type to locate isolated edge network failures.
9. Initiate rollback or rate-limiting: If the budget breach is systemic, rollback recent changes or apply rate-limiting limits to protect system stability.
10. Establish warning alert rules: Define multi-window burn rate alerts (e.g. tracking both 1-hour fast burn and 6-hour medium burn trends) to detect slow degradations before they exhaust your entire 30-day budget.