How to Monitor Shopify Webhooks in Production: SRE Reliability Guide

How to Monitor Shopify Webhooks in Production: SRE Reliability Guide

For e-commerce brands, ERP integrations, and 3PL fulfillment networks, Shopify webhooks are the core event mechanism that triggers warehouse packing, inventory syncs, and customer notifications. However, Shopify enforces a strict reliability policy: if an endpoint fails to return an HTTP 200 OK within 5 seconds for 19 consecutive delivery attempts over 48 hours, Shopify automatically deletes the webhook subscription.

When a webhook is automatically deleted, incoming orders stop syncing completely without warning. Site Reliability Engineers prevent order fulfillment drops by monitoring webhook endpoints with high-resolution synthetic checks, verifying X-Shopify-Hmac-Sha256 signatures, and offloading heavy tasks to asynchronous job queues. This guide details Shopify webhook mechanics, signature verification, and automated troubleshooting runbooks.

1. Shopify Webhook Resilience and Deletion Risk Mathematics

Evaluate e-commerce fulfillment pipeline reliability using the Webhook Fulfillment Ratio:

[R_{\text{fulfillment}} = \frac{\text{Orders Acknowledged and Enqueued}}{\text{Total Dispatched Shopify Webhooks}} \times 100]

Shopify's retry system operates on an exponential backoff schedule:

If consecutive failures reach 19:

[\text{ConsecutiveFailures} \ge 19 \implies \text{Webhook Auto-Deleted by Shopify}]

Once deleted, merchants experience silent order drops until manual intervention recreates the subscription via the Shopify Admin API.

2. Critical Shopify Webhook Topics & Risk Matrix

Prioritize engineering monitoring across core store event topics:

3. SRE Shopify Webhook Threshold Matrix

Establish operational boundaries to protect your receiving infrastructure from getting blocked:

4. Test Shopify Webhook Payloads (Zero Setup)

Before deploying updates to order webhooks, inspect raw headers and verify HMAC hash construction.

You can capture and inspect live Shopify webhook events using our free cloud debugger without registration:

👉 Launch Free Webhook Tester & Payload Debugger

1. Open the Webhook Tester to obtain a temporary HTTPS catcher endpoint.
2. In Shopify Admin, go to Settings (\rightarrow) Notifications (\rightarrow) Webhooks (\rightarrow) Create Webhook.
3. Select Order creation (JSON) and paste your tester URL.
4. Click Send Test Notification.
5. Inspect the X-Shopify-Hmac-Sha256, X-Shopify-Topic, and X-Shopify-Shop-Domain headers in real-time.

5. Production Node.js HMAC Signature Verification

Shopify signs payloads with HMAC-SHA256 using your App Secret Key, encoding the result in Base64 in the X-Shopify-Hmac-Sha256 header:

import crypto from "crypto";
import { Request, Response } from "express";

export function verifyShopifyWebhook(req: Request, res: Response) {
  const hmacHeader = req.headers["x-shopify-hmac-sha256"] as string;
  const topic = req.headers["x-shopify-topic"] as string;
  const shop = req.headers["x-shopify-shop-domain"] as string;

  // req.rawBody must be the unparsed raw Buffer string
  const generatedHmac = crypto
    .createHmac("sha256", process.env.SHOPIFY_API_SECRET!)
    .update(req.body, "utf8")
    .digest("base64");

  // Use timing-safe comparison to prevent timing attacks
  const isMatch = crypto.timingSafeEqual(
    Buffer.from(generatedHmac),
    Buffer.from(hmacHeader || "")
  );

  if (!isMatch) {
    console.error(`[Shopify Webhook] Invalid HMAC signature from ${shop}`);
    return res.status(401).send("Unauthorized Webhook Signature");
  }

  // 1. Return 200 OK immediately to satisfy Shopify's 5s deadline
  res.status(200).send("Webhook Received");

  // 2. Offload order processing to background queue
  if (topic === "orders/paid") {
    const orderData = JSON.parse(req.body.toString());
    enqueueOrderFulfillment({
      orderId: orderData.id,
      totalPrice: orderData.total_price,
      lineItems: orderData.line_items,
    });
  }
}

Simulate incoming Shopify webhooks locally:

# Calculate mock HMAC signature and dispatch via cURL
PAYLOAD='{"id":1029384756,"email":"customer@example.com","total_price":"89.99"}'
SECRET="your_shopify_secret_key"
HMAC_SIG=$(echo -n "$PAYLOAD" | openssl dgst -sha256 -hmac "$SECRET" -binary | base64)

curl -X POST http://localhost:3000/api/webhooks/shopify \
  -H "Content-Type: application/json" \
  -H "X-Shopify-Topic: orders/create" \
  -H "X-Shopify-Shop-Domain: store.myshopify.com" \
  -H "X-Shopify-Hmac-Sha256: $HMAC_SIG" \
  -d "$PAYLOAD"

[!TIP] Production SRE Tools: Calculate order pipeline downtime allowances with our SLA Calculator. If Shopify webhook delivery servers experience DNS resolution stalls, inspect zone records with the DNS Lookup tool.

6. Implementing Heartbeat Alerts for Shopify Order Sync

To catch silent order sync stoppages before warehouse teams notice delays, configure an hourly reconciliation worker connected to Pingzo Heartbeats:

┌─────────────────────────────────────────────────────────┐
│        Hourly Shopify Order Reconciliation Job          │
│ 1. Fetch orders created in Shopify in the last 60 mins │
│ 2. Check each order exists in ERP / Warehouse DB        │
│ 3. If 100% matched: Dispatch Heartbeat Ping to Pingzo   │
└───────────────────────────┬─────────────────────────────┘
                            │ (POST /api/ping/pb_xxx)
                            ▼
                ┌───────────────────────┐
                │   Pingzo Heartbeat    │
                │   Monitoring Engine   │
                └───────────┬───────────┘
                            │ (If sync worker fails)
                            ▼
                ┌───────────────────────┐
                │ Instant Alert:        │
                │ WhatsApp & Telegram   │
                └───────────────────────┘

If your worker detects missing orders or an API token expiration, it suppresses the heartbeat ping. Pingzo immediately alerts your on-call engineers via WhatsApp, Telegram, or Slack, giving you immediate visibility before customers complain about delayed shipments.

7. Troubleshooting Shopify Webhook Ingestion Outages

Follow this structured runbook when Shopify webhook delivery alerts trigger:

1. Check Shopify Partner / Admin logs: Navigate to Settings (\rightarrow) Notifications (\rightarrow) Webhooks to verify if the webhook subscription is active or marked Failing.
2. Verify endpoint response latency: Ensure your server returns an HTTP 200 OK within (500\text{ ms}) and never executes synchronous database queries inside the request handler.
3. Confirm HMAC secret consistency: Verify that SHOPIFY_API_SECRET matches your live Shopify app credentials.
4. Check for webhook deletion: If an endpoint was down for (> 48\text{ hours}), query the Shopify REST Admin API GET /admin/api/2024-04/webhooks.json to check if the subscription was automatically deleted.
5. Re-subscribe deleted webhooks: Execute an automated script to recreate deleted webhook endpoints via the Admin API:

   curl -X POST "https://store.myshopify.com/admin/api/2024-04/webhooks.json" \
     -H "X-Shopify-Access-Token: $SHOPIFY_ACCESS_TOKEN" \
     -H "Content-Type: application/json" \
     -d '{"webhook":{"topic":"orders/paid","address":"https://api.yourdomain.com/webhooks/shopify","format":"json"}}'
   

6. Validate recovery: Send a test notification from Shopify Admin and confirm that the Pingzo heartbeat transitions back to Healthy (Green).