How to Monitor Payment Webhooks in Production

Webhooks are the structural support beams of modern e-commerce and SaaS platforms. When a customer purchases a subscription, signs up for a trial, or triggers a billing event on a checkout platform like Stripe or Lemon Squeezy, that platform registers the event and fires an asynchronous HTTP POST request (a webhook) to your application server.

Your receiver endpoint intercepts the payload, verifies its authenticity, and performs critical database writes to provision user accounts, sync subscription statuses, and dispatch transactional onboarding emails.

If this receiver goes offline, takes too long to respond, or fails to validate incoming packets, your backend database drifts out of sync with your true financial records. This leads to immediate customer friction, failed account provisioning, support ticket overload, and eventual revenue churn.

This guide outlines a production-grade strategy to monitor, test, and protect your payment webhooks.

Why Webhooks Fail in Production

Unlike standard REST API calls initiated by your client applications, webhooks are triggered externally by third-party systems. This makes debugging them harder because you do not control the client-side lifecycle. In production, webhook receivers commonly fail due to these structural issues:

1. Application Server and Gateway Downtime

During high-traffic events or code deployments, your application servers can experience temporary outages. If your server is restarting or experiencing memory saturation, incoming webhook requests are met with gateway errors (like HTTP 502 Bad Gateway or 504 Gateway Timeout). Learn how this fits into overall reliability in our guide, What is API Monitoring?.

2. Database Connection Exhaustion and Deadlocks

Webhook endpoints execute write operations to update user roles or transaction ledgers. During a launch or automated billing cycle, hundreds of webhooks can hit your receiver simultaneously. If your database connection pool is saturated, or if concurrent queries lock the same user row, the database throws a timeout or deadlock error. This returns a 500 Internal Server Error to the payment provider.

3. Strict Provider Connection Timeouts

Payment gateways do not wait indefinitely for your server to respond. They enforce strict HTTP timeout limits:

• Stripe expects a response within 3 seconds (and does not follow redirects).
• Lemon Squeezy enforces a timeout of 10 seconds before severing the socket connection.

If your webhook handler performs synchronous, heavy computations (like generating a PDF invoice or calling a third-party CRM API) before returning an HTTP response, it will exceed these limits. The gateway will register the attempt as a failure and initiate retry protocols.

4. SSL/TLS Negotiation Failures

Payment processors require secure HTTPS connections. If your SSL certificate expires, or if your server is configured with legacy TLS protocols (like TLS 1.0 or 1.1) that the processor has deprecated for PCI compliance, the handshake fails silently. The payment platform will block the connection attempt before your code is ever reached.

Implementing Webhook Verification in Node.js

To prevent malicious users from spoofing payment webhooks and forcing free account provisioning, your endpoint must verify the signature of every incoming payload. Both Stripe and Lemon Squeezy sign their requests using an HMAC-SHA256 signature generated with a secret key shared only between you and the payment provider.

Here is a complete, production-ready Node.js implementation using Express and the native crypto module to verify a Lemon Squeezy webhook signature:

const express = require('express');
const crypto = require('crypto');

const app = express();
// Lemon Squeezy webhooks require the raw body buffer to verify signatures correctly
app.use(express.json({
  verify: (req, res, buf) => {
    req.rawBody = buf;
  }
}));

const LEMON_SQUEEZY_WEBHOOK_SECRET = process.env.LEMON_SQUEEZY_WEBHOOK_SECRET;

app.post('/api/webhooks/lemon-squeezy', (req, res) => {
  const signature = req.headers['x-signature'];

  if (!signature) {
    return res.status(400).send('Missing webhook signature header.');
  }

  if (!LEMON_SQUEEZY_WEBHOOK_SECRET) {
    console.error('LEMON_SQUEEZY_WEBHOOK_SECRET environment variable is not configured.');
    return res.status(500).send('Internal server configuration error.');
  }

  // Generate the expected HMAC-SHA256 signature from the raw body buffer
  const hmac = crypto.createHmac('sha256', LEMON_SQUEEZY_WEBHOOK_SECRET);
  const digest = Buffer.from(hmac.update(req.rawBody).digest('hex'), 'utf8');
  const sigBuffer = Buffer.from(signature, 'utf8');

  // Use timingSafeEqual to prevent timing attacks
  if (sigBuffer.length !== digest.length || !crypto.timingSafeEqual(digest, sigBuffer)) {
    console.warn('Invalid webhook signature detected.');
    return res.status(401).send('Signature verification failed.');
  }

  // Webhook is authentic. Process payload asynchronously to avoid timeouts.
  const event = req.body;
  
  processWebhookEventAsync(event)
    .catch(err => console.error('Failed to process webhook event:', err));

  // Acknowledge the event instantly within the timeout limit
  res.status(200).send('Webhook received.');
});

async function processWebhookEventAsync(event) {
  const eventName = event.meta.event_name;
  console.log(`Processing event: ${eventName}`);
  
  // Perform database writes, account provisioning, etc.
  // e.g. await db.user.update(...)
}

app.listen(3000, () => console.log('Webhook receiver listening on port 3000'));

Gateway Retry Schedules: Stripe vs. Lemon Squeezy

When your webhook handler throws an error or times out, the payment gateway assumes a network or server outage has occurred. To prevent data loss, the gateway keeps the event in a queue and triggers a series of retries.

Understanding how these schedules work helps you configure alert escalations before the event is permanently dropped.

| Gateway | Initial Retry Window | Retry Strategy & Duration | Disablement Threshold | | :--- | :--- | :--- | :--- | | Stripe | Minutes | Exponential backoff over 3 days (up to 20 retries). | Multiple consecutive days of 100% failure. | | Lemon Squeezy | ~1 Hour | Attempts retries up to 4 times over a 24-hour window. | Manual webhook endpoint status disabled. |

Stripe's Retry Policy

Stripe automatically retries webhook deliveries if your server does not return a status code in the 2xx range. It retries with exponential backoff. If you are experiencing an outage that lasts several hours, Stripe will safely store the events and replay them once your server is back online. However, if an endpoint consistently fails for multiple days, Stripe will send an email warning and eventually disable the webhook registration.

Lemon Squeezy's Retry Policy

Lemon Squeezy uses a tighter retry window. If your webhook receiver endpoint fails, Lemon Squeezy will retry the event up to 4 times over a 24-hour period. Because of this shorter lifecycle, a day-long server outage can lead to permanently dropped webhook events. You must configure real-time alert routing to catch these errors immediately.

Best Practices for Webhook Monitoring

To ensure your webhooks stay healthy and functional, implement a multi-layered monitoring strategy:

1. Decouple Receipt from Processing (Queueing)

To guarantee your handler never hits connection timeouts, decouple webhook receipt from database processing. When a webhook arrives:

1. Read the raw payload and headers.
2. Verify the signature.
3. Push the payload into a message queue (like Redis, BullMQ, or QStash).
4. Immediately return a 200 OK response to the payment gateway.
5. Process the queue asynchronously using background worker threads.

This ensures your API returns a response in less than 50ms, protecting you from timeouts even when database queries run slowly.

2. Configure Uptime Probes

Regularly ping your webhook receiver endpoint using automated HTTPS checks. Ensure that it resolves correctly and does not return server errors.

• Assert response status is either 200 OK or a client-level error like 400 Bad Request or 401 Unauthorized (confirming the endpoint is active and validating signatures, but rejecting empty test requests).
• Monitor SSL certificate status. A pre-alert warning you 14 days before certificate expiry prevents sudden TLS validation outages.

3. Set Up Multi-Tier Alerting (ChatOps)

When a payment webhook fails, developers need to know immediately. Email is too slow. Route your webhook delivery failures directly to where your team is active:

• High-Priority Alerts: Send to instant, invasive mobile channels like SMS or WhatsApp if the endpoint goes down completely or begins returning 5xx errors.
• Warning Diagnostics: Route warning details (like signature mismatches or database retry triggers) to team slack, discord, or Microsoft Teams channels. To build standard templates for your channels, check our ChatOps Best Practices guide.

Checklist for Production Webhook Safety

| Check Item | Validation Method | Target Threshold | Escalation Channel | | :--- | :--- | :--- | :--- | | Endpoint Health | HTTPS GET/POST Probe | Status < 500 | Instant SMS/WhatsApp | | Response Latency | HTTP Handshake time | < 1000ms | Slack/Discord alert | | SSL Handshake | Certificate validity check | > 14 days remaining | Team email / log ticket | | Error Rate | 5xx responses on webhooks | 0% tolerance | High-priority page |