How to Configure Apache Kafka Cluster Uptime and Port Monitoring

Apache Kafka is the standard foundation for modern, real-time distributed event streaming architectures. It powers high-throughput data pipelines, microservices communication, and live activity tracking. However, because Kafka is a complex distributed system, broker instances can experience sudden failures due to JVM out-of-memory errors, disk partition saturation, network partition issues, or coordination desynchronization.

If a single broker goes offline, your producers will fail to publish events, causing messages to pool or drop. To catch these broker failures before message pipelines stall, you must configure automated health checks. This guide explains how to monitor your Apache Kafka brokers using Pingzo's native HTTP/HTTPS and Cron check features.

🛠️ Step 1: Monitor Kafka Using Cron Heartbeats (Push Check)

Since Kafka brokers run on dedicated ports (like 9092) and do not speak HTTP directly, you can monitor broker connection availability locally and report the status to Pingzo using our native Cron / Heartbeat monitor.

With a push check, a simple cron script runs on your broker host. It attempts to connect to port 9092 locally. If the check succeeds, it pings your unique Pingzo heartbeat URL. If the broker is down, the ping is skipped, and Pingzo alerts your team of a missed check-in.

1. Create a Cron Monitor in Pingzo

1. Log into your Pingzo Dashboard and click Create Monitor.
2. Select Cron / Heartbeat (Push) as the monitor type.
3. Name the monitor (e.g. Kafka Broker 1 Heartbeat).
4. Set the check interval to 1 minute and the grace period to 2 minutes.
5. Copy your unique Ping Secret URL (e.g. https://auth.pingzoapp.com/ping/your-unique-secret).

2. Configure the Check Script on Your Host

Create a simple bash script named check-kafka.sh on your Kafka broker server:

#!/bin/bash
# Test local connection to Kafka port 9092 with a 5-second timeout
nc -z -v -w5 localhost 9092

if [ $? -eq 0 ]; then
  # Port is open: Ping Pingzo to report healthy status
  curl -fsS -m 10 https://auth.pingzoapp.com/ping/your-unique-secret > /dev/null 2>&1
else
  echo "Kafka port 9092 is closed or unreachable."
  exit 1
fi

3. Schedule the Cron Job

Make the script executable: chmod +x check-kafka.sh

Add a crontab entry to execute the script every minute: * * * * * /path/to/check-kafka.sh

If Kafka crashes or fails to accept connections, the script will skip the ping, and Pingzo will alert you within 120 seconds.

⚡ Step 2: Set Up Kafka REST Proxy Health Checks (HTTP Status Check)

If you use the Confluent Kafka REST Proxy to publish messages, you can monitor your cluster health directly using standard HTTP checks. The REST Proxy exposes internal metadata endpoints that return a JSON list of active clusters.

A healthy REST Proxy returns an HTTP 200 status code and a JSON response containing the active clusterId.

How to configure a REST Proxy check in Pingzo:

1. In the Pingzo Dashboard, click Create Monitor and select Website / API Uptime (Pull).
2. Enter the URL of your REST Proxy metadata endpoint: https://kafka-proxy.yourdomain.com/v3/clusters
3. Set the check interval to 1 minute.

Securing and Authenticating the Check

To keep your endpoint secure while allowing Pingzo to inspect it, we recommend the following approaches:

• IP Whitelisting / Firewall Rules: Restrict incoming traffic on your /v3/clusters endpoint to only allow connections from Pingzo's monitoring nodes. This allows you to host a secure, unauthenticated check route.
• URL Basic Authentication: If your REST Proxy sits behind a reverse proxy (like Nginx or Apache) that uses basic authentication, you can pass credentials directly in the URL field inside Pingzo: https://username:password@kafka-proxy.yourdomain.com/v3/clusters

Validate the JSON Response Body

To ensure the proxy is actively communicating with the Kafka brokers:

1. Enable Keyword Match (optional) inside Pingzo.
2. Configure it to search for your unique cluster ID string (for example, cluster_id or kafka-cluster).
3. If the REST Proxy loses its connection to the broker cluster, it will fail to return the cluster metadata, causing the check to fail. This alerts you even if the REST Proxy server itself is online.

📣 Step 3: Configure Outage Alert Routing

When a broker or proxy fails, alerts must reach your team immediately.

Set up alert routing in your Alert Channels page:

• WhatsApp Alerts: Route instant down and recovery notifications to your on-call team mobile numbers.
• Slack or Discord: Send incident status reports detailing the target broker host, port number, and connection failure time.
• Webhooks: Integrate with developer automation systems to trigger instance restarts or failover routings automatically.