Critical Kafka Metrics to Monitor in Production: Consumer Lag, ISRs, and JMX

Operating large-scale Apache Kafka clusters in production requires vigilant telemetry. A cluster can appear healthy on basic CPU and memory dashboards while silently dropping consumer groups, falling hours behind SLA processing targets, or risking data loss due to degraded replication quorums.

This guide details the Four Golden Metric Signals of Apache Kafka production monitoring, their underlying JMX MBean paths, and Prometheus alert thresholds.

---

1. The Four Golden Signals of Kafka Telemetry

---

2. Metric 1: UnderReplicatedPartitions & OfflinePartitions (P0 Alarm)

When a broker dies, experiences network partitioning, or suffers disk I/O stall, follower replicas fall behind and are expelled from the In-Sync Replicas (ISR) list.

Key JMX Metrics & Thresholds:

---

3. Metric 2: Consumer Group Lag (records-lag-max)

Consumer Lag measures the delta between the highest offset committed to a partition by producers (Log End Offset - LEO) and the latest offset processed by a consumer group (Current Offset):

$\text{Consumer Lag}=\text{LEO}-\text{CommittedOffset}$

JMX MBean & Prometheus Export:

• Metric: kafka.consumer:type=consumer-fetch-manager-metrics,client-id={client-id},name=records-lag-max
• Alert Rule: If records-lag-max > 50000 OR rate(consumer_lag)[15m] > 0 (lag is monotonically increasing, meaning consumer cannot keep up with ingress rate).

---

4. Metric 3: Broker Request Handler Thread Utilization

Kafka brokers process client requests across two thread pools:

1. Network Processor Threads (num.network.threads): Read/write bytes to non-blocking NIO sockets.
2. Request Handler Threads (num.io.threads): Process Produce and Fetch requests against the local PageCache/disk.

RequestHandlerAvgIdlePercent:

• MBean: kafka.server:type=KafkaRequestHandlerPool,name=RequestHandlerAvgIdlePercent
• Health Benchmark: Should remain $>0.60$ ($60\%$ idle) under peak load.
• Alert Threshold: If RequestHandlerAvgIdlePercent < 0.20 ($<20\%$ idle), the broker's CPU is saturated handling socket requests. New connections will experience timeout spikes. Scale out broker nodes immediately.

---

5. Complete Production Prometheus Alerting Rules

---

6. Summary Checklist

• Export JMX Metrics using Prometheus JMX Exporter (jmx_prometheus_javaagent) on all broker JVM instances.
• Set P0 PagerDuty alarms on OfflinePartitionsCount > 0 and UnderReplicatedPartitions > 0.
• Deploy Burr or Kafka Exporter to monitor consumer group lag externally without burdening client application runtimes.