Why your Kafka consumer lags

The first time a lag alert fires, the instinct is to add consumers. Sometimes that works. More often it does nothing, because lag is a symptom with half a dozen causes, and only one of them is "not enough consumers."

This post is about reading lag properly: what the number means, the causes I check in order, and the case most dashboards get wrong — lag that is completely healthy.

What lag actually measures

For every partition, Kafka tracks two offsets that matter here:

• the log end offset — the position of the newest message written to the partition
• the committed offset — the position your consumer group has confirmed it processed

Lag is the difference, per partition. A topic with twelve partitions has twelve lag numbers, and the single number on your dashboard is usually their sum.

That sum hides almost everything useful. A total lag of 50,000 could be 4,000 on every partition — the whole group is slow — or 48,000 on one partition and nearly zero on the rest, which is a completely different problem with a completely different fix.

So the first rule: always look at lag per partition before you look at it in total.

You can get it from the CLI:

kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --describe --group email-sender

Or from inside a Node service with kafkajs, which is useful for exporting it as a metric:

import { Kafka } from "kafkajs";
 
const kafka = new Kafka({ brokers: ["localhost:9092"] });
const admin = kafka.admin();
 
export async function partitionLag(groupId: string, topic: string) {
  await admin.connect();
  const [latest, [committed]] = await Promise.all([
    admin.fetchTopicOffsets(topic),
    admin.fetchOffsets({ groupId, topics: [topic] }),
  ]);
  await admin.disconnect();
 
  return latest.map(({ partition, high }) => {
    const done = committed.partitions.find((p) => p.partition === partition);
    // A group that has never committed reports offset -1.
    const offset = Math.max(Number(done?.offset ?? 0), 0);
    return { partition, lag: Number(high) - offset };
  });
}

Cause 1: one slow message at a time

The default way to write a kafkajs consumer processes one message, awaits it, then takes the next:

await consumer.run({
  eachMessage: async ({ message }) => {
    await sendEmail(JSON.parse(message.value!.toString()));
  },
});

That is correct and safe — ordering within a partition is preserved — but it means a partition can never move faster than one call's latency allows. If each message spends most of its time waiting on a network call, you are paying that full wait, one message after another.

Before adding consumers, check how many partitions each consumer is working on at once. kafkajs processes partitions one at a time by default; partitionsConsumedConcurrently lets a single consumer work several partitions in parallel while keeping order within each one:

await consumer.run({
  partitionsConsumedConcurrently: 4,
  eachMessage: async ({ message }) => { /* ... */ },
});

This is often the cheapest fix available, and it doesn't touch your deployment at all.

Cause 2: more consumers than partitions

A partition is consumed by exactly one member of a group. If a topic has six partitions and you run eight consumers, two of them sit idle — permanently. Scaling the deployment from eight to sixteen changes nothing.

The partition count is your ceiling on parallelism, so choose it with growth in mind. Adding partitions later is possible, but it changes which partition each key maps to, which breaks any ordering you relied on keys for.

Cause 3: one hot partition

If the per-partition view shows lag concentrated on one or two partitions, the problem is the key. Messages with the same key always land on the same partition, so if one customer, tenant, or account produces most of the traffic, one partition does most of the work while the others idle.

Adding consumers can't help — that partition still has exactly one reader. The options are all on the producer side:

• A more even key. If you keyed by customer only to keep a customer's events ordered, ask whether you need ordering per customer or only per entity — per order, per email, per campaign. The narrower key usually spreads far better.
• A composite key for the heaviest producers, such as the customer plus a bucket number, accepting that ordering now holds within each bucket instead of across the whole customer.

Cause 4: the group keeps rebalancing

A rebalance pauses consumption across the group while partitions are reassigned. If a consumer takes longer than the group allows between heartbeats, the coordinator assumes it has died, removes it, and triggers a rebalance. When it comes back, another rebalance. The group spends its time reshuffling instead of consuming, and lag climbs on every partition at once.

The signature is easy to spot: lag rising evenly across all partitions, and logs full of rebalance and rejoin messages. The usual culprits are processing work that blocks for a long time between heartbeats, or a sessionTimeout set tighter than your slowest batch needs. With eachBatch, call heartbeat() as you go through a large batch so the coordinator knows you are alive.

Cause 5: the consumer is waiting on something else

Sometimes the consumer is fast and the partitions are balanced, and it is still slow — because what it calls is slow. A database under load, a third-party API applying rate limits, a downstream service having a bad day.

Lag is where that shows up first, but the fix is never in the consumer. Measure the time spent inside the handler separately from the time spent waiting for messages; if nearly all of it is inside the handler, look downstream.

Lag that is supposed to be there

This is the part most alerting gets wrong.

On an email platform I work on, sending runs through Kafka with one consumer group per email provider, and each provider enforces its own rate limit. When a large campaign lands, the topic fills far faster than any provider will accept mail. Lag climbs — and that is the system working exactly as designed. Kafka is the buffer. The consumers drain it at the speed each provider allows, and nothing is dropped.

When a provider pushes back, the consumer stops pulling from that partition for a while instead of hammering the provider into a harder throttle:

await consumer.run({
  eachMessage: async ({ message, pause }) => {
    const result = await provider.send(parse(message));
    if (result.throttled) {
      // Stop fetching from this partition, then resume after the provider's
      // retry window. Throwing makes kafkajs redeliver this same message.
      const resume = pause();
      setTimeout(resume, result.retryAfterMs);
      throw new Error("provider throttled — retrying after backoff");
    }
  },
});

An alert on "lag above N" fires every time this happens, and it trains people to ignore it. What actually matters is whether lag drains:

• Healthy: lag spikes when a burst arrives, then falls steadily back toward zero.
• Unhealthy: lag keeps rising while input is steady, or never returns to baseline.

The better alert is on the trend — lag growing continuously over a window — or on time lag: how old is the oldest unprocessed message? Someone waiting on an email cares about minutes, not about offsets.

The order I check things in

1. Lag per partition — even or concentrated?
2. If concentrated: a hot key. Fix the producer's key.
3. If even: are there idle consumers? Compare consumer count to partition count.
4. Rebalance messages in the logs? Look at session timeouts and long blocking work.
5. Time inside the handler — is it the consumer, or what the consumer is waiting on?
6. Only then: more partitions, more consumers, or more concurrency per consumer.

Most lag I've seen resolves somewhere in steps 1 to 5. Scaling out is the last step, not the first — and when lag is the buffer doing its job, the right fix is often a better alert rather than any change to the consumer at all.