Kafka in production: duplicates, skew and backpressure

Three of Kafka’s defaults are individually sensible and collectively responsible for most of the operational trouble I’ve seen with it:

1. At-least-once delivery. Messages arrive at least once. Not exactly once.
2. Deterministic partitioning. The same key always lands on the same partition — including when that key becomes 80% of your traffic.
3. No backpressure signal. Kafka is a log, not a queue. A slow consumer produces lag, not pushback.

None of these is a bug. Each is a deliberate trade that buys you something important. But all three are silent until they aren’t, and the failure modes look nothing like the config that caused them.

At-least-once means duplicates

The mechanism is worth being precise about, because “just set acks=all” is the common wrong answer.

A producer sends a message. The broker writes it and sends an ack. The ack is lost on the way back — an ordinary network blip. The producer now has no evidence the write succeeded, so it retries. The broker writes the same message a second time. The consumer processes both.

acks=all does not help. It guarantees the message is on all replicas before the broker acks, which is a different problem. If the ack itself is lost, the retry still happens. The gap sits one layer below the setting.

This matters for any operation that isn’t idempotent — moving money, incrementing a counter, sending an email. SET balance = 451 is idempotent. SET balance = balance - 49 is not.

Producer idempotence is one flag

Since 0.11, Kafka solves the broker-side half with sequence numbers. Each producer gets a producer ID on startup, each message gets a monotonic sequence number per partition, and the broker tracks the last sequence it saw per producer. A repeat is discarded.

enable.idempotence=true

That’s the entire application-side cost. Kafka handles the ID assignment, sequence tracking and deduplication internally. Setting it also implies acks=all, retries=Integer.MAX_VALUE and max.in.flight.requests.per.connection=5 — the safe configuration, chosen for you.

It doesn’t cover the consumer

Producer idempotence guarantees the message appears once in the partition. It says nothing about how many times the consumer processes it.

Consider a consumer that reads message X, processes it, and dies before committing the offset. Kafka has no idea any work happened. The consumer restarts, reads from the last committed offset, and processes X again. The message exists exactly once and was handled twice.

So the consumer side is application code, and no framework takes it off you. Each message carries an idempotency key identifying the logical operation, and the consumer checks it before doing the work:

async function processPayment(message: KafkaMessage) {
  const { idempotencyKey, amount, accountId } = message.value;

  const exists = await redis.get(`idem:${idempotencyKey}`);
  if (exists) return;

  await db.query("UPDATE accounts SET balance = balance - $1 WHERE id = $2", [
    amount,
    accountId,
  ]);

  await redis.set(`idem:${idempotencyKey}`, "1", "EX", 86400);
}

The key has to come from the producer, not the consumer and not Kafka, because only the producer knows which logical operation it represents. A UUID per request, an order ID, ${userId}:${timestamp}:${action} — anything that identifies the operation uniquely.

That code has a gap. Between the Redis check and the database write, two consumer instances can both check, both miss, and both perform the write. Close it by making the check and the write one transaction:

async function processPayment(message: KafkaMessage) {
  const { idempotencyKey, amount, accountId } = message.value;

  await db.transaction(async (tx) => {
    const { rowCount } = await tx.query(
      `INSERT INTO processed_keys (key, processed_at)
       VALUES ($1, NOW())
       ON CONFLICT (key) DO NOTHING`,
      [idempotencyKey],
    );

    if (rowCount === 0) return; // already handled

    await tx.query("UPDATE accounts SET balance = balance - $1 WHERE id = $2", [
      amount,
      accountId,
    ]);
  });
}

ON CONFLICT DO NOTHING in the same transaction as the update. No gap.

That processed_keys table grows forever, so it needs pruning — anything older than the maximum lag your consumer group can accumulate is safe to drop.

Kafka also offers transactions (transactional.id on the producer, read_committed on the consumer), which give you exactly-once within Kafka: the produce and the offset commit become atomic. That’s genuinely useful for read-process-write stream topologies. It does nothing for an external database. If someone tells you Kafka gives you exactly-once out of the box, ask whether they’ve used it to update Postgres.

Deterministic partitioning means skew

With no key, Kafka distributes round-robin. With a key, hash(key) % numPartitions picks the partition, which guarantees that the same key always goes to the same place.

That guarantee is the whole point — it’s what makes per-key ordering possible. It’s also why a key that gets hot stays hot. Nothing in the hash function knows anything about load distribution. There’s no mechanism that notices one key is taking 40% of traffic and spreads it out.

The unpleasant part is that you usually don’t find out in testing, because test data has evenly distributed IDs. Nobody simulates the one account that suddenly gets popular.

Start normal traffic, then trigger the viral event and watch what happens to a single partition — then turn on key salting and try again:

The cluster has many times the capacity it needs and a consumer still falls behind, because the load isn’t spread.

Three ways out

Key salting. Use userId-{random(0, N)} instead of userId, spreading one user across N partitions. You lose per-user ordering, which rules it out if you need it — but for analytics, notifications or log aggregation, per-user ordering usually isn’t required. You also don’t have to salt everything; an allowlist of the top few keys is often enough, identified from per-partition consumer lag.

Compound keys. Instead of a random salt, use a second dimension that means something: userId-orderId, userId-sessionId. You keep ordering within the sub-entity — all events for one order stay together — while distributing the user across partitions. This is the option I reach for first when the domain has a natural sub-key.

const partitionKey = `${userId}-${orderId}`;

Dedicated hot-key routing. Detect hot keys at runtime and route them to a separate topic with its own partition count and consumer group. More machinery, full control. Worth it when a small number of keys are predictably enormous.

Catching it before it hurts

Three metrics, in order of usefulness:

• Consumer lag per partition, not aggregated across the group. Aggregate lag hides exactly the situation you’re looking for. In Prometheus that’s kafka_consumergroup_lag with the partition label kept.
• Throughput skew — messages/s across the partitions of a topic. What counts as “too skewed” depends on your headroom, so watch the trend rather than a fixed ratio.
• Key cardinality. If a large share of messages come from a handful of keys, one of them going viral is a matter of time. Sampling is enough.

This isn’t Kafka-specific. DynamoDB has the same property with partition keys, and Redis Cluster has it with hot keys on a single shard. The symptom is always the same: one partition saturated, the rest idle.

My actual opinion here: most teams treat the partition key as an afterthought — “use the user ID, it’s fine” — and for ordinary traffic it is fine. But the partition key is a load-balancing decision. It determines how your system distributes under extreme load, and extreme load rarely means “uniformly 10× more.” It means one key suddenly accounts for most of it.

No backpressure signal

If your consumer handles less than your producer sends, nothing happens at first. Lag grows, but the system runs. Then the heap fills. Kafka notices the consumer has stopped polling and triggers a rebalance. The consumer stops, partitions get reassigned, and the producer keeps producing throughout — so lag gets worse. The consumer comes back to a bigger backlog, fills memory faster, and triggers the next rebalance.

The nasty part is that the rebalance, which exists to help, accelerates the failure.

Kafka has no built-in signal from consumer to producer. That’s by design: it’s a log, not a queue. max.poll.records limits how many records a single poll returns, which caps memory per batch at the cost of throughput. It’s a bandage.

Three real options, each trading something different:

For drop-oldest, put a bounded ring buffer between poll() and your processing:

BlockingQueue<ConsumerRecord> buffer = new ArrayBlockingQueue<>(5000);

while (true) {
    var records = consumer.poll(Duration.ofMillis(100));
    for (var record : records) {
        if (!buffer.offer(record)) {
            buffer.poll();          // drop oldest
            buffer.offer(record);
            droppedCounter.increment();
        }
    }
}

Offset management gets subtle here: if you drop messages you must not commit their offsets, or they’re gone for good rather than merely delayed. Commit the offset of the last message you actually processed.

Rate limiting is the cleanest and the most work — consumer lag as the signal, producer rate as the control:

async function produceWithBackpressure(messages: Message[]) {
  const lag = await getConsumerLag("my-group", "my-topic");

  let rate = BASE_RATE;
  if (lag > 10_000) rate = Math.max(rate * 0.5, MIN_RATE);
  if (lag > 50_000) rate = MIN_RATE;
  if (lag < 1_000) rate = Math.min(rate * 1.2, MAX_RATE);

  await rateLimiter.acquire(rate);
  await producer.send({ topic: "my-topic", messages });
}

That only works when you own the producer. For webhooks or IoT devices you don’t have the option, and drop-oldest or a durable buffer is what’s left.

The common response to all of this is to scale horizontally once it hurts — more consumer instances, more partitions. That works until it doesn’t, and it buys time rather than fixing anything: adding consumers does nothing at all for a hot partition, since one partition is still read by exactly one consumer in the group.

Track consumer lag per partition from day one. Decide which backpressure strategy fits before you need it. And set enable.idempotence=true now, because it costs one line and the failure it prevents is the kind you find out about from a customer.