A Kafka consumer looks like a tight loop: read a message, process it, commit the offset. In production the loop has to be honest about every failure mode it can encounter. Most consumer bugs I have seen come from skipping one of those failure modes and then rediscovering it under load.
At-least-once is the default; design for it
Kafka consumers are at-least-once by default. The broker hands a message to a consumer, the consumer processes it, and the offset is committed. If the consumer crashes between processing and committing, the next consumer in the group reprocesses the message. This is not a corner case — it happens in every reasonably busy topic on a regular enough cadence that you have to design around it.
The fix is to make every consumer's side-effect idempotent. The pattern that has held up across the OrderX order-processing consumer and the ESOL payment IPN consumer is the same: every message carries an idempotency key derived from the business event (the order id, the IPN id, the transaction id), and every consumer writes through a unique index keyed on that key.
A duplicate delivery then becomes a no-op at the storage layer rather than a duplicate at the side-effect layer.
func handle(ctx context.Context, msg kafka.Message) error {
key := msg.Headers["idempotency-key"]
if exists, err := effects.AlreadyHappened(ctx, key); err != nil {
return err
} else if exists {
return nil // already processed, nothing to do
}
if err := runSideEffect(ctx, msg); err != nil {
return err
}
return effects.Record(ctx, key, msg)
}
The interesting line is the one that checks effects.AlreadyHappened. Without it, a crash between runSideEffect and effects.Record becomes a duplicate the next time the message is delivered.
Offset commits are not the progress marker
A common mistake is treating the committed offset as the source of truth for "what has been processed". It is not. The committed offset is the source of truth for "where this consumer group is currently reading from". The actual progress marker is the side-effect — which is why idempotency at the storage layer matters.
A consumer that commits offsets only after the side-effect succeeds is a consumer that does the right thing under crash. A consumer that commits first and processes second is a consumer that has lost messages and does not know it.
Manual offset management is worth the code it costs. Auto-commit looks convenient and is wrong for anything except the simplest consumers.
Consumer groups, partitions, and parallelism
Kafka parallelizes through partitions, not through threads. A consumer in a group gets assigned a subset of partitions and processes them in parallel only insofar as the group has multiple members. A topic with one partition is a topic with one consumer; a topic with twelve partitions is a topic with up to twelve consumers in parallel.
That means the parallelism of a consumer group is determined by the topic, not by the consumer. Designing consumers without naming the partition count is designing for a future where someone asks "why is the throughput capped".
Poison pills are a feature
Some messages will fail forever. A malformed payload, a foreign-key violation that no retry will resolve, a downstream that has hard-rejected a specific id. The right behavior is to remove the message from the consumer's working set and surface it in a place a human can look at.
The pattern that has held up: a fixed max-attempt count (three is a reasonable default) with exponential backoff and jitter, then a write to a dead-letter topic with the original payload, the headers, the failure reason, and the attempt count attached. The dead-letter topic is part of the system; the dead-letter topic is on a dashboard; the dead-letter topic is small enough that someone notices when it grows.
const maxAttempts = 3
func shouldRetry(attempt int, err error) bool {
if errors.Is(err, errForeignKeyViolation) {
return false
}
return attempt < maxAttempts
}
A consumer that retries forever on a foreign-key violation is a consumer that has hidden a bug under retry pressure.
Scaling is not the goal
The reflex when consumer lag grows is to add consumers. Sometimes that is the right answer. Usually the right answer is to look at what the consumer is actually doing. Slow consumer growth is usually a slow side-effect, not a capacity problem, and adding more consumers to a slow database makes the database slower.
Scale the consumer group only after you have measured what is actually slow. The fix for a slow consumer is usually a smaller consumer (a more targeted topic, a faster side-effect, a better index behind the side-effect) rather than a larger one.
The minimum shape
The smallest consumer that holds up in production has: a single purpose per topic, a single idempotency contract, a single offset management strategy, a defined dead-letter path, and a dashboard over consumer lag. Everything else is detail, and the detail matters — but the shape is the shape.