Message Queues & Streaming (Kafka)

A message queue decouples the system that produces work from the system that does it: the producer drops a message in and moves on, and consumers process messages at their own pace. That one idea is the standard answer to two interview problems — absorbing traffic spikes, and surviving a slow or crashed downstream service.

Kafka is the log-based version: messages are appended to a durable, ordered log split into partitions, and each consumer tracks its position with an offset — a bookmark saying "I have processed everything up to here."

Watch the animation in two phases. First, a firehose of producer dots slams into the log far faster than the consumers drain it. The queue-depth bar rises — that is the buffer doing its job. Nothing is dropped; the burst is simply absorbed and worked off steadily.

Then a consumer is killed. Its partition's messages pile up while the others keep draining. When a replacement joins the consumer group, it rewinds to the last committed offset and catches up — which means some messages get delivered twice. That is at-least-once delivery, and it is why the follow-up question is always the same: consumers must be idempotent, meaning processing the same message twice has the same effect as once.