Kafka

Problem

Millions of events arrive continuously. Multiple consumers need to process them independently, possibly replaying yesterday's events again. A message disappearing immediately after one consumer reads it isn't always desirable.

Solution — Kafka

Kafka stores ordered records in durable partitioned logs. Partitions allow storage and processing to be spread across servers. Ordering is guaranteed within each partition, rather than across the entire topic.

Partition:
1 → 2 → 3 → 4 → 5 → ...
      ↑         ↑
 Consumer A  Consumer B

Consumers track their own position.

Core idea: Kafka provides durable, scalable event streams that consumers can independently read and replay.