What a post office can teach you about Kafka
Apache Kafka is a popular way for services to talk to each other, yet its core ideas (topics, partitions, offsets, consumer groups) are usually introduced as a list of definitions. That makes them easy to memorise and hard to really understand.
This article takes a different route. We start with a simple online shop, run into the problems every growing system meets, and solve each one with a familiar analogy: the post office. By the end, every Kafka concept will look like the natural answer to a real problem.
A brief history
Kafka was created at LinkedIn in 2010, before the company was acquired by Microsoft, and open-sourced a year later. It is written in Java and Scala. The name is a tribute to Franz Kafka: its creator, Jay Kreps, admired his work, and the system was built to be very good at writing data.
A shop with too much backend
Say we want to sell merch online. We create our own web storefront for it. Version one is simple: a frontend, a backend, a database.
At first the backend only handles logins and orders. Then the shop grows, and the backend quietly takes on payments, email, fraud detection, analytics and much more. One big service is fine on its own; you can scale it up or out. Auth, Orders, Transactions, Email, Fraud and Analytics, each deployed on its own.
The challenge is that, as the application grows, coordinating development and deployments becomes more difficult. A forgotten migration or a broken build can affect other parts of the system, especially when everything lives in a single codebase.
Rather than jumping straight to microservices, we could explain why the growing complexity and operational requirements justify this architectural shift.
Now the services have to talk, and a call to another process isn't a function call anymore. We could wire them up with REST or GraphQL, but if they call each other directly, one service going down still leaves the others stuck.
Think letters, not requests
Squint at just two services and they stop being machines. One is a sender with a letter, the other a recipient. The sender is ready at one time, the recipient at another, so they keep rescheduling, and the letter never changes hands. The real problem is availability.
The fix is older than computers: a mailbox. Drop the letter off whenever you like, pick it up whenever you like. In software, that's a message queue.
It works fine until three services need the same letter. Transactions takes it out, and Email and Analytics never see it.
Upgrade to a post office
So we open a post office. We hire clerks to keep every letter, in the order it arrived, and hand out copies. This way, anyone can read it, and anyone can read it again.
That's a topic: a log that doesn't forget.
But if all your letters are always available you need to keep track of what you’ve already seen. To know what's new for you, the clerk jots down one number, the last letter you read. That's an offset. One number per reader, instead of a copy of every message.
Every reader gets its own copy of the same packets and keeps its own offset. Here, Email hasn't read Packet 2 yet.
When it gets busy
The shop keeps growing and one clerk can't keep up. Letters pile up, shelf space runs out, the post office falls behind. So we hire more clerks, each with their own desk. Each desk is a partition, its own neatly ordered pile.
The catch: order of messages holds within a desk, not across desks. Clerks can’t always give out letters in the order they came in, some clerk will just be faster then others. "Order paid" could get processed before "order created" if they end up at different desks - not ideal.
So every letter gets a label, a key such as the order ID, and the same key always lands on the same desk. The trick is simple: hash the key, divide by the number of desks, and the remainder is the desk number. No need to check what's already on each desk. Everything about one order stays in line, and letters without a key just go to whichever clerk has been bored the longest.
Then the Transactions service itself can't keep up. So we run copies of it, and each copy takes its own desk. But Email wants the same letters too, and two Transactions copies must never read the same desk, or orders get paid twice. Twice the money sounds nice, but it's bad for your reputation. So readers are grouped by purpose into consumer groups: inside a group the desks are split up, and each group sees every letter. When readers come and go, Kafka rebalances the desks between them.
The Transactions group has one reader per partition; the Email group splits the same three partitions between two readers.
Last problem: a building can burn down. So we run several post offices and copy all the mail between them. Every desk has a regular version and backup copies. New letters always go to the regular one, the leader partition, and its clerk copies them to the backups, the follower partitions. You can read from any copy, even the one that's geographically closest. All you decide is how many copies to keep when you create the partition.
Kafka calls each post office a broker: one running Kafka process, wherever it lives. The whole set of brokers is a broker cluster.
.jpg?table=block&id=3f9a1c4a-c8fb-80d7-85fd-d765e7e8a74d&cache=v2)
One small confession: the head clerk who decides which desk a letter goes to doesn't actually work at the post office. In Kafka, that choice is made by the client inside the sending service.
That's Kafka
Key takeaways
- Topics keep every message, so many services can read the same data and read it again later
- Offsets track how far each reader has got, using a single number instead of copies
- Partitions split a topic so the load is shared, and keys keep related messages in order
- Consumer groups let several instances share the work without processing a message twice
- Brokers and replicated partitions keep the system running when a machine fails
Where to go next
If you want to try Kafka in practice, there are two main routes. You can run it yourself (modern versions use the built-in KRaft mode instead of Apache ZooKeeper), or use a managed service such as Confluent Cloud, Amazon MSK or Aiven. It is also worth knowing the Kafka-compatible alternatives, Redpanda, WarpStream and AutoMQ, in case they fit your needs better.
The next time you see a Kafka architecture diagram, try reading it as a post office: desks, a carefully ordered archive of letters, and a clerk keeping track of who has read what. The terminology will feel far less abstract.
