Module 05 // Async and communicationLesson 18 / 3418 min read
Event-driven architecture and streams
In an event-driven system, services announce what happened and others react. An event stream, like Kafka, keeps those events in an ordered log that many consumers can read and replay.
Think of it like this
Analogy
Think of a newspaper. The paper doesn't phone each reader when something happens; it publishes stories, and whoever cares reads them. Old issues are kept in an archive, so a new subscriber can read back through last month, and a researcher can go through the archive again with new questions. Each reader keeps their own bookmark.
An event stream is that archive, and each consumer group's bookmark is its offset.
The problem it solves
As a system grows, services start calling each other for everything. The order service calls payments, inventory, shipping, email, loyalty points and analytics. Each new feature adds another call, and if any of them is slow, ordering is slow. Adding a new consumer of order data means changing and redeploying the order service.
Meanwhile, valuable history disappears: once a queue message is processed it's gone, so you can't replay last week's events to fix a bug or fill a new database.
A real example
Example
A card payment at a shop
You tap your card. The payment service approves it and writes a PaymentApproved event to a stream, partitioned by account ID so each account's events stay in order.
Several services read that stream on their own: fraud checks look for unusual patterns, the notifications service sends a push to your phone, the ledger updates your balance, and analytics counts sales per shop. The payment service doesn't know or care who's listening.
A year later the bank launches cashback. The new rewards service subscribes to the same stream and replays the last 30 days of events to give early customers their points. No other service had to change.
How it works
In an event-driven architecture, services publish events: facts about something that happened, in the past tense. “OrderPlaced”, “PaymentCaptured”, “ItemShipped”. Interested services subscribe and react. The publisher doesn't know or care who's listening.
Event streams: the log
An event stream is an append-only log. Kafka, Kinesis, Event Hubs and similar systems store events durably, in order, for a retention period (days, weeks, or forever).
- Partitions split a topic so it can scale across machines. Events with the same key always land in the same partition, so they stay in order.
- Consumer groups: each group gets every event. Within a group, each partition is read by one member at a time, so the number of partitions caps a group's parallelism.
- Offsets: each group remembers its position per partition. It can resume after a crash, or rewind to reprocess.
Queue or stream?
| Aspect | Message queue | Event stream |
|---|---|---|
| After reading | Message is deleted | Event stays until retention ends |
| Many independent readers | Needs fan-out (pub/sub) | Built in (consumer groups) |
| Replay history | No | Yes |
| Ordering | Optional (FIFO) | Per partition |
| Typical use | Background jobs, task distribution | Event backbones, analytics, change data capture |
Patterns built on events
- Event notification: “OrderPlaced, id 42”. Consumers fetch details if they need them.
- Event-carried state transfer: the event includes the data, so consumers don't call back.
- Change data capture (CDC): stream every database change into the log, to feed caches, search indexes and warehouses without dual writes.
- Event sourcing: the log of events is the source of truth; current state is rebuilt by replaying them. Powerful for audit trails, but more complex to build.
- Sagas: a multi-step business process (reserve stock, charge card, ship) coordinated through events, with compensating events (“release stock”) when a step fails.
Designing good events
- Name them as past-tense facts, not commands.
- Include an ID, a timestamp and a schema version; use a schema registry to evolve formats safely.
- Assume consumers will see duplicates and occasionally old events, and make them idempotent.
Trade-offs
What you gain
- Loose coupling: add new consumers without touching producers.
- Replayable history for recovery, debugging and new features.
- Huge throughput with ordering per key.
- Natural fit for real-time analytics and keeping many stores in sync.
What it costs you
- Eventual consistency: consumers update at different times.
- Harder to follow the flow of one request across services; needs good tracing.
- Schema changes need discipline, since old events live on.
- Running a streaming cluster is real operational work (managed services help).
When not to use it
- A simple app with a few services, where direct calls are easier to understand.
- The caller needs an immediate answer to continue.
In the cloud
Event streaming (Kafka-style logs)Every major cloud sells this as a managed service, so you configure it rather than build it.
AWS2
- Amazon Kinesis Data Streams (opens vendor docs)
Ordered, replayable stream split into shards.
- Amazon MSK (opens vendor docs)
Managed Apache Kafka.
Azure1
- Azure Event Hubs (opens vendor docs)
High-throughput event stream with partitions; also speaks the Kafka protocol.
Google Cloud2
- Google Cloud Managed Service for Apache Kafka (opens vendor docs)
Managed Apache Kafka clusters.
- Pub/Sub (opens vendor docs)
Serverless messaging that can also replay retained messages and deliver them in order per key.
Producers append events to a partitioned log; each consumer group reads at its own position.
- Producers
- Amazon MSK topic
- Consumer groups
- S3 archive
Event routingEvery major cloud sells this as a managed service, so you configure it rather than build it.
AWS1
- Amazon EventBridge (opens vendor docs)
An event bus that routes events from AWS services, SaaS apps and your code to targets by rule.
Azure1
- Azure Event Grid (opens vendor docs)
Routes events from Azure services and your apps to handlers, by topic and filter.
Google Cloud1
- Eventarc (opens vendor docs)
Routes events from Google Cloud services and your apps to Cloud Run and other targets.
Interview questions
Try answering out loud first, then open the model answer.
When would you choose Kafka over a queue like SQS?
Model answer
When many independent services need the same events, when I need to replay history (rebuild a read model, fix a bug, backfill a new service), when I need ordering per key at high throughput, or for streaming analytics and change data capture. For simple background jobs where each task is done once and then forgotten, a queue is simpler and cheaper to run.
How do you keep a search index in sync with your database?
Model answer
Avoid having the app write to both, since one can fail and they drift apart. Use change data capture: stream the database's change log into a topic, and have an indexer consumer update the search index idempotently, using the row's primary key and version so out-of-order updates are ignored. If the index needs rebuilding, replay the topic from the start or from a snapshot.
How many partitions should a topic have?
Model answer
Enough for peak throughput and parallelism: partitions ≥ peak messages per second ÷ what one consumer can process, with headroom for growth, because adding partitions later changes which partition keys map to and can break ordering. But not thousands without reason, as each adds overhead. Choose a partition key with many distinct, evenly used values so load spreads and there's no hot partition.
Quiz
Pick an answer to see whether it’s right and why.
0 of 6 answered
Key terms
- Event-driven architecture
- Services announce things that happened (events), and other services react, instead of calling each other directly.
- Event stream (log)
- An append-only, ordered record of events that many consumers can read and replay, like Kafka.
- Partition
- One slice of a dataset or stream. Sharding splits data into partitions spread across machines.
- Consumer group
- A set of consumers that share the work of reading a stream; each partition goes to one member.
- Offset
- A consumer's position in a stream partition, so it can resume or replay from any point.
- Event sourcing
- Storing every change as an event and rebuilding the current state by replaying them.
- Pub/sub
- Publish/subscribe. A publisher sends a message to a topic and every subscriber gets its own copy.
Progress is saved in this browser only.