// i-ikigai · field guide

topic: field-guide.eda  ·  partition 0

Stop asking.
Start listening.

An interactive field guide to event-driven architecture. Brokers, streams, sagas, delivery guarantees, and the failure modes that only show up in production.

The event bus is live. Click any producer node to publish.

event acknowledged dead-lettered
offset 000

The shape of an event

An event is a fact: something happened, at a point in time, and it cannot un-happen. OrderPlaced, PaymentCaptured, SensorOverheated. Event-driven architecture is what you get when services stop calling each other and start reacting to these facts instead.

before Request / response

Checkout calls Payments and blocks. If Payments is slow, Checkout is slow. If Payments is down, Checkout is down. Every new consumer of "an order happened" means another synchronous call added to the critical path, and coupling compounds.

after Event-driven

Checkout publishes OrderPlaced and moves on. Payments, Inventory, and Email each consume at their own pace. Adding a fourth consumer touches zero existing code. Failure in one consumer strands that one workflow, not the checkout.

Anatomy of an event

{
  "id":        "evt_01J8ZQ4T",          // unique — enables dedupe
  "type":      "order.placed",        // past tense: it already happened
  "time":      "2026-07-02T09:14:03Z",
  "source":    "checkout-service",
  "subject":   "order/8842",
  "data": {
    "orderId":  "8842",
    "total":    129.90,
    "currency": "EUR"
  }
}
  • Events are immutable. You never update an event; you publish a new one. OrderAmended follows OrderPlaced, and the log keeps both.
  • Events are named in past tense. A command (PlaceOrder) asks for something. An event (OrderPlaced) reports that it happened. Mixing the two is the most common modeling mistake in the field.
  • The producer doesn't know its consumers. That ignorance is the point. It's what lets teams ship independently, and it's also what makes tracing harder. Both are true.
  • Carry enough state. A "thin" event (orderId only) forces consumers to call back for details, reintroducing coupling. A "fat" event carries the data consumers need. Most mature systems drift fat.
offset 001

Six patterns that carry the load

Nearly every event-driven system in production is a combination of these six. Hover a card to watch the pattern run.

Publish / Subscribe

One event, many independent readers. Each subscriber gets its own copy; none can slow the others down. The backbone of notification fan-out and cross-team integration.

Reach for it when: multiple teams care about the same fact.

Work queue

Events line up; a pool of workers competes to process them. Each message is handled by exactly one worker. Load-leveling for free: bursts pile into the queue instead of crushing the service.

Reach for it when: work is expensive and must happen once, like image resizing, PDF generation, or payment capture.

Event sourcing

Don't store state. Store the events, and derive state by replaying them. Current balance = sum of every deposit and withdrawal ever. Perfect audit trail, time travel for debugging, and the ability to build views you didn't know you'd need.

Reach for it when: the history is the business, like ledgers, medical records, or legal compliance.

CQRS

Split the write model from the read model, with events syncing the two. Writes stay strict and normalized; reads become denormalized views shaped exactly for each screen. The cost: the read side lags the write side. Design for that staleness or it will design itself.

Reach for it when: read and write workloads want incompatible shapes or scale.

Saga, distributed transactions without 2PC

A long business process (place order → reserve stock → charge card → ship) becomes a chain of local transactions stitched together by events. When a step fails, you don't roll back. You publish compensating events that undo earlier steps, like StockReleased and PaymentRefunded. There are two flavors. Choreography means each service reacts to the previous event, with no coordinator and a flow that's harder to see. Orchestration means a saga coordinator issues commands and tracks state, so you get one place to look and one more thing to run. Watch one run end-to-end in Scenarios ↓.

Reach for it when: a business flow spans services and "all or nothing" still matters.

Transactional outbox, closing the dual-write gap

A service that saves to its database and then publishes an event performs two writes. Crash between them and state and log disagree. The order exists, but no one ever heard about it. The outbox pattern removes the gap: write the state change and the event into the same database transaction (the event goes into an outbox table), then a relay (a poller or change-data-capture) reads the table and publishes to the broker. The event is published at-least-once, so consumers still need idempotency. What you gain is that a committed change can never go unannounced.

Reach for it when: an event must never be lost relative to the state it describes, which for business events is almost always.

offset 002

Feel backpressure

Queues absorb bursts, until they don't. Drive the producers faster than the consumers can drain, and watch consumer lag grow. Consumer lag is the first metric to check when an event system misbehaves. Find the settings where the system breathes, then break it.

queue depth0
consumer lag0 ms
throughput0 ev/s
verdict

Rule of thumb: a queue that only ever grows is an outage on a delay timer. Capacity must exceed arrival rate on average. The queue's job is to smooth variance, not to fix undersizing.

offset 003

Pick your guarantee

Networks drop packets, consumers crash mid-message, acks get lost. Every broker forces a choice about what happens next. There's no free option. Each mode trades message loss, duplication, or added complexity.

sent0
delivered0
duplicates0
lost0

* "Exactly-once delivery" over a network is impossible (Two Generals problem). What systems like Kafka transactions actually provide is effectively-once processing: at-least-once delivery plus idempotent handling. You dedupe on event.id, idempotency keys, or transactional offsets. The dedupe box in this animation is doing the real work, and it belongs in every consumer you ship.

offset 004

Four systems, end to end

Real architectures, played one event at a time. Step through each flow, including the step where something goes wrong.

offset 005

When events hurt

Event-driven architecture trades one set of problems for another. The trade is often worth it. It is never free.

What you gain

  • Team autonomy. Producers and consumers deploy on their own schedules. The event contract is the only meeting they need.
  • Burst tolerance. Queues turn a 10× traffic spike into a temporarily longer queue instead of a cascade of timeouts.
  • Cheap extension. New feature that reacts to orders? Subscribe. No one else's code changes, no one else's release train.
  • A built-in audit log. With event sourcing, "what happened and when" stops being a forensics project and becomes a query.
  • Failure isolation. A crashed consumer strands its own work. The producer never notices.

What you pay

  • Eventual consistency, everywhere. The read model lags. The inventory count is seconds stale. Every screen and every support script must tolerate it.
  • Debugging goes distributed. "Why did this happen?" now spans five services and a broker. Without correlation IDs and tracing, reconstructing a flow is slow, manual work.
  • Schema evolution is forever. Old events never go away. Every consumer must handle every version ever published, so a schema registry stops being optional at scale.
  • Duplicates and reordering are normal. At-least-once delivery means every consumer needs idempotency. Miss one and you'll double-charge someone.
  • Invisible flows. Choreographed sagas have no single place where the process is written down. The architecture lives in people's heads until you document it.

Skip events entirely when: the workflow is a simple CRUD form, one team owns the whole flow, the caller genuinely needs the answer right now (auth checks, price quotes), or your team hasn't built the observability muscle yet. A synchronous monolith you can debug beats a distributed system you can't.

offset 006

Choose your broker

The broker decides your semantics: replay or not, ordering or not, push or pull. Pick by workload, not by fashion.

BrokerModelReplayOrderingSweet spot
Apache Kafka Distributed log, consumers pull & track offsets Yes. Retention is a config, not a delete Per partition Event streaming, event sourcing, high-throughput pipelines, "the log is the system of record"
RabbitMQ Smart broker, queues + flexible routing, push No. Consumed means gone (streams plugin excepted) Per queue Task queues, complex routing topologies, RPC-over-messaging, modest scale with rich semantics
AWS SNS + SQS Managed fan-out (SNS) into managed queues (SQS) No (limited via FIFO + DLQ redrive) FIFO queues only Serverless architectures, zero-ops teams, spiky workloads billed per request
NATS JetStream Lightweight subjects + optional persistent streams Yes, with JetStream enabled Per stream Edge/IoT, low-latency service mesh messaging, small footprint deployments
Apache Pulsar Segmented log, compute/storage separated, multi-tenant Yes, tiered storage offloads to object stores Per partition Multi-tenant platforms, geo-replication, queue + stream semantics in one system

Sensible default: if you need replay and scale, start with Kafka or a managed clone. If you need work queues and routing, RabbitMQ or SQS. If you're all-in on one cloud, its native broker usually wins on operational cost, and the exotic choice has to earn its keep.

offset 007

The schema is the API

Producers and consumers never meet. The event schema is the only contract between them, and the only thing a producer can break for a team it's never spoken to. Mature event platforms treat schemas the way service teams treat REST APIs: described in a spec, versioned deliberately, and checked in CI.

The contract, written down in AsyncAPI

asyncapi: 3.0.0
info:
  title: Order events
  version: 1.2.0
channels:
  order.placed:
    messages:
      OrderPlaced:
        payload:
          type: object
          required: [id, type, time, data]
          properties:
            id:   { type: string }
            time: { type: string, format: date-time }
            data: { $ref: '#/components/schemas/Order' }
  • Describe events like you describe APIs. AsyncAPI is OpenAPI's sibling for event channels: which topics exist, what messages they carry, what every field means. The spec becomes documentation, codegen input, and review artifact in one.
  • Old events never go away. A consumer replaying a five-year-old stream will meet every schema version ever published. Evolution rules aren't bureaucracy. They're what makes replay possible at all.
  • Enforce compatibility in CI, not in code review. A schema registry with compatibility checks rejects a breaking change at build time. Humans reviewing diffs miss what machines catch mechanically.
  • Version the payload, not just the topic. Additive changes evolve a schema in place. Incompatible redesigns get a new major version, often a new topic (order.placed.v2) with both published during migration.
Change to the schemaVerdictWhy
Add an optional fieldSafeOld consumers ignore fields they don't know; new consumers get the extra data.
Add a new event type to a topicUsually safeConsumers should skip unknown types, so verify yours actually do before relying on it.
Add a required fieldBreakingEvery already-published event lacks the field; replays and lagging consumers fail validation.
Rename or remove a fieldBreakingConsumers reading the old name get nothing. A rename is a remove plus an add, so both ends break.
Change a field's type or narrow an enumBreakingDeserialization fails on the consumer side, typically at runtime and rarely where you're looking.

Default posture: additive-only evolution, compatibility checked by the registry in CI, and a major-version topic when a redesign is unavoidable. Consumers must tolerate unknown fields from day one, because that single habit buys most of the flexibility.

offset 008

Ordering is a per-key promise

A topic scales by splitting into partitions, and everything subtle about event systems lives in that split: ordering holds within one partition, never across the topic. The partition key is therefore a modeling decision disguised as a config value. It decides which facts the system is allowed to see out of order.

Anatomy of a partitioned topic

topic orders   // key = orderId → same order, same lane
├─ p0  #8842 #8842 #9011        consumer A
├─ p1  #7734 #7734 #7734 #8100  consumer A
├─ p2  #6621 #9455              consumer B
└─ p3  #5590 #5590              consumer B

group billing    // 2 consumers share 4 partitions
group analytics  // reads the same log, own offsets

rebalance: consumer B leaves →
  p2, p3 reassigned to A · processing pauses · resumes
  from committed offsets — duplicates possible, loss not
  • Key by the invariant, not by convenience. Events for one order must stay ordered, so key on orderId. Key on customerId and one big customer becomes a hot partition, one lane jammed while the rest idle. The right key is the entity whose story must read in order.
  • Partition count is a ceiling on parallelism. A consumer group can't usefully have more members than partitions, so the extras stand idle. Size partitions for peak, not average. Raising the count later reshuffles which keys land where.
  • Rebalancing is routine, not failure. Deploys, crashes, and scale-out all trigger it. Well-behaved consumers commit offsets promptly, keep per-message work short, and tolerate the duplicates a rebalance can replay. Idempotency again.
  • Cross-partition ordering doesn't exist, so design for it. If two entities' events must correlate in order, either they share a key or the consumer reorders with sequence numbers and a small buffer. Pretending the topic is globally ordered works right up to the first production load spike.
offset 009

Field glossary

Backpressure
Signals that flow upstream to slow producers when consumers can't keep up. The alternative is unbounded queues and an eventual outage.
Consumer group
A set of consumers sharing one subscription; the broker splits partitions among them so each event is processed once per group.
Dead-letter queue
Where messages go after repeated processing failures. Quarantine for poison messages, and the first place to look during an incident.
Idempotency
Processing the same event twice produces the same result as once. The property that makes at-least-once delivery survivable.
Offset
A consumer's position in the log. Committing it too early loses messages on crash; too late reprocesses them. (You've been scrolling one this whole page.)
Outbox pattern
Write the event into the same database transaction as the state change, then relay it to the broker, closing the "saved but never published" gap.
Partition
A shard of a topic. Ordering holds within a partition, never across them, so choose partition keys accordingly (e.g. orderId).
Poison message
A message that crashes its consumer every time. Without a retry limit and a DLQ, it blocks the queue forever.
Projection
A read model built by folding events into a view. The "Q" side of CQRS.
Schema registry
A service that versions event schemas and rejects incompatible changes before they reach production consumers.
Topic
A named channel of events. Producers write to it; any number of consumer groups read from it independently.
Watermark
In stream processing, an estimate of "we've now seen all events up to time T". It's how windowed aggregations decide when to close.