Everything is code
A platform you can't recreate from a repository is a platform you don't fully have. Infrastructure, workloads, dashboards, alert rules: all of it declared in code, reviewed, applied by machines. The test is simple. Could you rebuild the whole environment from git alone, on an ordinary Tuesday?
Anatomy of a plan
# terraform plan — review the diff, not the console
Plan: 3 to add, 1 to change, 0 to destroy
+ aws_sqs_queue.orders_dlq
message_retention_seconds = 1209600
+ aws_lambda_event_source_mapping.orders
batch_size = 25
maximum_concurrency = 40
+ kubernetes_horizontal_pod_autoscaler.consumer
min_replicas = 2
max_replicas = 12
~ aws_lambda_function.enricher
timeout: 30 → 60 # the actual change under review- The plan is the review artifact. Nobody approves "I'll bump the timeout." They approve this exact diff against real state, and what you review is what gets applied, byte for byte.
- State is the second source of truth. The world drifts: a console hotfix at 2am, a quota change, a deleted tag. Scheduled plan runs surface drift as a diff instead of a surprise.
- Terraform or CDK, by team and workload. Terraform's declarative HCL reads like an inventory and diffs cleanly. CDK gives application teams real types and loops in the language they already write. Both beat the console, and mixing them by layer (Terraform for foundations, CDK or serverless frameworks for app stacks) is common and fine.
- Separate by blast radius. Network and identity change monthly; workloads change hourly. Different repos, different pipelines, different approvers, so a routine app deploy can never take the VPC with it.
One artifact, many environments
The pipeline's job is to produce one immutable, pinned artifact and prove things about it. Environments differ by configuration, never by build. The image that passed staging is bit-for-bit the image in production.
gates What the pipeline proves
Lint → test → build → scan, in that order, cheapest first. Every gate is automated and blocking: a failed contract test stops the artifact the same way a failed compile does. Human review is for design; machines gate correctness. If a check can be argued past, it's a suggestion, not a gate.
Pinned and reproducible. Base images by digest, dependencies locked, build args recorded. "Works on my machine" ends where the lockfile begins, and a CVE announcement becomes a grep instead of an archaeology project.
artifacts What comes out
An image and its provenance. The tag maps to a commit; the commit maps to a review; the review maps to a plan. Any pod in production traces back to a human decision in under a minute.
Config stays out of the artifact. Endpoints, feature flags, and limits arrive at deploy time; secrets never live in git or in images. They're injected from a secrets manager the runtime trusts. If you have to rebuild to change a URL, the boundary is drawn wrong.
Git is the deploy button
GitOps inverts the push. Nothing deploys to the cluster; the cluster pulls what git says should run, continuously reconciling reality against the declared state. A deploy is a merged PR, a rollback is a revert, the audit trail is git log. Run one below, then ship a broken build and watch the health checks catch it.
Rolling replaces pods a few at a time behind the load balancer: cheap, with steady capacity. Two versions serve traffic at once, so changes have to stay backward-compatible for the duration. Try the broken build. Readiness probes fail, the rollout halts, the surge pods are torn down, and capacity never drops.
Two shapes of platform
The same event-driven system runs in two very different bodies: managed serverless primitives, or a Kubernetes cluster you run yourself. What separates them is where your operational effort goes. Your workload and your team decide which body fits. The two panels lay out the trade, so you can find yours in it.
shape a Serverless on AWS
Lambda consumers, EventBridge routing, SQS buffering, DynamoDB state: the broker patterns from the EDA guide, run as managed services. Scaling, patching, and failover sit with the provider. You pay per request, so idle costs nothing and spiky traffic stays cheap.
Where it costs you: cold starts show up on latency-sensitive paths, execution caps out at 15 minutes, and per-request pricing crosses over expensive once throughput stays high. You also inherit one vendor's semantics: event replay, DLQ redrive, and concurrency limits all behave AWS's way.
Fits when: traffic is spiky or modest, the team is small, nobody's dedicated to ops, and getting to production matters more than staying portable.
shape b Kubernetes and GitOps
Long-running consumers, Kafka clients with real rebalancing control, GPU workloads, anything stateful or latency-critical. One deployment model covers every workload, and the same manifests run on EKS, AKS, or bare metal. Processes that hold connections open, like agents and stream processors, live here comfortably, where Lambda's execution ceiling would cut them off.
Where it costs you: the cluster becomes a product your team runs, with upgrades, node pools, ingress, certificates, and capacity planning attached. GitOps automates the toil, not the responsibility, and the baseline bill never reaches zero even at idle.
Fits when: throughput is sustained, workloads run long or hold state, multi-cloud is a real requirement rather than a someday, or many teams share the same rails.
In practice most systems end up running both: serverless for the spiky, glue-shaped edges, a cluster for the sustained core. The expensive mistake isn't picking one over the other. It's leaving each team to improvise per service, with no shared path anyone agreed on. Worth settling where your own split falls before the codebase settles it for you.
Scale on the queue, not the CPU
CPU-based autoscaling answers the wrong question for event systems. A consumer sitting at 40% CPU can still be hours behind. The signal that matters is queue depth and consumer lag: scale out when the backlog grows, scale in when it drains. Drive the load below and watch the autoscaler chase it.
Same physics as the backpressure playground, one layer up: the queue absorbs the spike, the autoscaler adds consumers, the backlog drains. Watch the lag between spike and scale-out; that window is what the queue is for. Serverless does this dance implicitly, with SQS driving Lambda concurrency. On Kubernetes you configure it, with KEDA or HPA reading queue metrics. Either way, the signal is the backlog.
Rehearsed failure is just operations
Every failure mode below has a boring, practiced answer on a healthy platform. You don't avoid these. You rehearse them until each one is a runbook rather than an incident.
The recurring five
- The bad deploy. Readiness probes fail, the rollout pauses itself, capacity holds on the old version. Fix forward or
git revert; both are one merge. If a rollback needs a meeting, the platform isn't finished. - The crash loop. A pod restarts forever on a poison config or missing secret. Backoff caps the churn; alerts fire on restart rate, not on the third page of logs.
- Config drift. Someone "fixed" production by hand, and now git and reality disagree. GitOps reconciliation reverts it automatically. The console change becomes a PR, or it doesn't survive.
- The stuck rollout. New version healthy, old version won't drain, usually a long-lived connection or a PodDisruptionBudget nobody remembers writing. Set drain timeouts and connection budgets before the deploy, not during it.
- The thundering restart. A node dies; thirty consumers rejoin at once; the rebalance storm hits the broker harder than the outage did. Staggered restarts and cooldowns belong in the manifests.
What makes them boring
- Health checks that tell the truth. Readiness gates traffic; liveness restarts the wedged. A readiness probe that just returns 200 is a decorative gate.
- Rollback as a reflex. Revert-merge-done, under five minutes, rehearsed on purpose in daylight. The first time you roll back shouldn't be during an outage.
- Blast-radius budgets. One service's deploy can't consume the cluster: quotas, PodDisruptionBudgets, and per-namespace limits are set before they're needed.
- Idempotent everything. Replays, retries, and restarts are the platform's weather. Consumers and jobs that tolerate duplicates (see the EDA guide) turn that weather into noise.
- Game days. Kill a node, expire a cert, break a build, on a Tuesday afternoon with coffee rather than a Saturday night with adrenaline.
The best secret is no secret
The perimeter of a modern platform is identity, not network. The ranking is strict: a credential you never issue beats one you rotate, and one you rotate beats one somebody remembers. Federation moves almost everything into the first category.
Anatomy of a keyless deploy
# ci deploy job — no cloud keys stored anywhere
permissions:
id-token: write # mint an OIDC token for this run
steps:
- uses: aws-actions/configure-aws-credentials
with:
role-to-assume: arn:aws:iam::…:role/deploy-web
# the role's trust policy pins exactly who may assume it:
condition:
sub: repo:org/web:environment:production
# credentials live for minutes and are scoped to one role —
# there is nothing to leak, nothing to rotate- Federation kills the stored key. CI proves who it is with a short-lived identity token; the cloud exchanges it for minutes-long credentials. A leaked token from a fork can't assume the production role, because the trust policy pins repository, branch, and environment.
- Workloads get identities too. Pods assume roles natively (IRSA, pod identity, workload identity); Lambda has execution roles. The key file that used to sit in the image simply doesn't exist.
- Real secrets go in a manager. Database passwords and third-party API keys live in a secrets manager and are injected at runtime. Git and images stay clean, rotation becomes a config change, and access is itself audited.
- Scope like you mean it. The deploy role can deploy and nothing else; namespaces get RBAC budgets like they get resource quotas. Least privilege isn't paperwork here, it's blast-radius engineering (see stage 005 ↑).
Alert on symptoms, debug with causes
Users experience latency, errors, and staleness, never CPU. Pages fire on symptoms tied to service-level objectives; everything else is a dashboard for the debugging that follows.
The signals that page
slo checkout-events
indicator p95 end-to-end delivery < 2s
objective 99.9% over 30d
budget 43m of breach per month
page error-budget burn > 14× // wake a human
page consumer lag growing 15m // see stage 004
warn deploy freq ↓, lead time ↑ // platform health
dash cpu, memory, gc, restarts // causes, not pages- Error budgets make reliability negotiable. 99.9% means 43 minutes of breach per month is allowed. Spend it on risky deploys or save it for bad weather. Burn it too fast and the budget freezes the release train, no manager required.
- Lag is the platform's heartbeat. For event systems, consumer lag is the one metric that unifies app health, broker health, and capacity. It pages before customers notice, which is the whole point.
- Deploy metrics are health metrics. Deployment frequency and lead time falling is the platform quietly failing its own users, the engineers. Watch them like uptime.
- Correlate by trace, not by timestamp. One trace ID from HTTP edge through broker to consumer (the agent guide extends this to model calls) turns "what happened at 3:07" from a meeting into a query.
Field glossary
- Blast radius
- Everything a single change can break. Good platforms shrink it structurally, with separate state, scoped credentials, and per-namespace quotas, before any human judgment is involved.
- Blue-green
- Two full environments; traffic flips atomically from blue to green and can flip back as fast. Costs double capacity for the duration; buys instant, total rollback.
- Canary
- The new version serves a small slice of real traffic first; metrics decide whether the rest follows. The safest strategy, and the one that most needs good observability.
- Cold start
- Latency of a serverless function's first invocation after idleness, from runtime boot plus init code. The tax serverless pays for scale-to-zero.
- Drift
- The gap between declared state (git) and actual state (the world). Detected by plan runs, reverted by reconciliation, caused by humans in consoles.
- Error budget
- The allowed unreliability inside an SLO, the currency risky changes spend. When it's gone, the release train stops without a debate.
- GitOps
- Deployment model where the cluster continuously pulls and reconciles the state declared in git. Deploy = merge, rollback = revert, audit = log.
- HPA / KEDA
- Kubernetes autoscalers. HPA reads resource metrics, KEDA reads external signals like queue depth. For event systems, backlog beats CPU as the trigger.
- Immutable artifact
- A build output that never changes after creation, with pinned digests and locked deps. Environments vary by config, never by rebuild.
- OIDC federation
- Exchanging a workload's short-lived identity token for short-lived cloud credentials, so CI and services authenticate with nothing stored and nothing to rotate.
- Readiness / liveness
- The two truths a pod tells the platform: "route traffic to me" and "I'm not wedged." Rollout safety is only as honest as these probes.
- Reconciliation
- The control loop that makes reality match declaration, continuously. The reason a console hotfix quietly un-happens.
- SLO
- Service-level objective: a user-visible target (latency, availability, freshness) that turns "is it reliable?" into a number with a budget.
- Workload identity
- Pods and functions assuming IAM roles natively (IRSA, pod identity, execution roles). The key file that doesn't exist can't leak.