An exciting deployment is a risky one. “Works on my machine” means the environment isn't reproducible; a stressful release means there's no safety net. The goal: to deploy on a Friday at 5pm without thinking about it.
The same command produces the same result, everywhere. Zero manual step, zero “don’t forget to also…”.
A merge to main triggers build, tests, deploy. Humans no longer copy files by hand.
You know in real time whether the deploy went well — health checks, logs, metrics — without waiting for a customer ticket.
A one-command rollback. The question isn’t “will it work” but “how fast can I roll back.”
Infra clicked by hand in the console isn’t reproducible and nobody knows what changed. In Terraform, infra becomes code: versioned, reviewed in a pull request, recreatable identically.
The key principle: the same deploy script runs in CI and locally. No secret logic hidden in the GitHub UI — if CI is down, you run the script by hand and get the same result.
The new version only takes traffic after proving it’s alive. The load balancer polls a /health endpoint; until it returns 200, the old version stays in place.
With a blue-green or rolling deploy, the switch is gradual: if /health fails, nothing switches and the user sees nothing. Rollback becomes a non-issue.
On the application side, the same observability logic applies to AI: I cover tracing in trace and score an agent.
I mentioned blue-green above; in practice I pick between two strategies depending on risk. With blue-green, two identical environments run in parallel: I switch 100% of traffic from "blue" to "green" at once, and rollback is just a traffic switch back to the still-running "blue." With canary, I shift traffic in steps — 5%, then 25%, then 100% — and only advance if error rates and latency stay healthy.
Instant cutover, instant rollback.
Gradual exposure, capped blast radius.
My rule of thumb: blue-green when instant rollback matters most and traffic is predictable; canary when I want to validate a risky version on a real sample before exposing everyone. Both are handled on AWS with CodeDeploy and ELB traffic shifting.
The /health I showed above actually hides two different questions, and Kubernetes splits them explicitly with its liveness, readiness and startup probes. Readiness says "I can take traffic now" (DB connected, cache warm): if it fails, I'm pulled from the load balancer without being killed. Liveness says "I'm alive, don't restart me": if it fails, the container is restarted. Conflating the two leads to restart loops the moment a dependency slows down.
Classic trap: putting dependencies in the liveness check. If the database hiccups for 30s, all your instances fail liveness at once, get restarted at once, and you turn a transient slowdown into a full outage. Liveness must stay local to the process.
"Boring" isn't just a feeling — it's measurable. The DORA metrics, from six years of Google research, capture both the velocity and the stability of a delivery pipeline. The report's key counterintuition: speed and stability aren't a trade-off — the best teams deploy more often AND recover faster.
Note: DORA has evolved its model to five metrics (adding a "rework rate"), but these four remain the foundation. I instrument them like everything else — metrics, logs and traces — in line with the three signals of OpenTelemetry.