Monitoring: logs, alerts, uptime
The worst way to learn prod is broken is from an angry customer. Monitoring is knowing before they do. Here are the three layers I set up: structured logs, metrics, and alerts that don't cry wolf.
The worst way to learn prod is broken is from an angry customer. Monitoring is knowing before they do. Here are the three layers I set up: structured logs, metrics, and alerts that don't cry wolf.
Monitoring isn’t a tool, it’s three questions: what happened (logs), how is the system doing (metrics), and am I warned when it goes wrong (alerts)?
A free-text console.log is unreadable as soon as volume grows. A structured (JSON) log is filterable, searchable and aggregatable — you find all of a user’s errors in one query.
For an AI agent, the same logic gives tracing: every step visible and searchable, as in trace and score an agent.
An alert that fires too often ends up ignored — that’s alert fatigue, and it’s worse than no alert at all. Alert on symptoms that matter to the user, not on every technical twitch.
The % of 5xx requests crosses a threshold: something is broken, now.
The p95 spikes: the app is slow for a real share of users.
A ping from outside every minute: is the site actually responding?
Disk full, DB connections maxed: the foreseeable outages, to catch before the crash.
Don't monitor everything at once. An uptime check and an error-rate alert already cover most real outages.
Monitoring turns outages from surprises into non-events: you're warned, you have the logs to understand, and you fix before it shows. Start minimal — uptime + error rate — and grow. The goal isn't to see everything, it's to know before the customer.
Grafana — Getting started ↗