The Incident That Changed How I Think About Monitoring

The dashboard showed green across the board. CPU under 40%. Memory under 60%. Disk latency under 10ms. All services responding. All health checks passing.

Production was down. Users couldn't sign in. The help desk queue had 47 tickets in 15 minutes. And every monitoring metric was telling us everything was fine.

What actually happened

The authentication service wasn't down. It was almost down. A connection pool was approaching exhaustion — 98 of 100 connections in use, with new requests queuing. Most requests completed successfully, so the health check passed. But the people at the end of the queue were waiting 30+ seconds to sign in, which meant their sessions timed out, which meant they tried again, which added more load.

The monitoring caught "service is responding." It didn't catch "service is responding slowly enough to be effectively unusable." Those are different things.

What I changed

After that incident, I added three types of monitoring to every critical service:

Saturation metrics, not just availability. CPU, memory, disk — these are coarse. I added connection pool utilisation, thread pool depth, request queue length. Metrics that show a system trending toward failure, not just already failed.

Percentile latency, not average. Average response time was fine. P99 response time was 30+ seconds. Averages hide the tail. The tail is where users live. I now alert on P95 and P99 latency for every user-facing service.

Synthetic transactions. A real login flow, from a real client, running every 60 seconds. Not a TCP check. Not an HTTP 200 check on the health endpoint. An actual sign-in that validates the entire authentication chain. If this transaction fails, the service is down, regardless of what the component-level metrics say.

The monitoring philosophy I follow now

Monitoring tells you two things: "is it working?" and "is it about to stop working?" Most monitoring only answers the first question. The second question is harder but more valuable.

Every alert should be actionable. If an alert fires at 3am and the correct response is "acknowledge and go back to sleep," the alert shouldn't exist. I deleted half our alerts after the authentication incident and replaced them with better ones. Fewer alerts. Higher signal.

Dashboards are for humans. Alerts are for problems. If you're relying on someone to notice a dashboard change, you don't have monitoring. You have a screen saver.

The authentication incident was embarrassing. The post-mortem was painful. But the monitoring architecture we built afterward has caught problems before they became incidents more times than I can count. Sometimes you need a failure to learn what "working" actually means.