Postmortem: what k6 found that staging never did

Why staging looked fine

Staging ran a couple of pods against a database with a generous connection limit and almost no concurrent traffic. Production ran many times that number, each pod holding its own pool, against a shared instance. The per-pod configuration was identical. The aggregate was not, and nothing in the environment made that visible.

What the load test showed

A ramping-arrival-rate scenario made the cliff obvious. Throughput scaled cleanly up to a point, then p95 went vertical while CPU stayed flat — the signature of queueing, not of work. Requests were waiting for a connection that never came free.

The fix

Size the pool against the database limit divided by replica count, not per pod in isolation. Put a pooler in front so pod count and connection count stop being the same number. Then set a hard acquire timeout so a saturated pool fails fast and visibly instead of queueing silently.

Action items

• Breakpoint test in CI on every release candidate, with the threshold as the gate.
• Export pool wait time and saturation as first-class metrics, not debug logs.
• Assert replicas × pool_size < max_connections at deploy time.