#01Summary
Deliveries stopped on a quarter of partitions right after a routine deploy; rollback restored flow, and the backlog drained without loss.
Delivery is fully caught up. No events were lost. The permanent fix (dead-letter routing for per-event failures) shipped the next morning.
The 14:03 deploy of courier-dispatch added payload signing. The signer throws on events missing tenant_id, which is true of about 0.2% of pre-2024 events still flowing through replays. The consumer's error path nacked the whole batch, so any partition containing one such event redelivered the same batch forever. Twelve partitions stalled; the other thirty-six kept delivering, which is why the first alert was a p99 latency breach rather than a hard outage.
#02Timeline
All times UTC. Detection took 16 minutes because no alert watched per-partition lag.
- 14:03 Deploy
2026.08.12-1rolls out payload signing to all dispatch pods. - 14:03 Partitions holding a legacy event begin redelivering the same batch. Backlog starts growing.
- 14:19 Delivery p99 alert fires (first signal). Aggregate lag is still under the 500k alert threshold.
- 14:21 On-call paged; dashboards show healthy aggregate throughput, which delays diagnosis.
- 14:31 Per-partition breakdown reveals 12 partitions pinned at the same offset since 14:03. Logs show repeating
MissingFieldError: tenant_id. - 14:37 Rollback to
2026.08.11-4approved and started. - 14:44 Rollback complete. Stalled partitions resume; backlog peaks at 2.1M events.
- 15:22 Backlog fully drained. Incident closed.
#03Root cause
The signing bug was the trigger. The mechanism that turned it into an outage was the consumer's error path.
- The trigger. The new signer calls
sign(event.tenant_id, payload)and throwsMissingFieldErrorwhen the field is null. Legacy events from before the 2024 tenant migration have exactly that shape. - The amplifier. The consumer wraps the whole handler in one catch. On any exception it calls
batch.nackAll(), which requeues the entire batch to the same partition. - The stall. The redelivered batch contains the same poison event, throws again, and nacks again. The offset never advances, so the partition delivers nothing while its backlog grows at the full produce rate.
The consumer had no escape path for a poison event: nackAll() retried indefinitely with no cap, no dead-letter route, and no offset advance. Any per-event exception on any partition would have produced the same stall; the signing deploy merely supplied one.
The fix routes the failing event to the dead-letter topic and acks it, letting the partition move on:
} catch (EventError e) {
- batch.nackAll(); // redelivers the whole batch, forever
+ deadLetters.publish(event, e);
+ batch.ack(event); // partition keeps moving
}
#04Contributing factors
Separate from the cause: what made this slower to see and slower to stop.
| Factor | Effect |
|---|---|
| Aggregate-only lag alert (500k threshold) | Threshold was sized for bulk imports, so 12 stalled partitions stayed invisible for 16 minutes. ● slowed detection |
| Dashboards sum lag across partitions | Healthy partitions masked the stalled ones; on-call spent 10 minutes ruling out downstream receivers. ● slowed diagnosis |
| Manual approval gate on rollback | Added 6 minutes between diagnosis and mitigation start. ● slowed mitigation |
Legacy null-tenant_id events |
About 0.2% of replay traffic predates the tenant migration; staging replays only post-migration data, so the deploy passed cleanly there. ● hid the trigger |
#05What changes
One change kills this exact bug; the rest shrink the class of failures it belongs to.
| Remediation | Scope | Status |
|---|---|---|
| Dead-letter route for per-event failures (the diff above) | stops this exact bug | shipped 08-13 |
| Per-partition max-lag alert (any partition pinned > 5 min) | reduces the class | in review |
| Canary partitions for dispatch deploys (2 partitions, 15 min bake) | reduces the class | planned · Q3 |
Backfill tenant_id on pre-2024 events |
removes the trigger | planned · Q3 |
#06What went right
Two things worth keeping exactly as they are.
Rollback tooling completed in 7 minutes once approved, and the same offset behaviour that caused the stall also guaranteed zero loss: because offsets never advanced past the poison events, every delayed event was still in the topic and drained cleanly after rollback.
Fictional example · INC-2417 · timeline reconstructed from pager log + deploy history · all times UTC.