Why usage metering breaks at scale (and how we fixed it)
At a few thousand events a second, naive aggregation falls over. Here's the ledger design that doesn't.
Usage-based billing sounds simple until you try to build it: ingest an event, add it to a running total, and bill the total. The trouble starts at the edges—late events, duplicate deliveries, retroactive corrections, and a billing period boundary that customers expect to be exact to the second.
The naive approach, and where it breaks
Our first prototype incremented a counter per customer per event type. It worked in a demo and fell over in week one of real traffic: two workers processing the same event after a network retry double-counted it, and an event that arrived four hours late landed in the wrong invoice.
// naive — breaks under retries and clock skew ledger[customer][event] += quantity;
The fix: an append-only ledger
Every event becomes an immutable row keyed by an idempotency key the caller supplies. Aggregation is a query over that ledger, not a mutation of a counter—so a duplicate delivery is a no-op, and a late event just lands in its correct billing period retroactively, recomputing only the invoice it touches.
“If you can't replay it, you can't trust it. The ledger is the source of truth; every total is a derived view.”
Handling the 72-hour window
We give events up to 72 hours of lateness before they're rejected outright. Invoices in that window are marked provisional and only finalised once the window closes—which means a customer's dashboard and the finalized invoice always agree, even if a batch job upstream was slow to flush.
The result: the same architecture handles ten events a second or ten thousand because the aggregation logic never assumed order or single delivery in the first place.