Coinbase Bet the Exchange on One Availability Zone

In May 2026, a handful of cooling units failed inside a single AWS data hall in US-East-1. That's the kind of localized fault Availability Zones exist to contain. Coinbase's exchange stopped trading anyway, and stayed down for most of a day.

Coinbase runs its matching engine, the system that actually executes trades, as a five-node Raft cluster. For low-latency high-frequency trading, the nodes were collocated inside a single AWS Cluster Placement Group, all within one Availability Zone, to shave the network hops between consensus members. That's a real, defensible engineering choice. Raft nodes talk to each other constantly, and cross-AZ round trips add milliseconds a trading venue genuinely cares about.

When the cooling failure took that AZ's compute offline, three of the five Raft nodes went down with it. A five-node cluster needs three to hold quorum. Losing exactly three meant losing quorum entirely, and a matching engine without quorum doesn't degrade, it stops. Kafka, carrying the operational data pipeline, sat in the same AZ and got stranded too, backing up messages nobody could process. Trading halted for several hours. Full recovery, per Coinbase's own postmortem, took most of the following day, because there was no automated path to reconstitute the cluster somewhere else. Someone did it by hand.

Coinbase's own writeup is honest about the tradeoff: it names the single-AZ placement "a classic engineering trade-off" of latency against resilience, and commits to automated cross-zone recovery for the matching engine, better quorum-restoration procedures, and more resilient cross-AZ messaging so Kafka doesn't strand itself the same way twice. That diagnosis is correct, and it's the right conclusion to draw.

Where I'd push back is on how easy the fix sounds as a bullet point. Raft already has a standard, safe way to change cluster membership: joint consensus, or single-node changes, exist precisely so a cluster is never governed by two overlapping quorums at once. That part isn't the unsolved problem.

The unsolved part is deciding, automatically, when to pull that trigger. This incident was easy to diagnose from the outside: cooling failed, an AZ went dark, the signal was unambiguous. Most future incidents won't hand you that. A network partition can look identical to a dead node from the other side of it. A health check can time out from load, not failure. An AZ can be degraded without being obviously gone, a gray failure, not a clean one. Trigger an automated failover on a false signal and, at best, you've evicted healthy nodes and handed yourself a smaller, more fragile cluster for no reason. At worst, the code driving that decision was itself written and tested under the pressure of "we need this before the next incident," and doesn't correctly implement the safe reconfiguration Raft actually offers. The algorithm has an answer. Whether the automation built to invoke it, quickly, after a bad quarter, implements that answer correctly is a separate question, and it's the one this postmortem doesn't touch.

The honest diagram of this system draws five Raft nodes inside one AZ boundary, labeled "quorum lives entirely here," next to a second, empty box labeled "cross-AZ failover: none built." That's the architecture Coinbase was actually running. Putting it on one canvas does something a page of prose can't: it makes the single point of failure the first thing the reader sees, rather than a sentence three paragraphs into a postmortem.

Strip away Coinbase, Raft, and high-frequency trading, and the story underneath is simpler: a latency optimization quietly became a resilience decision nobody revisited. Every system that collocates its stateful, quorum-bearing components for speed is making the same bet Coinbase made, whether or not anyone wrote it down as a bet. The AWS event was the trigger. The failure was deciding, at some point, that one Availability Zone was an acceptable blast radius for the entire trading engine, and never deciding what would happen on the day that turned out to be wrong.

If you're running anything where being wrong costs more than being slow, don't take "we'll automate the recovery" at face value, not from Coinbase's remediation list and not from your own team's postmortem next time this happens to you. The hard engineering isn't wiring up an automated failover path. It's making sure that path can't fire, get it almost right, and quietly create the one failure mode worse than the outage it was built to prevent. That's a harder problem than the one line item makes it look like, and it deserves its own argument before anyone ships it, not a checkbox next to "done."