the Knight Capital glitch

programmingops culture

The 2012 trading firm that lost about 440 million dollars in 45 minutes from a botched software deployment.

An engineer failed to copy new code to one of eight servers, and a reused feature flag reactivated long-dead test logic that fired millions of unintended orders into the market. The loss nearly bankrupted the firm within the hour - the definitive cautionary tale about deployment hygiene and dead code left in place.

On 1 August 2012 Knight Capital deployed new trading software to eight servers, or rather to seven of them. The eighth kept old code, in which a retired flag now meant something entirely different, and that server began buying high and selling low across a hundred and fifty stocks at machine speed.

The loss was roughly 440 million dollars in about forty-five minutes, which exceeded the firm's assets and effectively ended it as an independent company. What makes the case instructive is that no individual component failed: the software worked, the servers worked, the exchange worked. The failure was in deployment, specifically that a manual process had no verification step and nobody noticed one host was inconsistent.

Everything about modern deployment practice is an answer to this shape of failure. Automated and repeatable deployment, so a manual step cannot be skipped. Verification that all hosts are running the intended version. Feature flags rather than reusing retired ones. Kill switches that stop a runaway system without requiring a diagnosis first. Knight's engineers took those forty-five minutes trying to work out what was happening, which is precisely the time nobody has.

Also known as: Knightmare, Knight Capital, $440 million

← All glossary entries