The setup

Group was, in mid-2012, responsible for roughly one tenth of all trading in United States equities. Its order router, SMARS, took a customer's order and broke it into child orders sent to the market.

Inside that router sat a function called Power Peg. According to the Securities and Exchange Commission's order, Knight had moved a section of code in 2005 to an earlier point in the sequence, which rendered the function defective - it could no longer recognise when orders had been filled, and would therefore keep sending. The function was not meant to be used. Knight left it in the router anyway, and it was not retested after the move to see whether the application would still work if it were ever invoked.

It sat there for seven years, unused and broken, behind a flag that switched it on.

The deployment

In late July 2012 Knight prepared to participate in a new NYSE facility, the Retail Liquidity Program. The new code was written to replace the unused Power Peg logic - and it reused the same flag. The reasoning is easy to reconstruct and was entirely ordinary: Power Peg is dead, so the switch is free.

The rollout began on 27 July. It was manual and staged, a technician copying the updated code to each of the eight SMARS servers in turn.

One server was missed.

No second technician reviewed the deployment. Knight had no written procedure requiring that anyone should. So nobody noticed that on the eighth server the new code was absent and the old, defective Power Peg code was still present - now reachable through a flag that the rest of the estate was about to start setting for an entirely different purpose.

The morning

At about 8:01 on the morning of 1 August, an hour and a half before the market opened, Knight's own systems began sending automated warnings. Ninety-seven emails, referencing SMARS, reporting an error described as "Power Peg disabled".

The system said the name of the thing that was about to destroy the company, ninety-seven times, before anyone had lost a cent. The messages were not routed to anyone who acted on them.

At 9:30 the market opened. Seven servers processed the new program correctly. The eighth received orders eligible for the new facility, set the reused flag, and invoked a function that had been broken since 2005 - one that could not tell when an order had been filled, and so kept filling it.

In forty-five minutes, attempting to satisfy 212 customer orders, the router sent more than four million orders into the market, producing over four million executions across 154 stocks and more than 397 million shares. Knight ended holding roughly $3.5 billion of unwanted long positions in 80 stocks and about $3.15 billion short in 74. The loss exceeded $460 million. The firm had about $365 million in cash.

It survived the weekend on a $400 million emergency investment, and was acquired within months.

The rollback

This is the part that makes the case worth its own article, and it is the part most retellings skip.

When the engineers worked out that one of the eight servers had not received the new code, they did what every instinct and every runbook says to do: they restored consistency by reverting all of the servers to the previous version.

The previous version contained Power Peg, wired to the flag that the market was now setting.

The rollback took a fault that existed on one server and put it on all eight. The single most reasonable action available, taken under time pressure by competent people with the correct diagnosis, amplified the damage - because the "known-good state" they were returning to was the state that contained the defect.

What a practitioner should take from it

A rollback is a deployment. It carries the same risk as any other change, it deserves the same scrutiny, and it should be reasoned about rather than reflexed. The instinct to restore a known-good state assumes the old state is good, and in every case where the fault predates the change - dead code, a latent configuration, an expired certificate, a dependency that has since moved - that assumption is exactly wrong. The question before pulling the lever is narrow: what am I going back to, and does it contain the thing that is hurting me?

Dead code is not inert. The Ariane 5 inquiry made the same finding about a routine that served a purpose only before lift-off and kept running anyway. Knight's function had been broken for seven years and cost nothing for seven years, which is precisely why nobody removed it. The cost of unused code is zero until the day it is not.

Never reuse an identifier that still means something somewhere. Flags, ports, virtual local area network identifiers, community strings, route tags, group names - the reasoning "that is obsolete, so the name is free" is only safe if the obsolete thing has actually been removed rather than merely disabled. Deprecate by deletion, or do not reuse.

Partial deployment is a state nobody designs for. Seven of eight is not "mostly deployed"; it is a configuration that was never tested, never intended and never documented. Automation removes this failure mode not by being clever but by being uniform, and a verification step that answers "is every node running the version I think it is" would have caught it in seconds.

And an alert nobody reads is worse than no alert. Ninety-seven messages, ninety minutes early, naming the component. The Target breach is the same failure with a different consequence: detection is not the control, response is the control, and the interval between them is where the loss accumulates. This is the clock argument at its starkest, because here the clock was running before the market even opened.

Sources