The defect

Division in the early Pentium used the Sweeney-Robertson-Tocher algorithm, which works by looking up partial quotients in a table. The table was implemented as a programmable logic array of 2,048 cells, of which 1,066 needed to hold one of five values: minus two, minus one, zero, plus one, plus two.

Because of an error in the script that loaded the array, five entries were left out. When the division hardware indexed one of those cells it read zero instead of plus two, and the answer came out wrong - by a relative error of up to about six parts in a hundred thousand.

That is the whole engineering fault. A script, five cells, a wrong answer in the fifth significant figure. Later steppings loaded the complete table and the problem disappeared.

The statistic, and why it was the wrong one

Intel's position was that the flaw was negligible, and the company published a study to support it: Statistical Analysis of Floating Point Flaw in the Pentium Processor, which estimated that a typical user would encounter an error roughly once in many thousands of years of ordinary computing. On random 64-bit divisions the rate is about one in nine billion.

The arithmetic is correct and the conclusion does not follow, for a reason that is worth internalising because it recurs constantly in security and reliability work.

The flaw was deterministic. For a particular pair of operands it did not fail rarely; it failed every time. A once-in-nine-billion figure describes what happens when inputs are drawn at random, and real computations do not draw at random. They run the same formula over the same data repeatedly, in a loop, in a spreadsheet model, in a nightly batch. A user whose work happened to touch an affected operand pair would meet the error on every run and never see a correct result - while the population average said the problem practically did not exist.

Averaging over the population is the right statistic for estimating total incidents. It is the wrong statistic for telling an individual whether they are affected, and Intel used it to answer the second question. Any risk argument of the form "the probability is tiny" deserves the follow-up: tiny across what, and is the failure independent or deterministic for a given input?

The timeline, which is the real story

May or June 1994. Intel's own testing finds it. Accounts differ on who and exactly when, and one detailed reconstruction suggests it was probably found more than once internally and kept quiet each time. The internal assessment, quoted in that account, was that it did not even qualify as an erratum. The company quietly revised the circuitry, disclosed nothing, and did not recall anything.

13 June 1994. Thomas Nicely, a mathematics professor at Lynchburg College computing sums over primes and twin primes, notices inconsistencies shortly after adding a Pentium to his machines. He spends four months eliminating other explanations - his own code, the motherboard, the chipset - because a professional's first assumption is that the fault is his.

19 October. He is certain the processor is wrong. Older machines give the right answer; three different Pentium systems give the same wrong one.

24 October. He reports it to Intel.

1 November. Intel replies that preliminary checks have found no issue.

30 October, and after. Nicely emails academic contacts describing the bug and asking them to test their own machines. Verification comes back quickly, and the story spreads across the young internet at a speed the company had not planned for.

Then the position that broke it. Intel offered replacements only to customers who could demonstrate that they were affected - a burden that was close to impossible to discharge, since establishing that your calculations touched one of the missing table entries required knowing which entries were missing, which Intel had not published. The people asked to prove harm were the ones who could not.

20 December 1994. Intel reversed: replacement on request, no questions asked, with a charge against fourth-quarter earnings. The charge was about $475 million. The company's annual report described itself as having been engulfed in a controversy.

What a practitioner should take from it

The defect was cheap and the handling was expensive. A different sequence was available at every step: publish the erratum in May, describe the affected operations, ship the fix, offer replacement to anyone doing numerical work. That path costs a bad week and some silicon. The path taken cost most of half a billion dollars and thirty years of being the example. This is the no-perfect-code argument in hardware: the failure was not in engineering, it was in the interval between knowing and saying.

"We fixed it quietly" is a decision with a shelf life. Intel corrected the circuitry while saying nothing, which works only for as long as nobody outside finds the original. Somebody always finds it - and when they do, the fact that you already knew becomes the story, replacing the defect entirely. DigiNotar revoked certificates for a month in silence and was found by a member of the public; the shape is identical.

Asking the customer to prove harm inverts the relationship. In practice it tells them they are the ones being doubted, at the moment they most need to be believed - which is the opposite of what the Therac-25 record teaches about reports you cannot immediately explain. Where a defect is deterministic and the affected inputs are known to the and not to the user, the burden cannot honestly sit with the user.

And the finder was an outsider doing his own work. Nicely was not auditing Intel. He was counting primes, noticed his numbers were wrong, and spent four months suspecting himself before suspecting the chip. Almost every case in this catalogue has someone like him in it - Kaminsky with DNS, a Gmail user in Tehran with DigiNotar, a statistics department with a map. The report that arrives from outside, about something that should not be possible, is the most valuable input a vendor receives and the one most likely to be dismissed.

Sources