The call

ran the central campus email system - it was called Isis - at the University of North Carolina at Chapel Hill, and also wrote the sendmail configuration files used by most of the departmental mail servers on campus. Around 2002 the chairman of the statistics department telephoned him.

They were having a problem sending email out of the department. Specifically: they could not send mail farther than 500 miles. A little more, actually. Call it 520 miles. But no farther.

Harris's reply is the one anybody would give: email really does not work that way.

Then came the detail that turns an implausible ticket into a genuine report. They had waited to call because they had not yet collected enough data to be sure. They had asked one of the geostatisticians to look into it. She had produced a map showing the radius within which mail could be sent, slightly over 500 miles. Inside the radius there were destinations they could not reach, or reached only sometimes. Outside it, nothing ever went.

It was the statistics department. They had done the analysis.

The cause

Harris tested it himself, and it was true.

Some time earlier a consultant had "patched the server". In doing so he had upgraded the operating system, and with it sendmail - from version 5 to version 8 - while leaving the old version 5 configuration file in place. Sendmail 8 read a configuration written for a different generation of itself. It parsed what it recognised. For the settings it did not recognise, the binary had no compiled-in defaults, so they came out as zero.

One of the settings that came out as zero was the timeout for connecting to a remote mail server.

A zero timeout does not mean "wait forever" and it does not mean "fail immediately". On that particular machine, under its typical load, it meant that a connect call was aborted after slightly over three milliseconds.

And then the campus network mattered. It was entirely switched, so an outgoing packet incurred no router delay until it left the campus point of presence and met a router on the far side. For a lightly loaded host on a nearby network, the time to establish a connection was therefore governed less by queuing and processing than by the distance the light had to travel.

Harris's own account records what he typed next, and it is the moment the whole thing resolves:

$ units
You have: 3 millilightseconds
You want: miles
        * 558.84719

Five hundred miles, or a little bit more.

Harris is careful about the telling, and so should anyone repeating it be. His own note says the story was slightly altered to protect the guilty, to skip irrelevant detail, and to make it more entertaining. Asked later whether it happened, his answer was: yes, it happened.

Why it is not just a good anecdote

Believe the report you cannot explain. The chairman was describing something that could not be true, and it was true. The Therac-25 article makes the same point at far greater cost: the strongest signal in that whole sequence was a patient insisting she had been burned by a machine that, according to its manufacturer, could not burn anyone. A report that contradicts your model of the system is data about your model. The instinct to explain it away is the failure mode.

An upgrade silently rewrote a configuration. Nobody set the timeout to zero. A newer program read an older file, did not understand parts of it, and filled the gaps with nothing - and nothing, in that context, meant three milliseconds. This is the Knight Capital failure in miniature and the Ariane 5 failure in miniature: a component behaving correctly on inputs that were written for a different version of the world. It is also entirely current. A container image whose defaults changed between tags, a device that accepts an old configuration and silently drops the directives it no longer supports, an infrastructure module whose unset variable is not "unset" but zero - the mechanism has not aged at all.

The symptom was in a layer nobody was looking at. The complaint was about email. The fault was in a timeout. The visible pattern was geography. Nothing about "we cannot send mail to Seattle" points at a connect timeout, and no amount of reading mail logs at the application layer would have produced the answer - which is the argument the capture-points article makes about position: the layer where a problem is reported is rarely the layer where it lives.

And latency is distance. This is the part practitioners forget until it bites, and it bites constantly: in replication that works between two sites and not a third, in a database timeout that only fails for the branch office, in a voice service that is fine within a country and unusable across an ocean. Light in fibre travels about two-thirds of the speed of light in vacuum, and no amount of bandwidth changes it. When a fault correlates with a map rather than with a configuration, the map is telling you something.

Set your timeouts. The final, unglamorous instruction. Every timeout that is not explicitly configured is set by something - a default you have not read, an inherited file, or a zero. Knowing the value is the difference between a system that degrades and a system that behaves like physics.

Sources