the 500-mile email

networkingops culture

A classic sysadmin tale of an email system that mysteriously could not send mail farther than about 500 miles.

Trey Harris's story turns out to have a real cause: a broken config defaulted a network timeout to near zero, so only servers close enough to answer within a few milliseconds ever connected, and a few milliseconds of light-speed round trip works out to roughly 500 miles. It is the perfect parable for why the absurd-sounding symptom deserves a real root-cause hunt.

The 500-mile email is the best debugging story the profession has, because the absurd premise turns out to be literally correct. A statistics department reported that email would not send further than about 500 miles, the sysadmin's first reaction was that this is not how anything works, and the complaint was accurate.

The chain is genuinely beautiful. A mail server had been upgraded and its configuration file was written for a newer version, so the parser silently defaulted the connection timeout to zero rather than failing. A timeout of effectively zero meant the SMTP connection had to complete within the round trip time, and light in fibre plus switching delay works out to roughly the distance the users had measured. Speed-of-light latency became a business rule by accident.

The reason it endures is not the punchline. It is that the user's report was precise, the sysadmin's disbelief was reasonable, and the truth was underneath both. A story where the ridiculous complaint is correct is a permanent argument for taking reports seriously and for following a causal chain all the way down instead of stopping at the point where it stops making sense.

Also known as: 500 mile email, 500-mile email, Trey Harris

← All glossary entries