The 500-mile email (2002)

In Real Life, I totally agree the details are important. And I think I have evidence of this: at Google, where they had peer bonuses where one engineer could give money to another as a pat-on-the-back, I got dozens for my post-mortems.

Post-mortems are a case where you must have both story and correct details: lack the first, and you won't create change because the people who need to know in order to implement the required recommendations won't read the whole thing (or retain it later); lack the second, and really, how can anyone trust your recommendations?

Here, I was just trying to quickly bang out a funny anecdote. The things that stuck in my mind I could use to reverse-engineer numbers. I did this because—at the time I worked the incident—I was working with real numbers, so the story needed them for verisimilitude, to give a sense of what I was wrestling with. If I'd had any clue this mail would have taken on such a life of its own, I would have been more careful with them and gotten a tech reviewer and copy editor before posting.

This gets posted on some forum or another several times a year; for a long time I had a Google Alert on it and would hop in threads whenever it happened, since it always followed a common pattern:

1. Someone posts a link to the story, but not my canonical copy with a link to the FAQ.

2. More trusting and/or less-technical respondents upvote or forward or Like or +1 or quasisuperplauditize or whatever the medium has until it gets notice from...

3 ... less trusting and/or more-technical types, who expose the "flaws", most of which are covered in the FAQ.

4. Someone thinks to do a Google on "500 mile email", which returns as the top two results my canonical copy and the FAQ, and posts a link.

5. Most people lose interest while a few continue to squabble over ever-finer details.

Depending on at what point I jumped in, I could affect the speed of the above cycle, but it never changed the cycle itself. The fun of the story is following me through my own emotional cycle I felt when I worked the issue, starting with the initial "no way" to "you're having me on, right?" to "maybe...", to "dear God, this is actually happening", to "I must be going crazy", and finally to "Eureka!"

My intervention in the above cycle really wasn't adding that much to the enjoyment of the story, so I stopped doing it. (I'm not sure it's adding anything today, either, but Hacker News is an important enough forum for people I respect and care about that I thought I'd break vow and rejoin the fray this once.)

That's pretty damn close (and impressive!), but I don't think that was it :)

For one thing, I'm pretty certain the story involved recovering a deleted file's inode via some other running process's FDs.

Well, technically, the geostatistician (Did I spell that right?) was doing research that was orthogonal to the actual problem and its symptoms. In this case, the results were sufficiently odd that they sort of pointed in the right direction, but I've been sent off on wild goose chases by people skillfully applying their own particular set of skills before.

On the other hand, there's the word document with nothing but a screen shot showing half of a useless error message.

Goodness, you're determined to find fault, aren't you? (For the record in re your comment later about my "basis to call [myself] a good system admin", those claims were a) jokey, and b) fairly well-substantiated by my reputation by that time, I should think. I was published by that point and had been on several conference committees along with many who'd be reading that mailing list; I hardly needed to peacock like you seem to think I was doing.)

But I think your criticisms seem a little uninformed (or possibly over-informed by later practice to the point where you aren't considering this in the context of mid-1990's practice). Let's see...

> > And also being a good system administrator, I had written a sendmail.cf [...]

> Say what? Nobody writes a sendmail.cf from scratch, unless they are crazy.

I didn't say "from scratch". I used the m4 macros to create a cf, like everyone did at the time. Using the default file would only work if you still used email programs that read raw mbox files, had no email lists, and needed no interesting aliasing or vacation script behavior. Oh, and ran in an environment where it was reasonable to assume someone's canonical email address could be found via the equivalent of "echo "${USER}@${HOST#.}".

Very few production systems could get away with that; writing a sendmail.cf was standard practice. And with m4, you usually spoke of "writing" a file where today we'd call it "configuring" a file; either way it was taking boilerplate and replacing bits with things that were right for your situation. I assume you wouldn't have had an issue with my writing that I'd "configured" the sendmail.cf. That's all I did.

> > ... that used the nice long self-documenting option and variable names available in Sendmail 8 rather than the cryptic punctuation-mark codes that had been used in Sendmail 5

> Good system administrators stick to conservative, portable subsets of configuration and scripting languages, rather than bleeding edge stuff.

Hmm, you either weren't administering SunOS in the mid-90's or you're forgetting some details. SunOS still came with Sendmail 5 years* after best practice was to use Sendmail 8. Check out the O'Reilly Sendmail book of the time's pagecount: it was longer than the prior and the later versions because it had to document both. I'm not entirely certain SunOS (as opposed to Solaris) ever was upgraded to Sendmail 8 in the distribution; obviously the people using SunOS still so late were change-averse.

"Bleeding edge" != "the version that all but the most conservative holdouts are using". Also, remember that this was the same period we were doing the rsh/rlogin conversion to SSH. Sendmail 5 still had known security issues that were fixed in Sendmail 8. We were used to replacing system components when what the OS vendor was shipping us was literally dangerous to run.

And Sendmail 8's Sendmail 5 compatibility mode was simply there for testing; it was never intended to be used production long-term, so using a least-common-denominator sendmail.cf wouldn't have been "conservative and portable"; it would have been risky, bordering on malpractice.

> Since SunOS came with Sendmail 5, the upgraded Sendmail 8 should have been installed in some custom location with its own path so that it coexists with the stock Sendmail, and is not perturbed if the OS happens to upgrade that. > A good syadmin would stick that in some /usr/local/bin type local directory, and not overwrite /usr/bin/sendmail.

Again, either you didn't run this installation in the mid-90's or you're forgetting some details. /usr/lib/sendmail (notice the "lib"! Your referring to "/usr/bin/sendmail" suggests to me you definitely weren't running SunOS 4 or have forgotten details; sendmail was never in /usr/bin) couldn't be left alone, as other tools hardcoded that path. The actual executable was there, so symlinking couldn't be used to get around that.

It is pretty trivial to translate distance to time for something like this, so I wouldn't consider the wrong direction at all.