Live data from Hacker News

What the Fastly outage can teach us about writing error messages

onlineornot.com

71–80 of 137 posts

Re: What the Fastly outage can teach us about writing error messages

#71
post #8

Earlier quoted context omitted.

Embed both in the page.

I suspect most folks, especially ones which operate at GB/s rates serving traffic would not like to serve up a 100kb base64 encoded image every time an error is thrown.

Why not? The non-error page was likely much larger.

Re: What the Fastly outage can teach us about writing error messages

#72
post #52

Isn't it time we had more error messages? I feel like we got 40x, 50x and the pioneers left lots of space for more. I quite liked the windows 0x800*** hex error numbers, though some were more useful than others.

The old, original HTTP error codes are useful because they are standardized in their meaning (even if you have to beat the difference between 404 and 410 into an SEO Consultant's head with a bat).

New error codes are only useful if they are generally understood. Maybe instead of using more codes for sub-use-cases, use the permissible error text to express these (HTTP-Status: 503 0x63F0 Data corrupt ?)

Re: What the Fastly outage can teach us about writing error messages

#74
My preferred format is simply what & why, which is the second piece of information in the list in the article.

For example: Cannot serve website. (What?) Reason: could not connect to database. (Why?)

Most of the time, it is very easy to programmatically assemble such messages. It is much harder to automatically figure out who caused it and when it will be fixed.

Re: What the Fastly outage can teach us about writing error messages

#75

Earlier quoted context omitted.

Good error messages have a template: a UUID, a description, and a suggested solution. This gives folks enough to try to solve it themselves and also enough to Google / grep with.

You can’t suggest a solution because if you were aware of the problem enough to suggest a solution you would just fix the problem. This 503 error from fastly was never meant to show to a user. You can’t do much to account for novel situations which shouldn’t ever happen.

A solution that the end-user can take. It's not unreasonable to ask user to check that their URL is correct for 404s, for instance. For 5xx errors, especially 502s and 503s, it's still helpful to tell users that there's nothing much they can do (apart from maybe monitoring the status page).

Re: What the Fastly outage can teach us about writing error messages

#76

Unfortunately the majority of internet users aren't trained in the art of reading HTTP status codes I think the majority know what 404 is, and possibly 403, but I agree about the more obscure ones. That said, I don't think it's a bad idea to rely on the "default exception handling behaviour" that the majority of users, even non-computer-literate ones, will have: they'll retry a few times, see that it doesn't work, an…

If we keep dumbing down the world for everyone instead of teaching those who don't know we will destroy ourselves. People are not stupid and if we keep assuming it we are doing a disservice to all.

No more www, no more protocol in the address bar and apple is selling iMac colors in it's commercials...

Re: What the Fastly outage can teach us about writing error messages

#77

Reminds me of the quintessential tweet about error messages https://twitter.com/cherrikissu/status/972524442600558594?s=...

Right up there with Slack etcs fake loading messages "Gearing up the dildonator" "Implicating the fairies" "Hogtying George Bush" Dude - just give me a spinner or a progress bar, and if something errors during the load out give me some sort of stack trace or error ID I can use to help

What's the first instance of joke progress messages? Is it "reticulating splines" from SimCity 2000 (1993)? Difficulty: must be presented to the user as if it were an actual status message during some process that might have real status messages, not as something explicitly fictional—despite occurring in a game, that message in SC2K qualifies.

Re: What the Fastly outage can teach us about writing error messages

#78
post #73

I'm a believer in not hiding things and logging the whole problem all the way to the end consumer: http://test.rupy.se/?id=2 Because when you develop things you are the end consumer.

Also, consumers may not understand backtraces but they generally understand how to Google them or ask other people what they mean, or tell you about them.

Technical errors are way better than just "I'm sorry we couldn't process that right now."

Re: What the Fastly outage can teach us about writing error messages

#79
post #2

Have they published a post-mortem for the outage yet? I'm curious as to what happened here

Here it is https://www.fastly.com/blog/summary-of-june-8-outage Posted about 17 hours after the incident. In short, a valid customer configuration change triggered a bug. One thing I don't see in this writeup is a commitment to ensure that customer configurations cannot break the whole system. Cloudflare does seem to make this promise with their zero trust architecture, https://www.cloudflare.com/learning/security/gl…

Zero Trust is not really applicable here. ZT is about not implicitly allowing someone elevated access just because they have access to network and requiring explicit grant based on the identity and other rules.

Fastly's downtime seems to be caused by an automatically generated config that got deployed in production as a result of change requested by a legitimate customer.

This happened to CF in its early days and I really doubt that ZT had anything to do with the fact that they do not have this kind of problem anymore. It's probably some sanity checks before they deploy updated lua scripts to their fleet of nginx's if anything.

Re: What the Fastly outage can teach us about writing error messages

#80
post #2

Have they published a post-mortem for the outage yet? I'm curious as to what happened here

Here it is https://www.fastly.com/blog/summary-of-june-8-outage Posted about 17 hours after the incident. In short, a valid customer configuration change triggered a bug. One thing I don't see in this writeup is a commitment to ensure that customer configurations cannot break the whole system. Cloudflare does seem to make this promise with their zero trust architecture, https://www.cloudflare.com/learning/security/gl…

Zero trust has little to do with individual clients and more to do with the idea of the internal "corp" network not being trusted. That is, there isn't a conventional network you VPN into at which point all traffic is trusted.

The generic topic your looking for is probably something like "customer isolation" ("service isolation" might also be relevant, but is used also in the context of "tenant isolation" which isn't really what you want). See this thread: https://news.ycombinator.com/item?id=25237836 for some talk about how AWS does "cellularization" which is a form of workload/service isolation/partitioning.

In general I don't think there's much discussion of this issue on the wider web.

Post reply on HN