Just today I had some code that helpfully said "should never get here" and aborted. It left us to figure out what the possible options of how we could end up in that spot "we should never get to". It was maddening because it's ~12 year old code that had some library updates. A little bit more help like the error code or really *anything* would've been helpful.
What the Fastly outage can teach us about writing error messages
61–70 of 137 posts
Re: What the Fastly outage can teach us about writing error messages
#62Earlier quoted context omitted.
Very true, at the individual error level. That said, Fastly fixed the entire issue in less time than it would have taken AWS to update the status page to the blue diamond, so there’s that.
"Service is experiencing increased error rates" - Error rates increased from 0 to 100%
No service has 0% error rate, and 100% errors doesn't even mean much without context (you never receive all possible errors at once).
Re: What the Fastly outage can teach us about writing error messages
#63Reminds me of the quintessential tweet about error messages https://twitter.com/cherrikissu/status/972524442600558594?s=...
Right up there with Slack etcs fake loading messages "Gearing up the dildonator" "Implicating the fairies" "Hogtying George Bush" Dude - just give me a spinner or a progress bar, and if something errors during the load out give me some sort of stack trace or error ID I can use to help
[deleted]
Re: What the Fastly outage can teach us about writing error messages
#64Earlier quoted context omitted.
these seem solvable. there's no reason that a good error message AND pertinent support information are mutually exclusive. show them both. show the errors that make sense for that action and that error. no one is saying that there should be a small list of "approved" errors that everyone would see. show what makes sense in that situation, and don't show something that doesn't help. you don't need to have prophetic in…
Showing the error itself only opens the provider to information disclosure - with the various automated infrastructure monitoring systems running on Fastly or CF they know when and where errors are happening with full traces that help them debug. > there's no reason that a good error message AND pertinent support information are mutually exclusive. show them both As the OP stated, there is quite literally nothing sup…
> Showing the error itself only opens the provider to information disclosure
No offence but as an end user I really despise this attitude. Sure there might be automatic monitoring at a place like fastly, but even they can use help in tracking down the problem. Also, if the error is distinct people can put it into Google and call on the vast power on the Internet to help figure out how to fix it or work around it, especially on smaller services where a fix may or may not be forthcoming anytime soon. A good error message leads to a Stackexchange page leads to a solution. A vague error leads to a support call with a bewildered frontline tech and a lot of work for some sorry engineer who has to dig through log files.
Re: What the Fastly outage can teach us about writing error messages
#65A great example of content marketing.
Re: What the Fastly outage can teach us about writing error messages
#66So that looks like a stock Varnish 503. I'm pretty sure the magic behind fastly is a ton of Varnish cache. What's interesting (VCL can be unforgiving), also that it was a big central stuff up. Which makes me think they have layers and layers of caching in depth and some top level config just blacked out everything. I kind of would have thought they'd have lots of small instances.
> So that looks like a stock Varnish 503 Looks like it, except that Varnish has it spelled “Guru Meditation”, not “Guru Mediation”. Anyone know why that would be? https://github.com/varnishcache/varnish-cache/search?q=medit...
this question was already answered in the fastly outage thread.
It was to identify fastly's Varnish vs customer's Varnish.
Re: What the Fastly outage can teach us about writing error messages
#67So that looks like a stock Varnish 503. I'm pretty sure the magic behind fastly is a ton of Varnish cache. What's interesting (VCL can be unforgiving), also that it was a big central stuff up. Which makes me think they have layers and layers of caching in depth and some top level config just blacked out everything. I kind of would have thought they'd have lots of small instances.
> So that looks like a stock Varnish 503 Looks like it, except that Varnish has it spelled “Guru Meditation”, not “Guru Mediation”. Anyone know why that would be? https://github.com/varnishcache/varnish-cache/search?q=medit...
https://twitter.com/dormando/status/1402466173778677764
my fault. I would sometimes monitor for "guru mediations" popping up to tell if we were throwing errors without it being caught by other systems. Among other reasons.
Re: What the Fastly outage can teach us about writing error messages
#68So that looks like a stock Varnish 503. I'm pretty sure the magic behind fastly is a ton of Varnish cache. What's interesting (VCL can be unforgiving), also that it was a big central stuff up. Which makes me think they have layers and layers of caching in depth and some top level config just blacked out everything. I kind of would have thought they'd have lots of small instances.
This is not true Edit: trying to reverse engineer the “magic”, architecture, and the failure from a varnish error message is folly, and misleads others. How do I know the comment is patently false? I ran those teams at Fastly for 3 years.
Alternately, document your 'magic architecture' in public so that people don't have to offend you or other employees of the firm by making incorrect assumptions based on the little you do show in public.
Also helps others pick up if you do go out of business or end up acquired, and before you do it even leads to a more level playing field.
However I guess a level playing field wouldn't have given you a NYSE listing... so I guess you're just being selfish?
Re: What the Fastly outage can teach us about writing error messages
#69Earlier quoted context omitted.
Showing the error itself only opens the provider to information disclosure - with the various automated infrastructure monitoring systems running on Fastly or CF they know when and where errors are happening with full traces that help them debug. > there's no reason that a good error message AND pertinent support information are mutually exclusive. show them both As the OP stated, there is quite literally nothing sup…
You could do a mix of both: give information useful for support (trace id, error code, etc) but don't explicitly say "send an email to support@bigcorp.com / call 01 23 45 67". The people on the tech side who are the CDN's clients will know who and how to call for support and if it's an end-user they'll be able to transmit this info. Bonus points for the tech guys to figure that there's an issue on their end and that…
> What now?
Event Viewer. (You already know, but for the benefit of whomever else.) Hiding things here stinks if you don't know there's something to see there.
Re: What the Fastly outage can teach us about writing error messages
#70Earlier quoted context omitted.
Very true, at the individual error level. That said, Fastly fixed the entire issue in less time than it would have taken AWS to update the status page to the blue diamond, so there’s that.
"Service is experiencing increased error rates" - Error rates increased from 0 to 100%
Or 50% error rates in case one of two nodes is down and you keep calling the faulty node.
But the faulty node should not be called in a well designed system.