Live data from Hacker News

Fastly Outage

fastly.com

431–440 of 740 posts

Re: Fastly Outage

#431

Yeah so it's been mentioned in the comments already, but to everyone in Fastly right now: I feel for you. Something like this must be insanely stressful, and not just during the outage. There will be (should be) a massive post-mortem. People will be losing sleep over this for days, weeks, months. :( Edit: There seems to be a major empathy outage in this thread. Disgusted but not surprised, unfortunately.

Call me old fashioned but the latest trend of showing "empathy" for a serious incident, then proceeding to dance around the aftermath of it, whilst people give themselves a pat on back in a retro/post-mortem, isn't the way to do it. People need to be blamed, and responsibility for actions taken (without covering asses)

I recommend reading about "blameless postmortems" [1]. Our natural tendency is to look for who is responsible for an incident and point the finger of blame. Over time this leads to a cover-your-ass culture, whether you like it or not. Therefore such a tendency needs to be actively fought against to keep the focus on quality engineering and not politics.

"An atmosphere of blame risks creating a culture in which incidents and issues are swept under the rug, leading to greater risk for the organization."

[1] https://sre.google/sre-book/postmortem-culture/

Re: Fastly Outage

#434

Yeah so it's been mentioned in the comments already, but to everyone in Fastly right now: I feel for you. Something like this must be insanely stressful, and not just during the outage. There will be (should be) a massive post-mortem. People will be losing sleep over this for days, weeks, months. :( Edit: There seems to be a major empathy outage in this thread. Disgusted but not surprised, unfortunately.

Call me old fashioned but the latest trend of showing "empathy" for a serious incident, then proceeding to dance around the aftermath of it, whilst people give themselves a pat on back in a retro/post-mortem, isn't the way to do it. People need to be blamed, and responsibility for actions taken (without covering asses)

Scapegoating in those situations happens more often than not. In an operations team all problems are systemic - having to do with decision makers throughout the process, sometimes acting on perverse incentives set up by others. Blame then gets diluted but still tends to fall upon the organization responsible rather than an individual, which is where it should be. Gross negligence is not so cut and dry.

Re: Fastly Outage

#436

Earlier quoted context omitted.

The m.media-amazon.com domain (and a few other CDN'd domains that they use) are running through Fastly: nslookup m.media-amazon.com Name: media.amazon.map.fastly.net It is very interesting that they are not using CloudFront!

Good to have 3rd party redundancy, time to fail over to something else now I'd think though.

Yep they did exactly this and are now running on cloudfront

Re: Fastly Outage

#438

Earlier quoted context omitted.

GitHub Pages appears to be down too, taking an awful lot of sites offline

Github is working fine for me in Canada but others aren't. Tried without browser cache too and it works okay. EDIT: Most sites seem fixed now here in Canada. Tested stackoverflow, reddit, GitHub, PayPal, gov.UK and all worked fine.

Yeah, things are mostly back now. Including my personal GitHub Pages site :)

Re: Fastly Outage

#439

It should be resolve soon. From fastly status page: The issue has been identified and a fix is being implemented. Posted 1 minute ago. Jun 08, 2021 - 10:44 UTC

Phew!

That time to find the issue is always the stressful part. < 1 hour is pretty good for weird stuff, and fortunately the east coast of the US is barely online this early (sorry Europe!).

Re: Fastly Outage

#440

Earlier quoted context omitted.

And this is (one reason) why using images instead of actual emojis is such a stupid idea. Why, Twitter, WHY?

err, to make representations platform independent?

That sounds antithetical to the purpose of emojis.
Post reply on HN