Live data from Hacker News

Fastly Outage

fastly.com

571–580 of 740 posts

Re: Fastly Outage

#571
post #60

https://easydns.com/blog/2020/07/20/turns-out-half-the-inter... The whole idea of the internet was a distributed network impervious to most attacks. The reality is that a single failure can knock out 90% of the services people use.

The internet still works, only the websites are returning the wrong response

Interestingly, server side rendered pages worked well during the outage. Most of the issues were caused by sites that are relying too much on Javascript.

Re: Fastly Outage

#572
post #564

Earlier quoted context omitted.

Fastly Engineer 1: Seems like a common error message. Can you check stackoverflow to see if there's an easy fix? Fastly Engineer 2: I have some very bad news...

Well, with SO, at least you can search on Google and view the version cached by Google just fine. With Reddit however, these days almost all comments are locked behind “view entire discussion” or “continue this thread”. In fact, just now I searched for something for which the most relevant discussion was on Reddit; Reddit was down so I opened the cached version, and was literally greeted by five “continue this thread…

Try “site:old.Reddit.com”

Re: Fastly Outage

#573

Yeah so it's been mentioned in the comments already, but to everyone in Fastly right now: I feel for you. Something like this must be insanely stressful, and not just during the outage. There will be (should be) a massive post-mortem. People will be losing sleep over this for days, weeks, months. :( Edit: There seems to be a major empathy outage in this thread. Disgusted but not surprised, unfortunately.

[deleted]

Re: Fastly Outage

#574

Earlier quoted context omitted.

Call me old fashioned but the latest trend of showing "empathy" for a serious incident, then proceeding to dance around the aftermath of it, whilst people give themselves a pat on back in a retro/post-mortem, isn't the way to do it. People need to be blamed, and responsibility for actions taken (without covering asses)

> People need to be blamed, and responsibility for actions taken (without covering asses) This. When people talk about "HugOps", "empathy" and all that when a worldwide incident affecting a huge amount of time critical customers (e.g. trading, hft, cargo, food delivery, etc.) is happening for an hour, it has catastrophic consequences. I hope the engineers also understand the other side and why we are paying huge sums…

It's empathy towards people managing the incident, not towards the company. It's a sign of solidarity from SRE to SRE, not a sign of solidarity with a company.

Re: Fastly Outage

#575
post #560

Earlier quoted context omitted.

Same reason why Gmail uses their own emojis rather than the system ones — (as said above) branding. When you send a tweet, Twitter wants it to look identical across all devices. The classic native UI vs cross-platform UI debate in a nutshell.

Cool, so instead of actually serving text, they could also just serve up little SVGs for each letter. Because god forbid the recipient chooses a different font than Gmail!

Which is why Slack style :tofu_on_fire: emoji notation is genius

Re: Fastly Outage

#576
post #357

Earlier quoted context omitted.

While we're here.. I am a bit surprised to see how many sites use Fastly. As a dev I've always been happy with Cloudflare.

Now imagine how many sites would go down if it was CF

Where is Akamai in this comparison?

Re: Fastly Outage

#577
post #574

Earlier quoted context omitted.

> People need to be blamed, and responsibility for actions taken (without covering asses) This. When people talk about "HugOps", "empathy" and all that when a worldwide incident affecting a huge amount of time critical customers (e.g. trading, hft, cargo, food delivery, etc.) is happening for an hour, it has catastrophic consequences. I hope the engineers also understand the other side and why we are paying huge sums…

It's empathy towards people managing the incident, not towards the company. It's a sign of solidarity from SRE to SRE, not a sign of solidarity with a company.

oh right nice, and this somehow makes this all ok then? have a post mortem and all sunshine and rainbows. #noblameculture

fucking hell, you guys have no idea of how this impacted other industries.

Re: Fastly Outage

#580
post #451

Earlier quoted context omitted.

I disagree. People implemented those systems, so if you are correct that it is the systems fault, then it is also a persons fault. People must be held accountable to have good incentives to reduce such outtages in the future. I do agree though that we should always be compassionate and realistic with other humans.

> I disagree. People implemented those systems, so if you are correct that it is the systems fault, then it is also a persons fault. How do you make sure that mistakes don't happen, then? Do you blame and fire people who make mistakes, and hope that the next person put in the same spot doesn't make a mistake? Or do you figure out what caused that person to make the mistake and ensure there are processes in place so t…

This sort of culture worked at Netflix. Did they go down today?
Post reply on HN