Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fastly Outage

fastly.com

Post reply on HN
Print view View original

531–540 of 740 posts

Re: Fastly Outage

#531

Post by aliasEli »

lpmitchell wrote: post #9

This seems to be impacting a number of huge sites, including the UK government website[0]. [0] https://www.gov.uk/ https://m.media-amazon.com/ https://pages.github.com/ https://www.paypal.com/ https://stackoverflow.com/ https://nytimes.com/ Edit: Fastly's incident report status page: https://status.fastly.com/incidents/vpk0ssybt3bj

What's far worse than half of the internet being down was that Hacker News also had problems. If I waited long enough on a comments page I got an error message. I don't quite understand what happened there. The communication between my system and HN must have been working otherwise I would never have gotten an error message, so it must have been some internal HN problem. But since HN should only need its own internal "database" to generate comment pages, I don't understand why it should be impacted by the Fastly problems.

Re: Fastly Outage

#532

Post by vlovich123 »

amirhirsch wrote: post #522
caseymarquis wrote: post #408

Earlier quoted context omitted.

Fastly Engineer 1: Seems like a common error message. Can you check stackoverflow to see if there's an easy fix? Fastly Engineer 2: I have some very bad news...

Stack Overflow is down, can someone tell me how to declare a static multidimensional array in C++?

Google and DDG surface SO results cached within their own page. Here’s the copied answer:

int main() { int arr[100][200][100]; // allocate on the stack

    return 0;
}

Re: Fastly Outage

#533

Post by altacc »

tyrex2017 wrote: post #429
q3k wrote: post #381

Earlier quoted context omitted.

The point isn't to dance around the incident, but to not blame people. You can blame systems, design, engineering culture, processes, but don't blame people. Even if someone accidentally pressed the 'destroy prod' button, that's not the fault of that person, it's the fault of that button existing and being accessible in the first place. I have no empathy for Fastly-the-company. I hate the fact that the Internet is ce…

I disagree. People implemented those systems, so if you are correct that it is the systems fault, then it is also a persons fault. People must be held accountable to have good incentives to reduce such outtages in the future. I do agree though that we should always be compassionate and realistic with other humans.

v1. "It's Bob's fault and so we fired Bob."

v2. "The issue was caused by a previously unidentified pathway that caused a feedback loop and overloaded our servers in a cascading fashion (or whatever). We have implemented a fix for this and updated our testing and deployment processes to stop similar cascades."

Which solves the problem long term?

As an architect making product choices, v2 wins every time.

(With the caveat that if the cause was something that reveals a fundamental problem with the larger processes/professionalism/culture of the company, especially to do with security concerns, then I'm not buying that product and migrating away if we already use it.

Re: Fastly Outage

#534

Post by jtdev »

austinjp wrote: post #166

Yeah so it's been mentioned in the comments already, but to everyone in Fastly right now: I feel for you. Something like this must be insanely stressful, and not just during the outage. There will be (should be) a massive post-mortem. People will be losing sleep over this for days, weeks, months. :( Edit: There seems to be a major empathy outage in this thread. Disgusted but not surprised, unfortunately.

Our fathers and mothers put man on the moon… we build shitty software that helps the technocrats sell more junk to the masses.

Re: Fastly Outage

#535

Post by dreamer7 »

lpmitchell wrote: post #9

This seems to be impacting a number of huge sites, including the UK government website[0]. [0] https://www.gov.uk/ https://m.media-amazon.com/ https://pages.github.com/ https://www.paypal.com/ https://stackoverflow.com/ https://nytimes.com/ Edit: Fastly's incident report status page: https://status.fastly.com/incidents/vpk0ssybt3bj

Firebase hosting has been affected as well

Re: Fastly Outage

#536

Post by zhte415 »

patentatt wrote: post #322
madeofpalk wrote: post #287

Earlier quoted context omitted.

> [0] https://www.gov.uk/ Just checked, thank god the NHS vaccine site is still available - vaccines just got rolled out for under 30s today.

Edit: I didn’t mean anything negative here! Just slightly shocked that as the UK is opening up under 30 vaccinations, the US is struggling to find any more willing takers. It’s really probably a sign that there’s fewer anti-vaxxers in the UK more than anything. And that universal healthcare is more efficient at distribution than an inherently for profit system. I don’t know, but I just didn’t realize it was so differ…

Both the UK and US are doing well.

https://ig.ft.com/coronavirus-vaccine-tracker for reference.

What's important is important to share vaccines with all nations, and non-nations.

Re: Fastly Outage

#537

Post by Jare »

paublyrne wrote: post #528
kylen wrote: post #494

Earlier quoted context omitted.

I hear that you're suggesting that those involved shouldnt feel bad because its a systemic / just a job / etc. But the reality is that incidents like this can be very traumatic for those involved and thats not something they can control. If it was that simple to manage, depression and anxiety would not be a thing. Think its best to show a large amount of support and empathy for the individuals having a really bad day…

I don't see how it's so traumatic for the engineers involved, unless the company culture in Fastly is really awful and there are punitive repercussions, or attempts to pin responsbility on individuals rather than systems, which I doubt. Many here have been responsible for web service outages albeit on much smaller scales, and in my experience it feels awful while it's happening but you quickly forget about it because…

> on much smaller scales

> you quickly forget about it because so does everyone else

This is definitely not the case here, and the experiences are bound to be very different.

Re: Fastly Outage

#538

Post by abluecloud »

alexchamberlain wrote: post #529

Stupid question: why didn't sites "just" fail over to their actual servers to handle the traffic, albeit slowly? I guess they won't be sized to handle the load in a lot of cases, and Fastly was responding, so DNS fail over didn't work?

yeah. the dns was up. the problem was the servers weren't able to proxy the traffic. Also, as you say, you'll probably end up bringing down the upstream servers if you just fail open (and not even sure that'd be a possibility with fastly in it's "down" state that we saw).

Re: Fastly Outage

#539

Post by fukmbas »

iso1631 wrote: post #60

https://easydns.com/blog/2020/07/20/turns-out-half-the-inter... The whole idea of the internet was a distributed network impervious to most attacks. The reality is that a single failure can knock out 90% of the services people use.

Yet another reason SaaS is trash

Re: Fastly Outage

#540

Post by austinjp »

mothsonasloth wrote: post #333
austinjp wrote: post #166

Yeah so it's been mentioned in the comments already, but to everyone in Fastly right now: I feel for you. Something like this must be insanely stressful, and not just during the outage. There will be (should be) a massive post-mortem. People will be losing sleep over this for days, weeks, months. :( Edit: There seems to be a major empathy outage in this thread. Disgusted but not surprised, unfortunately.

Call me old fashioned but the latest trend of showing "empathy" for a serious incident, then proceeding to dance around the aftermath of it, whilst people give themselves a pat on back in a retro/post-mortem, isn't the way to do it. People need to be blamed, and responsibility for actions taken (without covering asses)

"Call me old-fashioned but..." is a dog-whistle harking back to "better days" that never existed.

Emapthy and responsiblity are not mutually exclusive.

Post reply on HN