Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fastly Outage

fastly.com

Post reply on HN
Print view View original

451–460 of 740 posts

Re: Fastly Outage

#451

Post by q3k »

tyrex2017 wrote: post #429
q3k wrote: post #381

Earlier quoted context omitted.

The point isn't to dance around the incident, but to not blame people. You can blame systems, design, engineering culture, processes, but don't blame people. Even if someone accidentally pressed the 'destroy prod' button, that's not the fault of that person, it's the fault of that button existing and being accessible in the first place. I have no empathy for Fastly-the-company. I hate the fact that the Internet is ce…

I disagree. People implemented those systems, so if you are correct that it is the systems fault, then it is also a persons fault. People must be held accountable to have good incentives to reduce such outtages in the future. I do agree though that we should always be compassionate and realistic with other humans.

> I disagree. People implemented those systems, so if you are correct that it is the systems fault, then it is also a persons fault.

How do you make sure that mistakes don't happen, then? Do you blame and fire people who make mistakes, and hope that the next person put in the same spot doesn't make a mistake? Or do you figure out what caused that person to make the mistake and ensure there are processes in place so that next time this is less likely to happen?

Extrinsic motivators like 'we will give you a bonus' or 'we will fire you' are surprisingly bad at getting people to not fuck things up.

Re: Fastly Outage

#452

Post by berkes »

Doxin wrote: post #324
abluecloud wrote: post #197

Earlier quoted context omitted.

yeah, the internet is working perfectly. if you want to view 503 errors.

Believe it or not but "the internet" and "the world wide web" are not synonyms.

True. But the vast majority of use goes via "WWW".

For example email - the other big "internet-user" is technically not part of the WWW, but most (? I don't have any stats, just a guess) of our mailclients run on the WWW, nonetheless.

Re: Fastly Outage

#453

Post by oneeyedpigeon »

spzb wrote: post #392
patentatt wrote: post #322

Earlier quoted context omitted.

Edit: I didn’t mean anything negative here! Just slightly shocked that as the UK is opening up under 30 vaccinations, the US is struggling to find any more willing takers. It’s really probably a sign that there’s fewer anti-vaxxers in the UK more than anything. And that universal healthcare is more efficient at distribution than an inherently for profit system. I don’t know, but I just didn’t realize it was so differ…

UK is ahead of the US https://ourworldindata.org/covid-vaccinations

For one dose. For full vaccination, the US is (slightly) ahead according to that same site.

Re: Fastly Outage

#454

Post by ceejayoz »

evouga wrote: post #373
ImpactStrafe wrote: post #337

Earlier quoted context omitted.

Fastly is a Content Distribution Network (CDN). Basically the closer the server serving the webpage is to the end user the faster it is for the end user to see and interact with. But running servers all over the world 1) isn't efficient 2) costs a lot of money. So a few companies (fastly, cloud flare, akamai) figured, hey, why don't we build a bunch of small data centers all over the world and then provide a distribu…

Thanks. That makes sense. Wouldn’t you build in a failsafe that bypasses Fastly and sends traffic to your own servers in the case of this kind of outage? Or outages are so rare that it’s not worth the trouble?

Many sites do this; Amazon's failed over to their own servers for images for me, it appears. It typically just takes some human intervention, I suspect.

Re: Fastly Outage

#455

Post by nkozyra »

maest wrote: post #125
lpmitchell wrote: post #9

This seems to be impacting a number of huge sites, including the UK government website[0]. [0] https://www.gov.uk/ https://m.media-amazon.com/ https://pages.github.com/ https://www.paypal.com/ https://stackoverflow.com/ https://nytimes.com/ Edit: Fastly's incident report status page: https://status.fastly.com/incidents/vpk0ssybt3bj

Amusingly, the Stackoverflow 503 page has a typo: Error 503 Service Unavailable Service Unavailable Guru *Mediation*: Details: cache-lon4236-LON 1623146049 854282175 Varnish cache server

This seems like it's intentional given the context.

Re: Fastly Outage

#456

Post by npteljes »

evouga wrote: post #373
ImpactStrafe wrote: post #337

Earlier quoted context omitted.

Fastly is a Content Distribution Network (CDN). Basically the closer the server serving the webpage is to the end user the faster it is for the end user to see and interact with. But running servers all over the world 1) isn't efficient 2) costs a lot of money. So a few companies (fastly, cloud flare, akamai) figured, hey, why don't we build a bunch of small data centers all over the world and then provide a distribu…

Thanks. That makes sense. Wouldn’t you build in a failsafe that bypasses Fastly and sends traffic to your own servers in the case of this kind of outage? Or outages are so rare that it’s not worth the trouble?

That's the fallback, but the original stack is not designed with the volume of traffic in mind. So it gets overwhelmed very quickly and makes the website practically unavailable.

Re: Fastly Outage

#457

Post by diveanon »

dspillett wrote: post #112
threeseed wrote: post #45

Shopify's CDN is down. Which is causing $15+ million in lost product sales for every hour of outage. Not to mention the loss of any new customers.

StackOverflow and all the StackExchange family of sites are down. I suspect the lost productivity from that will be more costly over the whole economy than potential lost sales via Shopify. People can go back to shopify so those transactions not definitely blocked for ever, any time "lost" due to reference resources being unavailable can't so easily be claimed back.

Believe it or not, but there are developers out there that read the docs.

Re: Fastly Outage

#458

Post by JosephK »

Extremely long call, but what are the chances this turns out connected to the raids on organised crime using the An0m app that started today?

Re: Fastly Outage

#460

Post by Symbiote »

spzb wrote: post #346
jakub_g wrote: post #290

Earlier quoted context omitted.

https://www.bbc.com/news/technology-57399628 "A number of leading media websites are currently not working, including the Guardian, Financial Times, Independent and the New York Times."

Not that the BBC are gloating that they're still up

The BBC.com site was down for about 10-15 minutes.
Post reply on HN