Live data from Hacker News

Fastly Outage

fastly.com

601–610 of 740 posts

Re: Fastly Outage

#601
post #576
post #357

Earlier quoted context omitted.

Now imagine how many sites would go down if it was CF

Where is Akamai in this comparison?

Fair point. Maybe Fastly is more akin to Akamai given it seems to be more enterprise-y. By market cap, Cloudflare is 26 billion, Akamai is 18, and Fastly is 6.

Fastly's free offering gives you "$50 worth of traffic" whereas Cloudflare has a perpetually free option. And for Akamai you have to apply for a free trial.

Re: Fastly Outage

#602
post #232
post #60

https://easydns.com/blog/2020/07/20/turns-out-half-the-inter... The whole idea of the internet was a distributed network impervious to most attacks. The reality is that a single failure can knock out 90% of the services people use.

The Web (World Wide Web) build atop of the Internet, is not impervious. ps. "The Internet was build to survive attacks" is not true. It's a myth made popular by Robert Cringely in the early 1990s. The Arpanet was simply a protocol for mainframes used by computer scientists to connect. The Internet is relatively resilient against attacks, but that was not the "whole idea". It was not in the design at all. Bob Taylor:…

BGP (the protocol underpinning the internet) is built entirely for avoiding outages of any size.

Even email has a method baked into to the protocol for handling failure.

Fallbacks are good, baking in resiliency is better.

Re: Fastly Outage

#603

Yeah so it's been mentioned in the comments already, but to everyone in Fastly right now: I feel for you. Something like this must be insanely stressful, and not just during the outage. There will be (should be) a massive post-mortem. People will be losing sleep over this for days, weeks, months. :( Edit: There seems to be a major empathy outage in this thread. Disgusted but not surprised, unfortunately.

Not my problem. Fastly should work as intended.

The fault is theirs and they have said that they have failover, this worldwide outage caused by them just goes to show you that Fastly does not actually have a failover system in place.

> "Fastly’s network has built-in redundancies and automatic failover routing to ensure optimal performance and uptime." - status.fastly.com

Even their status page was down. Very embarrassing, Fastly did not work as advertised and mislead its customers.

Edit: Offended flaggers circling around silencing misled Fastly customers. How pathetic.

Re: Fastly Outage

#604
post #410

This is one of the things that excites me about IPFS: in a world of decentralized data storage, yes self-hosting and control over your data is nice and all, but serious resilience to most random infrastructure outages is a much bigger deal. It's still early days, but I'm hopeful that it can provide a real solution to today's CDN centralization.

Agree, but currently, ipfs would serve as a fallback, since it's about files. Decentralized/distributed generally has slower network performance. Unless most nodes are high performance, I guess? Personally I think a distributed database system, where entries are being made redundant in something like a blockchain+dht, would be a good start? Decentralizing the internet works if it financially makes sense for platforms…

> Agree, but currently, ipfs would serve as a fallback, since it's about files.

Isn't a CDN fundamentally all about files too?

> Decentralized/distributed generally has slower network performance. Unless most nodes are high performance, I guess?

There is definitely more work to do here before this is really useful, but it's well within the realm of things that IPFS should be able to do at reasonable performance for production sites in future. Good performance still requires a serious CDN node network similar to traditional CDNs today (to seed your content for day to day use) but with IPFS if that CDN goes down then existing users on your site can _also_ serve the site to other nearby users directly, or other CDNs can serve your site too, etc etc. Your DNS wouldn't be linked to any specific CDN in any way, just to the hash of the content itself, so anybody could serve it.

> Decentralizing the internet works if it financially makes sense for platforms to build such tools.

There's a platform company called Fleek who already do this today: https://fleek.co/hosting/ (no affiliation, and I've never even used the product, just looks cool). Seems to be designed as a Netlify competitor: push code with git and it builds it into static content and then deploys to IPFS.

The benefits don't exist today of course, because no browsers natively support IPFS, so most users can only access the content via an IPFS gateway, which means you're back to fully centralized server infrastructure again... If we can get IPFS support into browsers though then fully decentralized CDN infrastructure for the web is totally possible.

Re: Fastly Outage

#605

This seems to be impacting a number of huge sites, including the UK government website[0]. [0] https://www.gov.uk/ https://m.media-amazon.com/ https://pages.github.com/ https://www.paypal.com/ https://stackoverflow.com/ https://nytimes.com/ Edit: Fastly's incident report status page: https://status.fastly.com/incidents/vpk0ssybt3bj

All of these work from here in Grenoble, France...

That's the problem with these black-box cloud offerings, that you can never know what will work (or not) and from where. You get semi-random, pseudo-localized outages that are not accounted for in all the 9's of availability.

With a standard TCP/UDP session, it mostly just works or doesn't and you can get a proper traceroute to know what's up. With these fancy CDNs, there's a whole new can of worms to deal with and from a client's perspective you have no clue what's happening because it's all taking place in their private network space where we have no "looking glass".

Fuck the cloud, i want real Internet.

Re: Fastly Outage

#606
post #352

This is one of the things that excites me about IPFS: in a world of decentralized data storage, yes self-hosting and control over your data is nice and all, but serious resilience to most random infrastructure outages is a much bigger deal. It's still early days, but I'm hopeful that it can provide a real solution to today's CDN centralization.

I'm pretty sure you can serve hundreds if not thousands of users from a single Raspberry Pi

I mean, yes, absolutely, and that works to start with, but I'm willing to bet the overall uptime and performance of a raspberry pi in your living room is quite a bit worse that Fastly's :-).

Re: Fastly Outage

#607
post #332

Earlier quoted context omitted.

Over one issue that highlights they have an abundance of top level customers? Interesting strategy when it's already at a low.

One issue that should have been mitigated at least by Fastly; worse if the client has to do it. They proudly stated this from their own website to their customers: > "Fastly’s network has built-in redundancies and automatic failover routing to ensure optimal performance and uptime." If that isn't one huge lie, I don't know what is.

It is definitely a lie, but it's the same lie sold by all cloud offerings. Can you name a single cloud/CDN operator without downtimes?

It's normal to have downtimes but they are usually scheduled and quick (think <10 minutes per month for rebooting and/or hardware parts replacement). I'm pretty sure most non-profit hosts like disroot.org or globenet.org have similar or better 9's than all these fancy cloud services.

Re: Fastly Outage

#608
post #362

Earlier quoted context omitted.

How is having a large chunk of the internet using the same CDN provider not "centralizing"? It's not a hard monopoly obviously but still it meets the definition of centralization.

how is private companies choosing to use a common supplier in a competitive market centralization? monopolies are not centralization either. you need to read a better book.

How is a market competitive when there's a quasi-monopoly on infrastructure? When public money is used to irrigate the same corporations with huge $$$, while non-profit network operators are left to rot?

Re: Fastly Outage

#609
post #332

Earlier quoted context omitted.

Over one issue that highlights they have an abundance of top level customers? Interesting strategy when it's already at a low.

One issue that should have been mitigated at least by Fastly; worse if the client has to do it. They proudly stated this from their own website to their customers: > "Fastly’s network has built-in redundancies and automatic failover routing to ensure optimal performance and uptime." If that isn't one huge lie, I don't know what is.

It can have all these things and still fail, suggesting otherwise would be fairly naive.

Re: Fastly Outage

#610
post #564

Earlier quoted context omitted.

Fastly Engineer 1: Seems like a common error message. Can you check stackoverflow to see if there's an easy fix? Fastly Engineer 2: I have some very bad news...

Well, with SO, at least you can search on Google and view the version cached by Google just fine. With Reddit however, these days almost all comments are locked behind “view entire discussion” or “continue this thread”. In fact, just now I searched for something for which the most relevant discussion was on Reddit; Reddit was down so I opened the cached version, and was literally greeted by five “continue this thread…

I wanted to suggest site:old.reddit.com since I use that version with automatic redirect, but this:

https://old.reddit.com/robots.txt

is very different from this:

https://reddit.com/robots.txt

I guess there is a market for search engine (maybe accessed through tor) which does not care about robots.txt, DMCAs, right to be forgotten etc. Bootstrapping it should not be that hard since it can also provide better results for some queries since nobody is fighting about the position until it's widely known.

I'm not sure how far are we from being able to do full text internet search. Or rather even quote search, preferably some fuzziness options. That would be cool, Google's quotation marks were really neat back when they were working.

Post reply on HN