Earlier quoted context omitted.
Now imagine how many sites would go down if it was CF
interesting thought ... a new type of 'to big to fail' ?
No more than a transatlantic cable...
Fastly Outage
671–680 of 740 posts
Re: Fastly Outage
#672Earlier quoted context omitted.
> this worldwide outage caused by them just goes to show you that Fastly does not actually have a failover system in place. I don’t know Fastly at all, but in my experience there’s no such thing as a foolproof failover system that covers all possible scenarios.
Even when they said this was a rare [0] case, they knew this case should be handled, but didn't handle it. > or in the extremely rare case our network isn’t serving traffic. reports also came in that this was a service configuration[1] issue, so not only there is no failover system, not even any validation automation was in place that could have prevented this. [0] https://status.fastly.com [1] https://twitter.com/fa…
Systems failing is not evidence of systems not existing.
Re: Fastly Outage
#673Earlier quoted context omitted.
I wanted to suggest site:old.reddit.com since I use that version with automatic redirect, but this: https://old.reddit.com/robots.txt is very different from this: https://reddit.com/robots.txt I guess there is a market for search engine (maybe accessed through tor) which does not care about robots.txt, DMCAs, right to be forgotten etc. Bootstrapping it should not be that hard since it can also provide better results…
Wonder what the story is behind these two... User-Agent: bender Disallow: /my_shiny_metal_ass User-Agent: Gort Disallow: /earth
Just some fun humor we added for other nerds who read robots.txt files.
Re: Fastly Outage
#674Earlier quoted context omitted.
Wonder what the story is behind these two... User-Agent: bender Disallow: /my_shiny_metal_ass User-Agent: Gort Disallow: /earth
I reod some people use false slugs in the robots.txt as a honey pot of sorts. IPs that actually read the robots.txt, ignore the disallow, and still access the uri are outright banned.
Ha, that would have been a really smart idea! Sadly we didn't think of that at the time. But we had other honey pot URLs.
Re: Fastly Outage
#675Re: Fastly Outage
#676Earlier quoted context omitted.
Even when they said this was a rare [0] case, they knew this case should be handled, but didn't handle it. > or in the extremely rare case our network isn’t serving traffic. reports also came in that this was a service configuration[1] issue, so not only there is no failover system, not even any validation automation was in place that could have prevented this. [0] https://status.fastly.com [1] https://twitter.com/fa…
Systems failing is not evidence of systems not existing.
So why didn't the 'automatic failover' kick in during the outage? Where was it then? I don't see anything about 're-routing traffic' anywhere in the status page [0]
Re: Fastly Outage
#677Earlier quoted context omitted.
> Not my problem. Fastly should work as intended. What's your SLA with them? Just assuming things will always work because the marketing copy said so is recipe for disaster. It's hoping that things never go wrong, and when they inevitably do, being caught pants down. Everything fails sometimes. You must know how much your SaaS provider contractually promises, ensure that any SLA breach is something financially accept…
> What's your SLA with them? Sorry what? You've just witnessed almost the entire internet break because of a catastrophic cascading outage that affected lots of huge companies, since third party services used and trusted Fastly. Shopify stores couldn't accept payments on their websites, Coinbase Retail/Pro transactions and trading apps failed to load, and delivery apps stopped loading all of a sudden. These are just…
> I don't care if everything fails sometimes
You can say you don't care for reality, but it's not going to help you have better systems.
> There are sites that should NOT go down
Then they surely either engineered their system to not 100% rely on Fastly or negotiated appropriate terms with Fastly (Or decided Fastly going down was an acceptable business risk, which it is for nearly everybody). Everything else would be negligent, and surely nobody would be negligent when operating a site that "should NOT go down"?
Re: Fastly Outage
#678Earlier quoted context omitted.
Systems failing is not evidence of systems not existing.
So why didn't the 'automatic failover' kick in during the outage? Where was it then? I don't see anything about 're-routing traffic' anywhere in the status page [0] [0] https://status.fastly.com/incidents/vpk0ssybt3bj
We don't know, but the usual scenarios would be "issue impacts failover mechanism too", "failover mechanism overloads other system components leading to cascading failure" or "something causes failover mechanism to to think all is fine".
Re: Fastly Outage
#679Earlier quoted context omitted.
Yeah it's crazy how bad user-hostile reddit.com has become. Fortunately old.reddit.com is still available, but for how long? If only Javascript did not exist, it would be impossible for UX people to come up with something that bad.
Yeah I’m going to stop using the platform when they get rid of this . Not interested
Why wait? You are wasting your life away.
Re: Fastly Outage
#680Earlier quoted context omitted.
Well, while engineers are getting paid $100K/yr to post #HugOps, I know someone in HFT and their dashboard uses the Fastly service, so this has had a huge impact on them for sure. Flag and downvote all you want, you know this is true.
And ignore the pre-agreed SLA targets and compensation for not meeting those targets that's in the contract they signed right? If you're going to say you're losing $X/minute of downtime, then either deal with it, architect around it, or negotiate the necessary SLA and compensation.
It's not me you should be telling this to though, if you know someone at Fastly, perhaps you should reminding them that.
I expect huge clients to be knocking on Fastly's door lining up for answers because of this.