Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fastly Outage

fastly.com

Post reply on HN
Print view View original

661–670 of 740 posts

Re: Fastly Outage

#661

Post by colesantiago »

nsp wrote: post #650
colesantiago wrote: post #586

Earlier quoted context omitted.

Well, while engineers are getting paid $100K/yr to post #HugOps, I know someone in HFT and their dashboard uses the Fastly service, so this has had a huge impact on them for sure. Flag and downvote all you want, you know this is true.

I suspect you'll have trouble convincing a forum of primarily engineers that a high frequency trader is more worthy of sympathy than an engineer. They're both pretty privileged jobs and HFT is not known for having tons of benefits to society

> I suspect you'll have trouble convincing a forum of primarily engineers that a high frequency trader is more worthy of sympathy than an engineer.

At least HFT traders don't get paid to spy on their own customers with trackers littered everywhere, I find that very unethical that engineers get paid to even do that sort of thing, and every damn website has these trackers because engineers put them there.

> They're both pretty privileged jobs and HFT is not known for having tons of benefits to society

So HFT firms don't have their own foundations and grants to give to charities and organisations then?

Re: Fastly Outage

#662

Post by unityByFreedom »

detaro wrote: post #621
unityByFreedom wrote: post #594

Earlier quoted context omitted.

I like Cloudflare's post mortems, and I like how they fight back against patent trolls. For me as a dev they are #1.

Do you have experience with the competitors?

I prefer tech that I can use both at work and on hobby projects at home.

To that end I've only used cloudflare and netlify. The others have too much friction to try out. I expect I would get experience on the job if necessary.

Re: Fastly Outage

#663

Post by q3k »

colesantiago wrote: post #603
austinjp wrote: post #166

Yeah so it's been mentioned in the comments already, but to everyone in Fastly right now: I feel for you. Something like this must be insanely stressful, and not just during the outage. There will be (should be) a massive post-mortem. People will be losing sleep over this for days, weeks, months. :( Edit: There seems to be a major empathy outage in this thread. Disgusted but not surprised, unfortunately.

Not my problem. Fastly should work as intended. The fault is theirs and they have said that they have failover, this worldwide outage caused by them just goes to show you that Fastly does not actually have a failover system in place. > "Fastly’s network has built-in redundancies and automatic failover routing to ensure optimal performance and uptime." - status.fastly.com Even their status page was down. Very embarras…

> Not my problem. Fastly should work as intended.

What's your SLA with them?

Just assuming things will always work because the marketing copy said so is recipe for disaster. It's hoping that things never go wrong, and when they inevitably do, being caught pants down.

Everything fails sometimes. You must know how much your SaaS provider contractually promises, ensure that any SLA breach is something financially acceptable for you, and ensure that you can handle failure time within SLA.

Re: Fastly Outage

#664

Post by colesantiago »

macintux wrote: post #612
colesantiago wrote: post #603

Earlier quoted context omitted.

Not my problem. Fastly should work as intended. The fault is theirs and they have said that they have failover, this worldwide outage caused by them just goes to show you that Fastly does not actually have a failover system in place. > "Fastly’s network has built-in redundancies and automatic failover routing to ensure optimal performance and uptime." - status.fastly.com Even their status page was down. Very embarras…

> this worldwide outage caused by them just goes to show you that Fastly does not actually have a failover system in place. I don’t know Fastly at all, but in my experience there’s no such thing as a foolproof failover system that covers all possible scenarios.

Even when they said this was a rare [0] case, they knew this case should be handled, but didn't handle it.

> or in the extremely rare case our network isn’t serving traffic.

reports also came in that this was a service configuration[1] issue, so not only there is no failover system, not even any validation automation was in place that could have prevented this.

[0] https://status.fastly.com [1] https://twitter.com/fastly/status/1402221348659814411

Re: Fastly Outage

#665

Post by Deathmax »

colesantiago wrote: post #586
austinjp wrote: post #166

Yeah so it's been mentioned in the comments already, but to everyone in Fastly right now: I feel for you. Something like this must be insanely stressful, and not just during the outage. There will be (should be) a massive post-mortem. People will be losing sleep over this for days, weeks, months. :( Edit: There seems to be a major empathy outage in this thread. Disgusted but not surprised, unfortunately.

Well, while engineers are getting paid $100K/yr to post #HugOps, I know someone in HFT and their dashboard uses the Fastly service, so this has had a huge impact on them for sure. Flag and downvote all you want, you know this is true.

And ignore the pre-agreed SLA targets and compensation for not meeting those targets that's in the contract they signed right? If you're going to say you're losing $X/minute of downtime, then either deal with it, architect around it, or negotiate the necessary SLA and compensation.

Re: Fastly Outage

#666

Post by mekkkkkk »

krzyk wrote: post #658
unityByFreedom wrote: post #601

Earlier quoted context omitted.

Fair point. Maybe Fastly is more akin to Akamai given it seems to be more enterprise-y. By market cap, Cloudflare is 26 billion, Akamai is 18, and Fastly is 6. Fastly's free offering gives you "$50 worth of traffic" whereas Cloudflare has a perpetually free option. And for Akamai you have to apply for a free trial.

This is market cap, but if you look at amount of traffic you have Akamai estimated at 15-30%, CF at 10%. So if it would go down, it would cripple vast amount of internet.

Akamai is balls deep in video streaming, which is probably the most bandwidth/traffic intense thing for a CDN to dabble with. My guess is that CF has much more diverse traffic. Hence the fallout from an interruption would be quite different.

Re: Fastly Outage

#667

Post by bombcar »

kayfox wrote: post #637
omk wrote: post #631

Earlier quoted context omitted.

Would be interesting to know what these fail over patterns are. As DNS takes a while to propagate, I thought DNS records already indicate fail over addresses.

I think only MX records indicate any priority for each additional record returned, for A records theres no indication of which records have priority over others and the usual behavior of authoritative DNS servers is to rotate the order in which records for the same thing are returned, so effectively returning more than one record for the same question results in a distribution of requests to the IPs returned rather t…

Some reliability systems change the routing for the IPs instead of updating the DNS as BGP can propagate faster than DNS caching.

Priority for A records would a nice feature.

Re: Fastly Outage

#668

Post by i386 »

Anyone want to talk about half the internet going out because one provider couldn’t keep their service up instead of SO jokes and feels for the engineers? the entire internet is like a stack of cards from the protocol to the economic model.

Re: Fastly Outage

#669

Post by colesantiago »

q3k wrote: post #663
colesantiago wrote: post #603

Earlier quoted context omitted.

Not my problem. Fastly should work as intended. The fault is theirs and they have said that they have failover, this worldwide outage caused by them just goes to show you that Fastly does not actually have a failover system in place. > "Fastly’s network has built-in redundancies and automatic failover routing to ensure optimal performance and uptime." - status.fastly.com Even their status page was down. Very embarras…

> Not my problem. Fastly should work as intended. What's your SLA with them? Just assuming things will always work because the marketing copy said so is recipe for disaster. It's hoping that things never go wrong, and when they inevitably do, being caught pants down. Everything fails sometimes. You must know how much your SaaS provider contractually promises, ensure that any SLA breach is something financially accept…

> What's your SLA with them?

Sorry what?

You've just witnessed almost the entire internet break because of a catastrophic cascading outage that affected lots of huge companies, since third party services used and trusted Fastly.

Shopify stores couldn't accept payments on their websites, Coinbase Retail/Pro transactions and trading apps failed to load, and delivery apps stopped loading all of a sudden. These are just a few that this outage has caused, and now you are trying to blame this onto me for not checking their SLA when millions were indirectly affected by this?

Fastly offered a product, their main product which is a CDN which took down lots of websites. I don't care if everything fails sometimes. There are sites that should NOT go down because of this configuration issue which they messed up.

Re: Fastly Outage

#670

Post by charonn0 »

stevewodil wrote: post #635
tomrod wrote: post #587

Earlier quoted context omitted.

Reddit's attempts at dark patterns are embarrassing from all perspectives. If you use dark patterns it's a laughably abysmal implementation. If you abhor dark patterns, it's a frustration.

It's just enough to annoy you but not enough to make everyone leave the platform

On the same day that old.reddit.com stops working I'll leave.
Post reply on HN