Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fastly Outage

fastly.com

Post reply on HN
Print view View original

681–690 of 740 posts

Re: Fastly Outage

#681

Post by colesantiago »

detaro wrote: post #677
colesantiago wrote: post #669

Earlier quoted context omitted.

> What's your SLA with them? Sorry what? You've just witnessed almost the entire internet break because of a catastrophic cascading outage that affected lots of huge companies, since third party services used and trusted Fastly. Shopify stores couldn't accept payments on their websites, Coinbase Retail/Pro transactions and trading apps failed to load, and delivery apps stopped loading all of a sudden. These are just…

> I don't care if everything fails sometimes You can say you don't care for reality, but it's not going to help you have better systems. > There are sites that should NOT go down Then they surely either engineered their system to not 100% rely on Fastly or negotiated appropriate terms with Fastly (Or decided Fastly going down was an acceptable business risk, which it is for nearly everybody). Everything else would be…

> You can say you don't care for reality, but it's not going to help you have better systems.

No where in my sentence I said this so quit the strawman argument.

I know a client using a service that has 100% uptime for the year, that also relies on huge clients, I don't understand why Fastly can't guarantee at the very least and a failover system to counteract this, but clearly didn't work. (or even existed)

> (Or decided Fastly going down was an acceptable business risk, which it is for nearly everybody).

Then why did this cascade to almost everybody even indirectly? Surely their advertised failover system would have prevented this from prolonging further but lasted longer than it should have.

I don't think a store, exchange or trading desk not accepting payments from people for an hour is acceptable at all.

Re: Fastly Outage

#682

Post by stuff4ben »

colesantiago wrote: post #580
q3k wrote: post #451

Earlier quoted context omitted.

> I disagree. People implemented those systems, so if you are correct that it is the systems fault, then it is also a persons fault. How do you make sure that mistakes don't happen, then? Do you blame and fire people who make mistakes, and hope that the next person put in the same spot doesn't make a mistake? Or do you figure out what caused that person to make the mistake and ensure there are processes in place so t…

This sort of culture worked at Netflix. Did they go down today?

Lets hope you don't ever go into management. You clearly have no idea how to motivate and retain people or have any insight on how hard it is to hire good people to begin with. And no, I'm pretty certain this is not how Netflix's culture is.

Re: Fastly Outage

#683

Post by colesantiago »

detaro wrote: post #678
colesantiago wrote: post #676

Earlier quoted context omitted.

So why didn't the 'automatic failover' kick in during the outage? Where was it then? I don't see anything about 're-routing traffic' anywhere in the status page [0] [0] https://status.fastly.com/incidents/vpk0ssybt3bj

We don't know, but the usual scenarios would be "issue impacts failover mechanism too", "failover mechanism overloads other system components leading to cascading failure" or "something causes failover mechanism to to think all is fine".

> We don't know...

So, the rarest of cases (our network isn’t serving traffic) just happened right now, and their failover system just took a snooze then, but 'it exists apparently' according to you.

Tell that the huge clients that lost sales because of this, and all you have to say is: "wE DoN'T kNoW..."

Re: Fastly Outage

#685

Post by marvin »

stevewodil wrote: post #635
tomrod wrote: post #587

Earlier quoted context omitted.

Reddit's attempts at dark patterns are embarrassing from all perspectives. If you use dark patterns it's a laughably abysmal implementation. If you abhor dark patterns, it's a frustration.

It's just enough to annoy you but not enough to make everyone leave the platform

They've actually done a masterful job of finding this balance. I've been on reddit for 15 years and would have quit if they didn't leave the old interface available.

Re: Fastly Outage

#686

Post by fareesh »

How does one design a system that has a redundancy for when the CDN goes down? Paying for more than one CDN is probably too expensive isn't it?

Re: Fastly Outage

#687

Post by colesantiago »

stuff4ben wrote: post #682
colesantiago wrote: post #580

Earlier quoted context omitted.

This sort of culture worked at Netflix. Did they go down today?

Lets hope you don't ever go into management. You clearly have no idea how to motivate and retain people or have any insight on how hard it is to hire good people to begin with. And no, I'm pretty certain this is not how Netflix's culture is.

> pretty certain this is not how Netflix's culture is.

> pReTtY CeRtAiN

This, the wording in of itself shows you have absolutely no clue whatsoever at all of Netflix's culture.

Re: Fastly Outage

#688

Post by dm319 »

austinjp wrote: post #166

Yeah so it's been mentioned in the comments already, but to everyone in Fastly right now: I feel for you. Something like this must be insanely stressful, and not just during the outage. There will be (should be) a massive post-mortem. People will be losing sleep over this for days, weeks, months. :( Edit: There seems to be a major empathy outage in this thread. Disgusted but not surprised, unfortunately.

Empathy is hard to find around here, maybe someone needs to study it. Is it a feature of people in tech? Don't remember much being on slashdot either.

Re: Fastly Outage

#689

Post by the_only_law »

dm319 wrote: post #688
austinjp wrote: post #166

Yeah so it's been mentioned in the comments already, but to everyone in Fastly right now: I feel for you. Something like this must be insanely stressful, and not just during the outage. There will be (should be) a massive post-mortem. People will be losing sleep over this for days, weeks, months. :( Edit: There seems to be a major empathy outage in this thread. Disgusted but not surprised, unfortunately.

Empathy is hard to find around here, maybe someone needs to study it. Is it a feature of people in tech? Don't remember much being on slashdot either.

Just wait till they figure out how to make money off it.

Re: Fastly Outage

#690

Post by petepete »

charonn0 wrote: post #670
stevewodil wrote: post #635

Earlier quoted context omitted.

It's just enough to annoy you but not enough to make everyone leave the platform

On the same day that old.reddit.com stops working I'll leave.

That and third party access to their API.

Sync is so much better than the official app it's not even funny.

Post reply on HN