Live data from Hacker News

Fastly Outage

fastly.com

731–740 of 740 posts

Re: Fastly Outage

#732
post #564

Earlier quoted context omitted.

Fastly Engineer 1: Seems like a common error message. Can you check stackoverflow to see if there's an easy fix? Fastly Engineer 2: I have some very bad news...

Well, with SO, at least you can search on Google and view the version cached by Google just fine. With Reddit however, these days almost all comments are locked behind “view entire discussion” or “continue this thread”. In fact, just now I searched for something for which the most relevant discussion was on Reddit; Reddit was down so I opened the cached version, and was literally greeted by five “continue this thread…

I am archiving subreddits on Github in plain-text org-mode. If you have some subreddit in mind, open an issue, and I'll create an archive repo for it.

- https://github.com/NightMachinary/r_HPfanfiction

- https://github.com/NightMachinary/r_rational

Re: Fastly Outage

#733

Earlier quoted context omitted.

Fastly Engineer 1: Seems like a common error message. Can you check stackoverflow to see if there's an easy fix? Fastly Engineer 2: I have some very bad news...

Oh man, how do we keep a pocket copy of SO? All of our jobs depend on it.

https://kapeli.com/dash also has the ability to download offline archives of SO. Its interface is very good.

Re: Fastly Outage

#734

Earlier quoted context omitted.

Riiiight... Anyways, you kept complaining of being downvoted, here's a clue: you're being an ass and no one likes you or what you have to say because you're wrong. So go scurry back to reddit where you belong troll...

> you're being an ass and no one likes you or what you have to say because you're wrong. So go scurry back to reddit where you belong troll... Okay? some proof please? This is not far off from a baseless character attack which isn't really effective when trying to convince me about your point on you knowing about Netflix's culture. If you really want a proper answer, the truth is, unfortunately for you I am in manage…

Proof? All the downvotes you got and why your comments are barely visible and all the crying you did in your comments about getting downvoted.

Re: Fastly Outage

#735
post #571

Earlier quoted context omitted.

The internet still works, only the websites are returning the wrong response

Interestingly, server side rendered pages worked well during the outage. Most of the issues were caused by sites that are relying too much on Javascript.

Yes, my personal project was working fine all the time. Only I couldn't access the Stripe payment system dashboard

Re: Fastly Outage

#736

Earlier quoted context omitted.

> you're being an ass and no one likes you or what you have to say because you're wrong. So go scurry back to reddit where you belong troll... Okay? some proof please? This is not far off from a baseless character attack which isn't really effective when trying to convince me about your point on you knowing about Netflix's culture. If you really want a proper answer, the truth is, unfortunately for you I am in manage…

Proof? All the downvotes you got and why your comments are barely visible and all the crying you did in your comments about getting downvoted.

You think downvotes and character attacks present as a good argument? Doesn't count as proof IMO if there isn't a valid argument presented, you're going to have to do a lot better than that.

And back to the main point, So I assume you agree that Netflix did go completely down the other day then right? It seems according to you that you know better of Netflix's management culture.

> I'm pretty certain this is not how Netflix's culture is.

Would you be willing to share your expert insight of this if you know better then?

Re: Fastly Outage

#737
post #694

Earlier quoted context omitted.

> You've just witnessed almost the entire internet break because of a catastrophic cascading outage that affected lots of huge companies, since third party services used and trusted Fastly. Blame the companies that relied on Fastly being up 100% of the time, even though Fastly explicitly states that they might be down any number of hours, and they will even give you money back for that [1]. If they did offer 100% SLA…

The main problem is that they had a failover system, the mystery is where was it in this outage? Why didn't this trigger? where was this system in place to prevent further cascading failures? > Blame the companies that relied on Fastly So it's everybody's fault Fastly went down now? That is a new one.

If companyA got affected by this, then either: 1- Its companyA's fault for not having a contingency plan or 2- Its companyA's accepted risk that this might happen.

We understand you're upset and passionate about this, perhaps now when more information has been published you understand better the circumstances that caused this problem.

Re: Fastly Outage

#738

Earlier quoted context omitted.

Proof? All the downvotes you got and why your comments are barely visible and all the crying you did in your comments about getting downvoted.

You think downvotes and character attacks present as a good argument? Doesn't count as proof IMO if there isn't a valid argument presented, you're going to have to do a lot better than that. And back to the main point, So I assume you agree that Netflix did go completely down the other day then right? It seems according to you that you know better of Netflix's management culture. > I'm pretty certain this is not how…

I'm not arguing Netflix, its mostly your attitude towards management and engineering culture. Basically your reply to the user "q3k". "Extrinsic motivators like 'we will give you a bonus' or 'we will fire you' are surprisingly bad at getting people to not fuck things up". You don't fire people just because they made a mistake. You find out what caused it, how to prevent it in the future, and you move on. That's what blameless post-mortems are about. No one is perfect and if you really are a manager that expects perfection, you really just suck as a person.

But now getting back to Netflix, they have post-mortems and they don't fire people willy-nilly over mistakes. Sure it's not hugops (a term I don't care for either), but they don't just up and fire people over a mistake. I never said anything about netflix going up or down on that day, but they also have problems just like everyone else. Their SLA is not 100% uptime and neither is Fastly.

In closing, you are being a pedantic little bitch who wants to argue minutia and I'm done with your trolling. I'm done responding to you, feel free to have the last reply as I really don't care anymore.

Re: Fastly Outage

#740

Earlier quoted context omitted.

Blame site operators that are single homing and not loadbalancing CDNs

For sites of any complexity with any dynamic content having CDN redundancy is akin to being multi-cloud -- it is not worth the effort. A lot of dynamic sites use Fastly for its programmatic edge control and a near immediate ( ~1s-4s, typically around 2 ) global cache invalidation for any tagged objects with a single call to the tag. That feature alone simplifies backend logic significantly. To make this feature porta…

>> For sites of any complexity with any dynamic content having CDN redundancy is akin to being multi-cloud — it is not worth the effort.

I proposed and lead our multi-CDN project at Pinterest for both static and dynamic content and I can tell you, many many times over, it has been well worth the effort. Everybody should do this if not only for contract negotiating leverage.

Cache invalidation is fast enough on all CDNs now for most use cases (yes, including Akamai). But realistically, most sites (Pinterest included) are not using clever cache invalidation for dynamic content because it’s not worth the integration effort (and it’s very difficult to abstract for large 1k+ engineering teams). Most customers are just using DSAs for the L4/L5 benefits (both security and perf). In that case, it’s not complicated to implement multi-cdn.

Post reply on HN