Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fastly Outage

fastly.com

Post reply on HN
Print view View original

691–700 of 740 posts

Re: Fastly Outage

#691

Post by iso1631 »

dijit wrote: post #602
nabla9 wrote: post #232

Earlier quoted context omitted.

The Web (World Wide Web) build atop of the Internet, is not impervious. ps. "The Internet was build to survive attacks" is not true. It's a myth made popular by Robert Cringely in the early 1990s. The Arpanet was simply a protocol for mainframes used by computer scientists to connect. The Internet is relatively resilient against attacks, but that was not the "whole idea". It was not in the design at all. Bob Taylor:…

BGP (the protocol underpinning the internet) is built entirely for avoiding outages of any size. Even email has a method baked into to the protocol for handling failure. Fallbacks are good, baking in resiliency is better.

BGP has its problems (that time centurylink blackholed traffic but wouldn't drop their connections, bgp hijacks etc), but it's not centralised in single (or very few) points of failure

Re: Fastly Outage

#692

Post by jaywalk »

southerntofu wrote: post #585
oefrha wrote: post #564

Earlier quoted context omitted.

Well, with SO, at least you can search on Google and view the version cached by Google just fine. With Reddit however, these days almost all comments are locked behind “view entire discussion” or “continue this thread”. In fact, just now I searched for something for which the most relevant discussion was on Reddit; Reddit was down so I opened the cached version, and was literally greeted by five “continue this thread…

Yeah it's crazy how bad user-hostile reddit.com has become. Fortunately old.reddit.com is still available, but for how long? If only Javascript did not exist, it would be impossible for UX people to come up with something that bad.

Impossible? Man, it's crazy how fast people forget things like good old fashioned GETs and POSTs. It would obviously be a full page refresh, but other than that the same awful UX could still be implemented.

Re: Fastly Outage

#693

Post by ihumanable »

adkadskhj wrote: post #657
H8crilA wrote: post #409

Earlier quoted context omitted.

Imagine losing sleep over a corporate problem where you're just the next Joe Engineer, to be fired the second you're not needed. Have some perspective people.

I'm confused, why isn't being fired something to lose sleep over in your eyes? I get that you're implying that the job itself is not worth that much concern, but it seems you're ignoring that jobs bring in income, pay your mortgage, etc. If i lost my job tomorrow i'd be terrified.

People rarely get fired for outages. The comment you are replying to is saying that engineers shouldn’t stress out over an outage that only impacts a corporation.

It’s a commentary on work / life balance and the all-too-common phenomenon of employees sacrificing for a company (in this case, feeling such personal stress that they would lose sleep) and contrasting it with the fact that most employers will fire you without a second thought if it’s what’s best for the business (they won’t lose any sleep).

It’s a critique of the asymmetry that often exists and is frequently exploited by companies. This is often seen in statements like, “we are one big family so put in a few more hours for this launch” coupled with announcements like, “profit projections didn’t meet expectations so we are downsizing 5% of the work force.” You are family when they need you to work hard, and an expendable free market agent when your continued employment might risk hitting the quarterly goal.

It is, of course, reasonable to lose sleep if you think your employment is in jeopardy. Very few companies, especially in the competitive SV market are firing engineers because of a single outage, even a bad one, because you just paid a bunch of money to train those engineers how to see this coming and fix it.

Re: Fastly Outage

#694

Post by q3k »

colesantiago wrote: post #669
q3k wrote: post #663

Earlier quoted context omitted.

> Not my problem. Fastly should work as intended. What's your SLA with them? Just assuming things will always work because the marketing copy said so is recipe for disaster. It's hoping that things never go wrong, and when they inevitably do, being caught pants down. Everything fails sometimes. You must know how much your SaaS provider contractually promises, ensure that any SLA breach is something financially accept…

> What's your SLA with them? Sorry what? You've just witnessed almost the entire internet break because of a catastrophic cascading outage that affected lots of huge companies, since third party services used and trusted Fastly. Shopify stores couldn't accept payments on their websites, Coinbase Retail/Pro transactions and trading apps failed to load, and delivery apps stopped loading all of a sudden. These are just…

> You've just witnessed almost the entire internet break because of a catastrophic cascading outage that affected lots of huge companies, since third party services used and trusted Fastly.

Blame the companies that relied on Fastly being up 100% of the time, even though Fastly explicitly states that they might be down any number of hours, and they will even give you money back for that [1]. If they did offer 100% SLA, it would probably be out of budget for most users, as that kind of systems are prohibitively expensive to run.

Depending on a single CDN like Fastly is building an SPOF into your product. It is not less of a design blunder that whatever Fastly did internally to have an outage. If Shopify lost millions because of a short, simple third-party outage they have at least as much of a high-priority postmortem to write and issues to address as Fastly.

[1] - https://docs.fastly.com/products/service-availability-sla

Re: Fastly Outage

#695

Post by namibj »

runawaybottle wrote: post #545
caseymarquis wrote: post #408

Earlier quoted context omitted.

Fastly Engineer 1: Seems like a common error message. Can you check stackoverflow to see if there's an easy fix? Fastly Engineer 2: I have some very bad news...

Oh man, how do we keep a pocket copy of SO? All of our jobs depend on it.

Here, just pin the underlying IPFS object, or use this one hosted by cloudflare: https://ipfs-sec.stackexchange.cloudflare-ipfs.com/

Re: Fastly Outage

#696

Post by bruce343434 »

comboy wrote: post #610
oefrha wrote: post #564

Earlier quoted context omitted.

Well, with SO, at least you can search on Google and view the version cached by Google just fine. With Reddit however, these days almost all comments are locked behind “view entire discussion” or “continue this thread”. In fact, just now I searched for something for which the most relevant discussion was on Reddit; Reddit was down so I opened the cached version, and was literally greeted by five “continue this thread…

I wanted to suggest site:old.reddit.com since I use that version with automatic redirect, but this: https://old.reddit.com/robots.txt is very different from this: https://reddit.com/robots.txt I guess there is a market for search engine (maybe accessed through tor) which does not care about robots.txt, DMCAs, right to be forgotten etc. Bootstrapping it should not be that hard since it can also provide better results…

try editing your hosts file to redirect reddit to old.reddit

/etc/hosts

reddit.com old.reddit.com

www.reddit.com old.reddit.com

np.reddit.com old.reddit.com

Re: Fastly Outage

#697

Post by dharmab »

runawaybottle wrote: post #545
caseymarquis wrote: post #408

Earlier quoted context omitted.

Fastly Engineer 1: Seems like a common error message. Can you check stackoverflow to see if there's an easy fix? Fastly Engineer 2: I have some very bad news...

Oh man, how do we keep a pocket copy of SO? All of our jobs depend on it.

You can download the database dump from https://archive.org/details/stackexchange.

Re: Fastly Outage

#698

Post by ilikepi »

driminicus wrote: post #633
southerntofu wrote: post #585

Earlier quoted context omitted.

Yeah it's crazy how bad user-hostile reddit.com has become. Fortunately old.reddit.com is still available, but for how long? If only Javascript did not exist, it would be impossible for UX people to come up with something that bad.

When it goes away you can try teddit.net

Why wait? Teddit has been a great substitute for reading in a mobile browser, and making an iOS shortcut for transforming Reddit links was pretty straightforward.

Re: Fastly Outage

#699

Post by q3k »

colesantiago wrote: post #683
detaro wrote: post #678

Earlier quoted context omitted.

We don't know, but the usual scenarios would be "issue impacts failover mechanism too", "failover mechanism overloads other system components leading to cascading failure" or "something causes failover mechanism to to think all is fine".

> We don't know... So, the rarest of cases (our network isn’t serving traffic) just happened right now, and their failover system just took a snooze then, but 'it exists apparently' according to you. Tell that the huge clients that lost sales because of this, and all you have to say is: "wE DoN'T kNoW..."

> Tell that the huge clients that lost sales because of this, and all you have to say is: "wE DoN'T kNoW..."

Tell these clients that they should've carefully read their contract with Fastly, especially the 'Service Level Agreement' part.

Re: Fastly Outage

#700

Post by nabla9 »

dijit wrote: post #602
nabla9 wrote: post #232

Earlier quoted context omitted.

The Web (World Wide Web) build atop of the Internet, is not impervious. ps. "The Internet was build to survive attacks" is not true. It's a myth made popular by Robert Cringely in the early 1990s. The Arpanet was simply a protocol for mainframes used by computer scientists to connect. The Internet is relatively resilient against attacks, but that was not the "whole idea". It was not in the design at all. Bob Taylor:…

BGP (the protocol underpinning the internet) is built entirely for avoiding outages of any size. Even email has a method baked into to the protocol for handling failure. Fallbacks are good, baking in resiliency is better.

User iso1631 talked about attacks, not just outages.

The basic design BGP is very vulnerable against malicious attacks. Email security is nonexistent.

Post reply on HN