Live data from Hacker News

Fastly Outage

fastly.com

641–650 of 740 posts

Re: Fastly Outage

#641

Earlier quoted context omitted.

StackOverflow and all the StackExchange family of sites are down. I suspect the lost productivity from that will be more costly over the whole economy than potential lost sales via Shopify. People can go back to shopify so those transactions not definitely blocked for ever, any time "lost" due to reference resources being unavailable can't so easily be claimed back.

Believe it or not, but there are developers out there that read the docs.

[citation needed]

Re: Fastly Outage

#642

Yeah so it's been mentioned in the comments already, but to everyone in Fastly right now: I feel for you. Something like this must be insanely stressful, and not just during the outage. There will be (should be) a massive post-mortem. People will be losing sleep over this for days, weeks, months. :( Edit: There seems to be a major empathy outage in this thread. Disgusted but not surprised, unfortunately.

I have worked for one of their competitors (I'm not saying which) for quite a while. I've indirectly caused multiple outages that were maybe 1% this bad before, that didn't make the news only due to luck. Code that I owned (but did not write) was once a key cause of a severe outage that did make the news, and it would have been worse if I weren't coincidentally halfway through replacing that code with something more modern. I also had to do some very rapid work on internal failsafes around the time of the infamous Mirai botnet, to minimize service degradation in case it was pointed at us.

It sucks. Working on CDN reliability is like working on wastewater management: the public forgets you exist until something breaks, when they start asking why you weren't doing your job. Fortunately, internal people at least seem to get it -- I hope this is the same as Fastly.

Re: Fastly Outage

#643
post #630
post #610

Earlier quoted context omitted.

I wanted to suggest site:old.reddit.com since I use that version with automatic redirect, but this: https://old.reddit.com/robots.txt is very different from this: https://reddit.com/robots.txt I guess there is a market for search engine (maybe accessed through tor) which does not care about robots.txt, DMCAs, right to be forgotten etc. Bootstrapping it should not be that hard since it can also provide better results…

Wonder what the story is behind these two... User-Agent: bender Disallow: /my_shiny_metal_ass User-Agent: Gort Disallow: /earth

It might be related to the time few years ago when Google added exclusions for user agent t1300 in regard to its founders. Gort seems to be a robot from old scifi and bender might be something similar.

Re: Fastly Outage

#644
post #587

Earlier quoted context omitted.

Reddit's attempts at dark patterns are embarrassing from all perspectives. If you use dark patterns it's a laughably abysmal implementation. If you abhor dark patterns, it's a frustration.

It's just enough to annoy you but not enough to make everyone leave the platform

The mobile version is literally unusable. Half the subs show an error and you can't load most comments.

Re: Fastly Outage

#645

Earlier quoted context omitted.

> Cloudflare as state-of-the-art in terms of security and ease-of-use Depends whose security. I value my security dearly and that's why i use the Tor Browser. Cloudflare has decided i cannot browse any of their websites if i care about my security (they filter out tor users and archiving bots agressively) so i'm not using any cloudflare-powered website. Is it good for security that we prevent people from using securi…

Speaking for myself here, I don't see how people can use the web without javascript. As for Tor, you're routing other people's traffic while they route yours, so I can understand how such connections would be blocked given that blocking IPs is still a method for mitigating security issues, and you can't determine the IP of a Tor browser.

> I don't see how people can use the web without javascript.

Its pretty easy: browse marked up documents, not applications. If some developer conflates the first for the second, move on.

Re: Fastly Outage

#646
post #381

Earlier quoted context omitted.

Call me old fashioned but the latest trend of showing "empathy" for a serious incident, then proceeding to dance around the aftermath of it, whilst people give themselves a pat on back in a retro/post-mortem, isn't the way to do it. People need to be blamed, and responsibility for actions taken (without covering asses)

The point isn't to dance around the incident, but to not blame people. You can blame systems, design, engineering culture, processes, but don't blame people. Even if someone accidentally pressed the 'destroy prod' button, that's not the fault of that person, it's the fault of that button existing and being accessible in the first place. I have no empathy for Fastly-the-company. I hate the fact that the Internet is ce…

> don't blame people

I feel like this requires some nuance.

Don't blame an IC for introducing a bug or misconfiguration that led to the outage.

Do consider blaming (and firing!) management if, during the postmortem, it turns out that it was in the way of fixing systemic problems.

Ultimately, rule #1 should be: don't blame somebody unless malice or gross negligence is proven. Rule #2 should be the assumption that ICs will not have done either. Rule #3 is that sometimes, individual responsibility is required.

Re: Fastly Outage

#647
post #630
post #610

Earlier quoted context omitted.

I wanted to suggest site:old.reddit.com since I use that version with automatic redirect, but this: https://old.reddit.com/robots.txt is very different from this: https://reddit.com/robots.txt I guess there is a market for search engine (maybe accessed through tor) which does not care about robots.txt, DMCAs, right to be forgotten etc. Bootstrapping it should not be that hard since it can also provide better results…

Wonder what the story is behind these two... User-Agent: bender Disallow: /my_shiny_metal_ass User-Agent: Gort Disallow: /earth

[deleted]

Re: Fastly Outage

#648
post #533

Earlier quoted context omitted.

I disagree. People implemented those systems, so if you are correct that it is the systems fault, then it is also a persons fault. People must be held accountable to have good incentives to reduce such outtages in the future. I do agree though that we should always be compassionate and realistic with other humans.

v1. "It's Bob's fault and so we fired Bob." v2. "The issue was caused by a previously unidentified pathway that caused a feedback loop and overloaded our servers in a cascading fashion (or whatever). We have implemented a fix for this and updated our testing and deployment processes to stop similar cascades." Which solves the problem long term? As an architect making product choices, v2 wins every time. (With the cav…

If an employee does something actively malicious, you should absolute remove them. This is very rare though - incompetence /broken systems is much more likely.

Otherwise you develop internal process that's entirely scar tissue, and only stops your teams doing their jobs.

Re: Fastly Outage

#649
post #630
post #610

Earlier quoted context omitted.

I wanted to suggest site:old.reddit.com since I use that version with automatic redirect, but this: https://old.reddit.com/robots.txt is very different from this: https://reddit.com/robots.txt I guess there is a market for search engine (maybe accessed through tor) which does not care about robots.txt, DMCAs, right to be forgotten etc. Bootstrapping it should not be that hard since it can also provide better results…

Wonder what the story is behind these two... User-Agent: bender Disallow: /my_shiny_metal_ass User-Agent: Gort Disallow: /earth

I reod some people use false slugs in the robots.txt as a honey pot of sorts. IPs that actually read the robots.txt, ignore the disallow, and still access the uri are outright banned.

Re: Fastly Outage

#650

Yeah so it's been mentioned in the comments already, but to everyone in Fastly right now: I feel for you. Something like this must be insanely stressful, and not just during the outage. There will be (should be) a massive post-mortem. People will be losing sleep over this for days, weeks, months. :( Edit: There seems to be a major empathy outage in this thread. Disgusted but not surprised, unfortunately.

Well, while engineers are getting paid $100K/yr to post #HugOps, I know someone in HFT and their dashboard uses the Fastly service, so this has had a huge impact on them for sure. Flag and downvote all you want, you know this is true.

I suspect you'll have trouble convincing a forum of primarily engineers that a high frequency trader is more worthy of sympathy than an engineer. They're both pretty privileged jobs and HFT is not known for having tons of benefits to society
Post reply on HN