Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fastly Outage

fastly.com

Post reply on HN
Print view View original

381–390 of 740 posts

Re: Fastly Outage

#381

Post by q3k »

mothsonasloth wrote: post #333
austinjp wrote: post #166

Yeah so it's been mentioned in the comments already, but to everyone in Fastly right now: I feel for you. Something like this must be insanely stressful, and not just during the outage. There will be (should be) a massive post-mortem. People will be losing sleep over this for days, weeks, months. :( Edit: There seems to be a major empathy outage in this thread. Disgusted but not surprised, unfortunately.

Call me old fashioned but the latest trend of showing "empathy" for a serious incident, then proceeding to dance around the aftermath of it, whilst people give themselves a pat on back in a retro/post-mortem, isn't the way to do it. People need to be blamed, and responsibility for actions taken (without covering asses)

The point isn't to dance around the incident, but to not blame people. You can blame systems, design, engineering culture, processes, but don't blame people. Even if someone accidentally pressed the 'destroy prod' button, that's not the fault of that person, it's the fault of that button existing and being accessible in the first place.

I have no empathy for Fastly-the-company. I hate the fact that the Internet is centralized around CDNs. I wish this idea of 'but we _must_ run a CDN for our 1QPM blog!' would die in a fire. But I can still empathize with the Fastly engineers handling this shitstorm right now.

Re: Fastly Outage

#382

Post by longwave »

jamespo wrote: post #297
lpmitchell wrote: post #108

Earlier quoted context omitted.

The m.media-amazon.com domain (and a few other CDN'd domains that they use) are running through Fastly: nslookup m.media-amazon.com Name: media.amazon.map.fastly.net It is very interesting that they are not using CloudFront!

Good to have 3rd party redundancy, time to fail over to something else now I'd think though.

They already have:

  $ host m.media-amazon.com
  m.media-amazon.com is an alias for c.media-amazon.com.
  c.media-amazon.com has address 99.86.119.84
(which is a Cloudfront IP)

Re: Fastly Outage

#383

Post by collyw »

lpmitchell wrote: post #9

This seems to be impacting a number of huge sites, including the UK government website[0]. [0] https://www.gov.uk/ https://m.media-amazon.com/ https://pages.github.com/ https://www.paypal.com/ https://stackoverflow.com/ https://nytimes.com/ Edit: Fastly's incident report status page: https://status.fastly.com/incidents/vpk0ssybt3bj

Click the new tab. Lots of posts about sites being down. All flagged.

Re: Fastly Outage

#384

Post by numpad0 »

oneeyedpigeon wrote: post #272
thrdbndndn wrote: post #30

Earlier quoted context omitted.

Interestingly, Twitter only has its emoji SVGs down.

And this is (one reason) why using images instead of actual emojis is such a stupid idea. Why, Twitter, WHY?

err, to make representations platform independent?

Re: Fastly Outage

#386

Post by _kyran »

Those of you that work in DevOps, SRE or are CTOs.

What kind of things do you put in place to manage these kind of centralised issues that are beyond your control?

Re: Fastly Outage

#387

Post by TheRealDunkirk »

Every other comment about what's down in this thread -- as if we needed dozens of site-by-site accountings of this outage in the first place -- is a bitch about reddit. Why is reddit so important to this crowd? The specific topics I used to read the site for (half a dozen years ago) have all been overrun by "bucket people," there is literally never an answer to any question I find a google link to there, and the site's design is actively user-hostile. Seriously: what's keeping that place afloat? Porn, I suppose.

Re: Fastly Outage

#389

Post by abluecloud »

john37386 wrote: post #378

It should be resolve soon. From fastly status page: The issue has been identified and a fix is being implemented. Posted 1 minute ago. Jun 08, 2021 - 10:44 UTC

Wonder if all the caches will have been wiped, causing knock on issues

Re: Fastly Outage

#390

Post by kypro »

barosl wrote: post #53

I didn't know so many sites were depending on Fastly. Stack Overflow, GitHub, reddit, .... Even pip is unavailable. My development workflow is completely janked up. It is a bit scary that we are putting too many eggs in one basket.

You would think sites like Github and key government sites would at least have a fall back at the ready. It reasonable to use a CDN like Fastly, but having a single point of failure seems silly if you're the BBC or Gov UK. Although, it does seem BBC managed to get back up and running pretty quick so perhaps they were prepared for this.
Post reply on HN