Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fastly Outage

fastly.com

Post reply on HN
Print view View original

581–590 of 740 posts

Re: Fastly Outage

#581

Post by gspr »

berkes wrote: post #452
Doxin wrote: post #324

Earlier quoted context omitted.

Believe it or not but "the internet" and "the world wide web" are not synonyms.

True. But the vast majority of use goes via "WWW". For example email - the other big "internet-user" is technically not part of the WWW, but most (? I don't have any stats, just a guess) of our mailclients run on the WWW, nonetheless.

I think that's the point the other person was making: The Internet is still fine, regardless of whether or not the content gets delivered.

There are roads (or shall I say tubes?). There are cars and busses on the road. Over time, almost everyone has migrated to just a few bus companies. One of them suffers a complete collapse for a few hours. Yes, this means chaos when it comes to transporting people. But the roads are just fine.

This doesn't mean that the situation is fine and that people aren't affected. But it would be entirely different if the roads had been washed away or something.

Re: Fastly Outage

#582

Post by colesantiago »

gurgus wrote: post #399
mothsonasloth wrote: post #333

Earlier quoted context omitted.

Call me old fashioned but the latest trend of showing "empathy" for a serious incident, then proceeding to dance around the aftermath of it, whilst people give themselves a pat on back in a retro/post-mortem, isn't the way to do it. People need to be blamed, and responsibility for actions taken (without covering asses)

Blame culture isn't the way forward here. Do a post-mortem, work out root causes, work as a unit to ensure this doesn't happen again. Obviously if there are levels of gross negligence or misconduct discovered during post-mortem, that will need to be dealt with accordingly, but coming into this with an attitude of "we must find someone to blame and incur repercussions" isn't healthy at all. We are humans - don't forge…

> Do a post-mortem, work out root causes, work as a unit to ensure this doesn't happen again. And if this happens again? They advertised they had failover and mitigations for this in the RAREST of cases:

> Notices will be posted here when we re-route traffic, upgrade hardware, or in the extremely rare case our network isn’t serving traffic. - status.fastly.com

The extremely rare case happened for an hour, which is a very long time in internet time.

Re: Fastly Outage

#583

Post by colesantiago »

colesantiago wrote: post #582
gurgus wrote: post #399

Earlier quoted context omitted.

Blame culture isn't the way forward here. Do a post-mortem, work out root causes, work as a unit to ensure this doesn't happen again. Obviously if there are levels of gross negligence or misconduct discovered during post-mortem, that will need to be dealt with accordingly, but coming into this with an attitude of "we must find someone to blame and incur repercussions" isn't healthy at all. We are humans - don't forge…

> Do a post-mortem, work out root causes, work as a unit to ensure this doesn't happen again. And if this happens again? They advertised they had failover and mitigations for this in the RAREST of cases: > Notices will be posted here when we re-route traffic, upgrade hardware, or in the extremely rare case our network isn’t serving traffic. - status.fastly.com The extremely rare case happened for an hour, which is a…

Edit: So the truth is also getting flagged here. unbelievable.

Re: Fastly Outage

#584

Post by efficax »

paublyrne wrote: post #528
kylen wrote: post #494

Earlier quoted context omitted.

I hear that you're suggesting that those involved shouldnt feel bad because its a systemic / just a job / etc. But the reality is that incidents like this can be very traumatic for those involved and thats not something they can control. If it was that simple to manage, depression and anxiety would not be a thing. Think its best to show a large amount of support and empathy for the individuals having a really bad day…

I don't see how it's so traumatic for the engineers involved, unless the company culture in Fastly is really awful and there are punitive repercussions, or attempts to pin responsbility on individuals rather than systems, which I doubt. Many here have been responsible for web service outages albeit on much smaller scales, and in my experience it feels awful while it's happening but you quickly forget about it because…

When CNN is reporting on the bug you deployed it might have some psychological impact

Re: Fastly Outage

#585

Post by southerntofu »

oefrha wrote: post #564
caseymarquis wrote: post #408

Earlier quoted context omitted.

Fastly Engineer 1: Seems like a common error message. Can you check stackoverflow to see if there's an easy fix? Fastly Engineer 2: I have some very bad news...

Well, with SO, at least you can search on Google and view the version cached by Google just fine. With Reddit however, these days almost all comments are locked behind “view entire discussion” or “continue this thread”. In fact, just now I searched for something for which the most relevant discussion was on Reddit; Reddit was down so I opened the cached version, and was literally greeted by five “continue this thread…

Yeah it's crazy how bad user-hostile reddit.com has become. Fortunately old.reddit.com is still available, but for how long? If only Javascript did not exist, it would be impossible for UX people to come up with something that bad.

Re: Fastly Outage

#586

Post by colesantiago »

austinjp wrote: post #166

Yeah so it's been mentioned in the comments already, but to everyone in Fastly right now: I feel for you. Something like this must be insanely stressful, and not just during the outage. There will be (should be) a massive post-mortem. People will be losing sleep over this for days, weeks, months. :( Edit: There seems to be a major empathy outage in this thread. Disgusted but not surprised, unfortunately.

Well, while engineers are getting paid $100K/yr to post #HugOps, I know someone in HFT and their dashboard uses the Fastly service, so this has had a huge impact on them for sure.

Flag and downvote all you want, you know this is true.

Re: Fastly Outage

#587

Post by tomrod »

oefrha wrote: post #564
caseymarquis wrote: post #408

Earlier quoted context omitted.

Fastly Engineer 1: Seems like a common error message. Can you check stackoverflow to see if there's an easy fix? Fastly Engineer 2: I have some very bad news...

Well, with SO, at least you can search on Google and view the version cached by Google just fine. With Reddit however, these days almost all comments are locked behind “view entire discussion” or “continue this thread”. In fact, just now I searched for something for which the most relevant discussion was on Reddit; Reddit was down so I opened the cached version, and was literally greeted by five “continue this thread…

Reddit's attempts at dark patterns are embarrassing from all perspectives. If you use dark patterns it's a laughably abysmal implementation. If you abhor dark patterns, it's a frustration.

Re: Fastly Outage

#588

Post by mrzool »

jmvoodoo wrote: post #171
mrzool wrote: post #54

Why is this a link to the Fastly homepage, where absolutely no information is provided? This is the page that should be linked: https://status.fastly.com

Oddly their homepage rendering an error was a more accurate description of the problem than "investigating potential impact to performance with our CDN"

I didn’t see any error whatsoever on their homepage, while now I see “Global CDN Disruption” on their status page.

Re: Fastly Outage

#589

Post by nephanth »

runawaybottle wrote: post #545
caseymarquis wrote: post #408

Earlier quoted context omitted.

Fastly Engineer 1: Seems like a common error message. Can you check stackoverflow to see if there's an easy fix? Fastly Engineer 2: I have some very bad news...

Oh man, how do we keep a pocket copy of SO? All of our jobs depend on it.

You can use kiwix (https://www.kiwix.org/en/).

It is an open-source software that allows you to keep and read offline static versions of websites in a specialized archive format (zim-files)

It was originally designed to allow you to read wikipedia offline, but there are also dumps of stackoverflow available on the relevant page : https://wiki.kiwix.org/wiki/Content_in_all_languages

Re: Fastly Outage

#590

Post by oefrha »

ornornor wrote: post #572
oefrha wrote: post #564

Earlier quoted context omitted.

Well, with SO, at least you can search on Google and view the version cached by Google just fine. With Reddit however, these days almost all comments are locked behind “view entire discussion” or “continue this thread”. In fact, just now I searched for something for which the most relevant discussion was on Reddit; Reddit was down so I opened the cached version, and was literally greeted by five “continue this thread…

Try “site:old.Reddit.com”

That's not going to work.

  $ curl https://old.reddit.com/robots.txt
  User-Agent: *
  Disallow: /
Also, even if search engines are allowed, old.reddit.com pages are not canonical ( points to the www.reddit.com version, which is actually reasonable behavior), so pages there would not be crawled as often or at all.
Post reply on HN