Fastly Outage • Hacker News Board

Re: Fastly Outage

#341

Post by patentatt » Tue, Jun 08, 2021, 10:43 AM UTC

the_mitsuhiko wrote: post #206

dijit wrote: post #163

Earlier quoted context omitted.

Mediation vs Meditation I think. I think Varnish uses mediation intentionally though, it was this way 7 years ago when I last used Varnish.

Does not seem like: https://github.com/varnishcache/varnish-cache/blob/606977bbf...

According to the Wikipedia page, it is intentional:

https://en.wikipedia.org/wiki/Guru_Meditation

Or did one of you already edit the Wikipedia page to reflect this discussion on hn?

Re: Fastly Outage

#342

Post by snookdebook » Tue, Jun 08, 2021, 10:43 AM UTC

I gave it about 10 tries, and it seems a very small percentage of transactions do go through.

A decent number of tries is rejected right at the Varnish front door:

Many more reach some backend system that just dumps "connection failure":

And a tiny few do get through:

...snip...

Re: Fastly Outage

#343

Post by cromulent » Tue, Jun 08, 2021, 10:43 AM UTC

ljm wrote: post #265

qeternity wrote: post #223

Earlier quoted context omitted.

> It's interesting because Varnish gets it write in their docs Sorry I couldn't help myself...it's too funny to misspell "right" in a thread about spelling mistakes

Mruphy's Law in action

Isn't it Muphry's Law? Or was that another example :)

https://en.wikipedia.org/wiki/Muphry%27s_law

Re: Fastly Outage

#344

Post by MyOnePiece » Tue, Jun 08, 2021, 10:43 AM UTC

Quick question if the cdns are down why cant traffic be routed to the web servers the central web servers the company owns ?

I thought cdns had fallback configured ?

Re: Fastly Outage

#345

Post by strictfp » Tue, Jun 08, 2021, 10:43 AM UTC

superzamp wrote: post #61

algo_cheese wrote: post #19

Earlier quoted context omitted.

> We're currently investigating potential impact to performance with our CDN services. Guys, you are offline with a 503 error, this is a little more than "potential impact to performance".

Lowkey status reports are the norm now :) "some users may experience degraded service" => site completely down for all locations

Yeah, that's my experience as well. I thought it meant "we have no idea what's going on" though.

Re: Fastly Outage

#346

Post by spzb » Tue, Jun 08, 2021, 10:43 AM UTC

jakub_g wrote: post #290

lpmitchell wrote: post #9

This seems to be impacting a number of huge sites, including the UK government website[0]. [0] https://www.gov.uk/ https://m.media-amazon.com/ https://pages.github.com/ https://www.paypal.com/ https://stackoverflow.com/ https://nytimes.com/ Edit: Fastly's incident report status page: https://status.fastly.com/incidents/vpk0ssybt3bj

https://www.bbc.com/news/technology-57399628 "A number of leading media websites are currently not working, including the Guardian, Financial Times, Independent and the New York Times."

Not that the BBC are gloating that they're still up

Re: Fastly Outage

#347

Post by raimondious » Tue, Jun 08, 2021, 10:43 AM UTC

gansai wrote: post #231

wouldn't websites have alternate CDN's managing their traffic, why should they have a single point of failure ? I was assuming there are couple of services like Fastly and companies might have architected keeping in mind the alternatives too, I guess.

Normally you configure your a record to point at the cdn as the cdn is the thing that gives you multiple points of failure (caches all over the world). Hard to have a fallback to that. Running multiple cdns would be extremely expensive. Cdn caches are kept useful by traffic running through them, so hard to have a backup for that too.

Re: Fastly Outage

#348

Post by richardwhiuk » Tue, Jun 08, 2021, 10:43 AM UTC

ysavir wrote: post #296

Tangential question, but with services like these, is there a known way to handle failure gracefully? Some way to automatically bypass these services if they are known to be down?

Have two different CDN partners, own your own DNS, and then withdraw one of the CDNs if they are down. Suspect that's what Amazon have done.

Re: Fastly Outage

#349

Post by taurath » Tue, Jun 08, 2021, 10:43 AM UTC

Harzza wrote: post #188

aunetx wrote: post #140

Earlier quoted context omitted.

Opened my browser, ad my three major Web pages : github, gitlab.gnome.org and old.reddit.com... They all are down.

Unless you're browsing reddit without logging in, you can just set the old reddit theme from your account settings so you don't need to use the old. prefix :)

They reset the setting, regularly just to piss off people who only want the old frames.

Re: Fastly Outage

#350

Post by abhiminator » Tue, Jun 08, 2021, 10:43 AM UTC

ZoomStop wrote: post #317

The outage has already been added to the Fastly Wikipedia page

Holy smokes these Wikipedia writers are quick! I'm sometimes impressed by how fast a page on a super recent happening gets populated with all of the currently known details.