Live data from Hacker News

Gmail, Google Maps and YouTube had outage issues

thenextweb.com

71–80 of 105 posts

Re: Gmail, Google Maps and YouTube had outage issues

#71

I think it's a little premature to be calling this a "meltdown". It's like calling one block of houses with power fluctuations a "blackout".

Media outlets are in the business of making sensational claims.

Googlegate 2017. Cataclysmic meltdown where I had to wait a whole 10 more seconds for a video to load as it found another source.

Re: Gmail, Google Maps and YouTube had outage issues

#72
post #46
post #3

Our site was using hosted libraries, google fonts, and google analytics. All of which seemed to be behind captchas, throwing CORS errors, and 503ing since this morning. Swapped out JQuery cdn for now.

For something like jquery, you can host locally and fallback to it if the CDN fails. https://stackoverflow.com/a/1014251

Loading JavaScript libraries synchronously should be avoided if possible, making the above solution not a great one.

Re: Gmail, Google Maps and YouTube had outage issues

#73
post #60

Earlier quoted context omitted.

I used to have a service that was like that, it was third party and only handled a specific function so I had a cron job that would reboot it at 4am every day. Sometimes the hammer approach is the only solution.

Was the software loading kernel modules? I'm curious why a more simple kill process routine wouldn't work.

Having been in a similar situation, many enterprise software require a precise start order and shutdown, sometime even depending to certain system services or some arcane stuff that then breaks in unpredictable times and often the only sane way to restart the environment is to bring it down whole

Re: Gmail, Google Maps and YouTube had outage issues

#74
post #19

Earlier quoted context omitted.

2 reasons: 1. If you're still using HTTP 1.x, sharding assets across origins lets the browser load them in parallel (if set up correctly). You can generally load just 6 assets in parallel per origin, and sharding is a way to get around that limit. 2. A library like jQuery is so popular, and is so often served from googles CDN, that chances are a user already has it in their local cache from when they downloaded it on…

> If you're still using HTTP 1.x Not sure why it was phrased that way but...isn't everybody? I know that HTTP/2 is released and browsers support it, but I'm fairly certain that next to nobody is actually doing anything with it.

It's getting rolled out in a lot of places. I think adoption is somewhere around 17% depending on how it's measured. I started using it for my personal website and it was really easy, the only part I missed was setting a cron job to reconfigure the web server whenever cert renewal went through.

Re: Gmail, Google Maps and YouTube had outage issues

#75

Earlier quoted context omitted.

[Removed]

There is a common practice for Node.js developers to use one of those tools that automatically reboot the site either whenever the single-threaded-event-loop-server crashes, or periodically...

Assuming you don't mean full "reboot", but reloading/restarting the app runtime every X requests, that's not unusual in many languages. Helps against memory leaks (which are not necessarily the fault of the application and thus not always fixable)

Re: Gmail, Google Maps and YouTube had outage issues

#76
post #60

Earlier quoted context omitted.

Was the software loading kernel modules? I'm curious why a more simple kill process routine wouldn't work.

Having been in a similar situation, many enterprise software require a precise start order and shutdown, sometime even depending to certain system services or some arcane stuff that then breaks in unpredictable times and often the only sane way to restart the environment is to bring it down whole

Exactly this, it was a horrible piece of enterprise software that depended on other similary horrible pieces of crap.

The only way to get it into a known good state when it shit the bed was to reboot the machine and then let it sort itself out.

We rapidly moved away from using it.

Funnily enough I'm in a similar situation at the new job with Jasper Reports, god damn if that thing isn't everything bad about Enterprise Java(TM).

Re: Gmail, Google Maps and YouTube had outage issues

#77
post #57
post #19

Earlier quoted context omitted.

2 reasons: 1. If you're still using HTTP 1.x, sharding assets across origins lets the browser load them in parallel (if set up correctly). You can generally load just 6 assets in parallel per origin, and sharding is a way to get around that limit. 2. A library like jQuery is so popular, and is so often served from googles CDN, that chances are a user already has it in their local cache from when they downloaded it on…

> You can generally load just 6 assets in parallel per origin This seems to only apply to Chrome, whereas Firefox will happily download everything as fast as possible. I know this because I fixed a bug recently where chrome was taking so long to download images that other resources on the page were timing out. No problem in Firefox.

Last I checked all the browsers had limits [1] when it came to HTTP 1.X.

[1] http://blog.olamisan.com/max-parallel-http-connections-in-a-...

Re: Gmail, Google Maps and YouTube had outage issues

#78
post #20
post #6

Does not seem to affect Europe. In fact, right now YouTube loads far quicker than it has for the last seven to ten days, where it would take ages to load any YouTube page.

reboot helps. May be Google runs Windows. Then they just need to reboot more frequently, ie. proactively.

Jaded developer here: a cronjob `pkill -9 foo` is cheaper than debugging a memory leak for a lot of businesses. Let the process supervisor handle the process death.

Re: Gmail, Google Maps and YouTube had outage issues

#79
post #72
post #46

Earlier quoted context omitted.

For something like jquery, you can host locally and fallback to it if the CDN fails. https://stackoverflow.com/a/1014251

Loading JavaScript libraries synchronously should be avoided if possible, making the above solution not a great one.

Is there a better/different way to handle this type of fallback?

Re: Gmail, Google Maps and YouTube had outage issues

#80

Earlier quoted context omitted.

Media outlets are in the business of making sensational claims.

Googlegate 2017. Cataclysmic meltdown where I had to wait a whole 10 more seconds for a video to load as it found another source.

I share the irony, but there's one nuance to bring: in our centralized web, sometimes there just is no such thing as "another source". Was trying, during that outage, to watch Radiohead's "Lift" video posted earlier today. Impossible, as it was posted today and only on YouTube, and all media coverage are YouTube iframe embeds.
Post reply on HN