Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gmail, Google Maps and YouTube had outage issues

thenextweb.com

Post reply on HN
Print view View original

91–100 of 105 posts

Re: Gmail, Google Maps and YouTube had outage issues

#91

Post by nix0n »

dchest wrote: post #35
bcherny wrote: post #19

Earlier quoted context omitted.

2 reasons: 1. If you're still using HTTP 1.x, sharding assets across origins lets the browser load them in parallel (if set up correctly). You can generally load just 6 assets in parallel per origin, and sharding is a way to get around that limit. 2. A library like jQuery is so popular, and is so often served from googles CDN, that chances are a user already has it in their local cache from when they downloaded it on…

2. A library like jQuery is so popular, and is so often served from googles CDN, that chances are a user already has it in their local cache from when they downloaded it on some other site. Which of these versions do you have cached? 3.2.1, 3.2.0, 3.1.1, 3.1.0, 3.0.0, 2.2.4, 2.2.3, 2.2.2, 2.2.1, 2.2.0, 2.1.4, 2.1.3, 2.1.1, 2.1.0, 2.0.3, 2.0.2, 2.0.1, 2.0.0, 1.12.4, 1.12.3, 1.12.2, 1.12.1, 1.12.0, 1.11.3, 1.11.2, 1.11…

Looking at just minified versions from googleapis.com, I have 1.12.4, 2.2.4, 1.8.2, 1.10.2, 1.8.1, 1.9.1, and 2.1.1

Re: Gmail, Google Maps and YouTube had outage issues

#92

Post by kccqzy »

speakeron wrote: post #85
kiernanmcgowan wrote: post #82

Lets just hope that 8.8.8.8 remains stable. That can cause a whole heck of a lot more problems if it goes down.

> Lets just hope that 8.8.8.8 remains stable. That's why the smart money defaults to 8.8.4.4...

And 2001:4860:4860::8888, 2001:4860:4860::8844 too…

Re: Gmail, Google Maps and YouTube had outage issues

#93

Post by takeda »

21 wrote: post #48
trhway wrote: post #20

Earlier quoted context omitted.

reboot helps. May be Google runs Windows. Then they just need to reboot more frequently, ie. proactively.

Funnily enough, in one previous company I worked, which used Linux exclusively, the software was very buggy, and the servers were rebooted daily. We jokingly called that the "Windows solution". When we proposed adding a daily reboot to cron, the tech lead (which encouraged practices which lead to this low quality software) retorted that "this in not Windows, it doesn't need constant reboots", totally missing the poin…

Am I idealist by wondering that people think adding a cron job is a good solution? Why not trying to fix the broken software?

Re: Gmail, Google Maps and YouTube had outage issues

#94

Post by giancarlostoro »

kyrra wrote: post #79
lamlam wrote: post #72

Earlier quoted context omitted.

Loading JavaScript libraries synchronously should be avoided if possible, making the above solution not a great one.

Is there a better/different way to handle this type of fallback?

Surprised Content-MD5 or a similar spec isn't used by the browser to avoid a web where only Google's hosted solution allows for efficient JS file caching. If you know two files are most likely equal by filename and checksum, you should be able to just reload the cached version, if loading the cached file produces too many errors, try downloading the new one or something, instead of forcing everyone to host it all under the same corp (in this case Google). Oh well.

Re: Gmail, Google Maps and YouTube had outage issues

#95

Post by kbenson »

21 wrote: post #48
trhway wrote: post #20

Earlier quoted context omitted.

reboot helps. May be Google runs Windows. Then they just need to reboot more frequently, ie. proactively.

Funnily enough, in one previous company I worked, which used Linux exclusively, the software was very buggy, and the servers were rebooted daily. We jokingly called that the "Windows solution". When we proposed adding a daily reboot to cron, the tech lead (which encouraged practices which lead to this low quality software) retorted that "this in not Windows, it doesn't need constant reboots", totally missing the poin…

> "this in not Windows, it doesn't need constant reboots", totally missing the point that just using Linux doesn't make you a developer of reliable software.

Yes... but no software should require the Linux OS to reboot unless you're running custom or known buggy kernel modules, or you've triggered a spiral of death through swap usage. If your program is misbehaving, kill the program and restart it.

Restarting the OS daily is like noticing that your car uses a lot of gas, and deciding that every time you fill it up you'll get an oil change too. You might need to fill up a lot, but the oil change is overkill and not really affecting the situation in one way or another.

Re: Gmail, Google Maps and YouTube had outage issues

#96

Post by disconnected »

21 wrote: post #48
trhway wrote: post #20

Earlier quoted context omitted.

reboot helps. May be Google runs Windows. Then they just need to reboot more frequently, ie. proactively.

Funnily enough, in one previous company I worked, which used Linux exclusively, the software was very buggy, and the servers were rebooted daily. We jokingly called that the "Windows solution". When we proposed adding a daily reboot to cron, the tech lead (which encouraged practices which lead to this low quality software) retorted that "this in not Windows, it doesn't need constant reboots", totally missing the poin…

I think that if you are around in this business long enough, you will eventually experience something like this.

We had a Linux server that had some issue with a network driver that was causing kernel oops with long uptimes (about once a week). This one was particularly nasty because the oops would disable the network interface, meaning that you couldn't ssh to diagnose the fault (or, more likely, reboot), which meant that you had to drive to the premises and kick the server in the guts.

Of course, Murphy's law ensured that the server would fail at the worst possible time. Late Friday, weekends, your mom's birthday, etc... Not fun, at all.

The solution was to write a cron job to reboot every day at ~4:30AM. Stupid? Yes. But we all agreed that it sure beat the alternative (driving, kicking, sobbing).

The driver was eventually fixed by the vendor, and this "hack" became unnecessary.

Re: Gmail, Google Maps and YouTube had outage issues

#97

Post by ethbro »

ronjouch wrote: post #80
Waterluvian wrote: post #71

Earlier quoted context omitted.

Googlegate 2017. Cataclysmic meltdown where I had to wait a whole 10 more seconds for a video to load as it found another source.

I share the irony, but there's one nuance to bring: in our centralized web, sometimes there just is no such thing as "another source" . Was trying, during that outage, to watch Radiohead's "Lift" video posted earlier today. Impossible, as it was posted today and only on YouTube, and all media coverage are YouTube iframe embeds.

Welcome to the recentralized web.

Re: Gmail, Google Maps and YouTube had outage issues

#98

Post by in9 »

superhans2 wrote: post #68
21 wrote: post #48

Earlier quoted context omitted.

Funnily enough, in one previous company I worked, which used Linux exclusively, the software was very buggy, and the servers were rebooted daily. We jokingly called that the "Windows solution". When we proposed adding a daily reboot to cron, the tech lead (which encouraged practices which lead to this low quality software) retorted that "this in not Windows, it doesn't need constant reboots", totally missing the poin…

Sounds familiar. I had a client once, who had to reboot one particular router every day because it stopped working after 8h uptime. So one of their employees did that every morning by logging into some server via remote desktop to click the "reboot" button. I asked why they don't use some kind of cron job to automate that task, they just said "it doesn't work that way, you have to do it manually".

Some people are anti-automation.

I am an economist, turned into "data scientist" since I've learned to program in the past 5 years (hate that name...).

At a macro consultancy firm I worked, everybody lost it when I suggested that we moved our manually downloaded data from a bunch of excel spreadsheets to a proper database (28 years of macroeconomic data) so that we could programatically extract data for online reports we sent/hosted. They said I was being lazy...

Re: Gmail, Google Maps and YouTube had outage issues

#99

Post by takeda »

bcherny wrote: post #19
ocdtrekkie wrote: post #10

Earlier quoted context omitted.

Is there a good reason to use these things hosted by a third party source? Libraries are tiny, the fonts can be downloaded from Google Fonts and embedded locally, etc. Even the Google Analytics JS script I presume can be stored and run local. Shouldn't a goal be to mitigate the number of possible failures which can bring down your site by reducing the number of single points of failure?

2 reasons: 1. If you're still using HTTP 1.x, sharding assets across origins lets the browser load them in parallel (if set up correctly). You can generally load just 6 assets in parallel per origin, and sharding is a way to get around that limit. 2. A library like jQuery is so popular, and is so often served from googles CDN, that chances are a user already has it in their local cache from when they downloaded it on…

3. Now 1/3 of pages I'm visiting I need to make request to google because someone wanted to use a 5 lines of JS from jQuery.

Re: Gmail, Google Maps and YouTube had outage issues

#100

Post by [deleted] »

dom0 wrote: post #45
wil421 wrote: post #31

Earlier quoted context omitted.

Self hosting will probably cause more downtime than if you are using a decent CDN. CDN is just one of many points of failure, I would expect there's a fine balancing act where you could achieve benefits of both.

By definition, if your site is down, you don't need your assets.

[deleted]
Post reply on HN