Gmail, Google Maps and YouTube had outage issues • Hacker News Board

Re: Gmail, Google Maps and YouTube had outage issues

#101

Post by lamlam » Tue, Sep 12, 2017, 5:48 PM UTC

kyrra wrote: post #79

lamlam wrote: post #72

Earlier quoted context omitted.

Loading JavaScript libraries synchronously should be avoided if possible, making the above solution not a great one.

Is there a better/different way to handle this type of fallback?

Yes. What you should do is use an asynchronous module loader. There are many small standalone ones like loadjs [1]. But the more widely used tools such as webpack also suppprt this as code splitting [2].

In general you want to avoid sync loads of js assets because depending on how the server serving the asset hangs it can cause the webpage to hang as well. For example, if the server responds with a 404 right away then there are no problems. But if the server does not respond and leaves the connection open the browser will just wait the max time.

[1] https://github.com/muicss/loadjs/blob/master/README.md

[2] https://webpack.js.org/guides/code-splitting/

Re: Gmail, Google Maps and YouTube had outage issues

#102

Post by s_kilk » Tue, Sep 12, 2017, 6:20 PM UTC

lamlam wrote: post #77

s_kilk wrote: post #57

Earlier quoted context omitted.

> You can generally load just 6 assets in parallel per origin This seems to only apply to Chrome, whereas Firefox will happily download everything as fast as possible. I know this because I fixed a bug recently where chrome was taking so long to download images that other resources on the page were timing out. No problem in Firefox.

Last I checked all the browsers had limits [1] when it came to HTTP 1.X. [1] http://blog.olamisan.com/max-parallel-http-connections-in-a-...

The limits in Firefox must be really high then, it was almost funny looking at the difference in behaviour between the two browsers in the situation I was testing.

Re: Gmail, Google Maps and YouTube had outage issues

#103

Post by tudorconstantin » Tue, Sep 12, 2017, 8:01 PM UTC

in9 wrote: post #98

superhans2 wrote: post #68

Earlier quoted context omitted.

Sounds familiar. I had a client once, who had to reboot one particular router every day because it stopped working after 8h uptime. So one of their employees did that every morning by logging into some server via remote desktop to click the "reboot" button. I asked why they don't use some kind of cron job to automate that task, they just said "it doesn't work that way, you have to do it manually".

Some people are anti-automation. I am an economist, turned into "data scientist" since I've learned to program in the past 5 years (hate that name...). At a macro consultancy firm I worked, everybody lost it when I suggested that we moved our manually downloaded data from a bunch of excel spreadsheets to a proper database (28 years of macroeconomic data) so that we could programatically extract data for online report…

I hope you took that as a compliment. Laziness is a virtue in programming, according to Larry Wall: http://threevirtues.com/

Re: Gmail, Google Maps and YouTube had outage issues

#104

Post by IntelMiner » Tue, Sep 12, 2017, 9:56 PM UTC

21 wrote: post #48

trhway wrote: post #20

Earlier quoted context omitted.

reboot helps. May be Google runs Windows. Then they just need to reboot more frequently, ie. proactively.

Funnily enough, in one previous company I worked, which used Linux exclusively, the software was very buggy, and the servers were rebooted daily. We jokingly called that the "Windows solution". When we proposed adding a daily reboot to cron, the tech lead (which encouraged practices which lead to this low quality software) retorted that "this in not Windows, it doesn't need constant reboots", totally missing the poin…

I had a fun issue for about 6 months with a Cable (DOCSIS 3.0) modem I'd purchased. A Motorola/Arris Surfboard 6183

The modem would randomly seem to keel over with some unknown fault. Causing my internet speed to drop from 300Mbps down to 0.25Mbps, ping to Google.com for instance would then spike from 5ms to 1900ms (or more)

Curiously the upload speed would stay pegged at 30Mbps however!

After a few days of this happening, I picked up a Chinese "Smart Switch" that ran OpenWRT, and set up a small shell script to simply ping Google.com, then cycle the modem if the average ping results exceeded a certain threshold (I think 100ms?)

It would also record the exact date and time and log that, so I could try and correlate the issue. Unfortunately it seemed to be utterly random, without any rhyme or reason.

Since I worked at Comcast at the time, I tried to gather more data on the issue internally. Eventually writing a report that totaled something around 10 pages.

From what I gathered: There were no physical signal deviations when the device would "hang". The device would respond normally to SNMP requests etc, everything on Comcast's side appeared normal. The device had some internal fault with its software that was causing problems (kernel bug perhaps?)

I contacted Motorola/Arris for support, and was advised that the warranty specifically excludes Software faults(!) and then kindly recommended to "upgrade" to the newer SB6190 model.

Unfortunately being a Cable modem, the firmware is completely controlled by the ISP. Since there were only 25,000~ SB6183's on Comcast's network at the time, and even fewer on the speed tier that I had, there was not enough data to report the issue back to Motorola/Arris through Comcast

Eventually a software update was pushed out which corrected the issue roughly 6 months later

Re: Gmail, Google Maps and YouTube had outage issues

#105

Post by loonattic » Wed, Sep 13, 2017, 1:32 AM UTC

takeda wrote: post #93

21 wrote: post #48

Earlier quoted context omitted.

Funnily enough, in one previous company I worked, which used Linux exclusively, the software was very buggy, and the servers were rebooted daily. We jokingly called that the "Windows solution". When we proposed adding a daily reboot to cron, the tech lead (which encouraged practices which lead to this low quality software) retorted that "this in not Windows, it doesn't need constant reboots", totally missing the poin…

Am I idealist by wondering that people think adding a cron job is a good solution? Why not trying to fix the broken software?

Sometimes, it's just not possible nor reasonably doable. As seen above, with proprietary drivers...