Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google Outage in Europe

google.com

Post reply on HN
Print view View original

21–30 of 186 posts

Re: Google Outage in Europe

#21

Post by moffkalast »

kroolik wrote: post #9

There were outages around the same time last year. Somebody in the HN thread commented back then that the employees evaluation and promotion window ends around december/eoy, thus more releases are made. https://en.m.wikipedia.org/wiki/Google_services_outages

Then they get fired instead, well done ha.

Re: Google Outage in Europe

#22

Post by kroolik »

moffkalast wrote: post #21
kroolik wrote: post #9

There were outages around the same time last year. Somebody in the HN thread commented back then that the employees evaluation and promotion window ends around december/eoy, thus more releases are made. https://en.m.wikipedia.org/wiki/Google_services_outages

Then they get fired instead, well done ha.

Why would you fire an engineer you have just spent millions to train?

Re: Google Outage in Europe

#25

Post by kreeben »

goldcd wrote: post #23
kroolik wrote: post #22

Earlier quoted context omitted.

Why would you fire an engineer you have just spent millions to train?

I'd guess because Europe-wide outages are costing more than millions

But now you have someone in your team who will never, ever make that same mistake again and should be your new go-to guy for all X related changes (X being DNS or what-have-you). Firing someone with that type of experience does not lead to success.

100% of all devs make huge mistakes, at least once.

Re: Google Outage in Europe

#26

Post by kroolik »

goldcd wrote: post #23
kroolik wrote: post #22

Earlier quoted context omitted.

Why would you fire an engineer you have just spent millions to train?

I'd guess because Europe-wide outages are costing more than millions

Still, that engineer (if its engineer's fault) is extremely unlikely to make that issue again. IMHO, the problem is systemic in that why the system allows such errors (if its human error) to happen. Given Google's scale I think a lot of the generally known scenarios are covered and what you see is tens of services interacting in not obvious ways. Those unobvious scenarios manifest in situations like this.

Re: Google Outage in Europe

#27

Post by scrollaway »

querez wrote: post #14
scrollaway wrote: post #10

Earlier quoted context omitted.

As time passes, the more big cloud providers there are, and the more complex they individually get (more products). Assuming the chance of an outage is actually static. If there is one (highly reliable and trusted) provider with five products one year, and three providers with ten products the next year, the chance of you seeing an outage has gone way up because of surface area.

Except that Google services don't run within GCP, but apart from it (mostly on Borg I'd guess).

The above applies to cloud as understood by techies just as b2c cloud as understood by laypeople.

Re: Google Outage in Europe

#29

Post by delroth »

kroolik wrote: post #9

There were outages around the same time last year. Somebody in the HN thread commented back then that the employees evaluation and promotion window ends around december/eoy, thus more releases are made. https://en.m.wikipedia.org/wiki/Google_services_outages

Perf has been over for almost a month now, and the evaluation period was over more than two months ago.
Post reply on HN