Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google Outage in Europe

google.com

Post reply on HN
Print view View original

31–40 of 186 posts

Re: Google Outage in Europe

#31

Post by evercast »

moffkalast wrote: post #21
kroolik wrote: post #9

There were outages around the same time last year. Somebody in the HN thread commented back then that the employees evaluation and promotion window ends around december/eoy, thus more releases are made. https://en.m.wikipedia.org/wiki/Google_services_outages

Then they get fired instead, well done ha.

Comments like this make me wonder if people really expect engineers to be fired because of an outage? I do not work at Google, but none of my workplaces would fire engineers because of a failure. Mistakes happen. As long as they are not repeated, everything is good.

If your company fires people in situations like this, run away and never look back.

Re: Google Outage in Europe

#32

Post by jsty »

querez wrote: post #14
scrollaway wrote: post #10

Earlier quoted context omitted.

As time passes, the more big cloud providers there are, and the more complex they individually get (more products). Assuming the chance of an outage is actually static. If there is one (highly reliable and trusted) provider with five products one year, and three providers with ten products the next year, the chance of you seeing an outage has gone way up because of surface area.

Except that Google services don't run within GCP, but apart from it (mostly on Borg I'd guess).

Their internal platform is likely analogous in terms of accumulating complexity (and a bit of cruft) over time though

Re: Google Outage in Europe

#33

Post by kroolik »

delroth wrote: post #29
kroolik wrote: post #9

There were outages around the same time last year. Somebody in the HN thread commented back then that the employees evaluation and promotion window ends around december/eoy, thus more releases are made. https://en.m.wikipedia.org/wiki/Google_services_outages

Perf has been over for almost a month now, and the evaluation period was over more than two months ago.

Then it might, indeed, be unrelated. Thanks for clarification!

Re: Google Outage in Europe

#34

Post by KronisLV »

kreeben wrote: post #25
goldcd wrote: post #23

Earlier quoted context omitted.

I'd guess because Europe-wide outages are costing more than millions

But now you have someone in your team who will never, ever make that same mistake again and should be your new go-to guy for all X related changes (X being DNS or what-have-you). Firing someone with that type of experience does not lead to success. 100% of all devs make huge mistakes, at least once.

> But now you have someone in your team who will never, ever make that same mistake again and that should be your new go-to guy for all DNS related changes.

I'm not entirely sure that's always true. For example, i've seen people introduce N+1 issues into a codebase, spend evenings fixing them and refactoring code to fix production issues... just to later introduce those very same types of issues.

Sure, you can learn from mistakes, have post-mortems and so on (provided that your org even does those and that anyone listens and cares about the conclusions from those), but to me it feels like the most foolproof way is to ensure that no-one can make these mistakes again, be it with a checklist (which tend to be ignored, honestly), or better yet, an automated CI step or a new test suite.

In my eyes, it's basically the same as with unit tests - everyone agrees that you need them, but people rarely write enough of them. So if you introduce something to prevent them from not doing what they should, e.g. a quality gate within a CI step which will disallow a merge once the coverage falls below a set margin, suddenly things are a lot better in the long run.

Re: Google Outage in Europe

#36

Post by ulzeraj »

Didn't even noticed. Work doesn't use any GCloud components and personally its been a while since I've degoogled myself. No Google docs, mail or search. Not even the quad 8 nameservers.

Re: Google Outage in Europe

#37

Post by ddalex »

evercast wrote: post #31
moffkalast wrote: post #21

Earlier quoted context omitted.

Then they get fired instead, well done ha.

Comments like this make me wonder if people really expect engineers to be fired because of an outage? I do not work at Google, but none of my workplaces would fire engineers because of a failure. Mistakes happen. As long as they are not repeated, everything is good. If your company fires people in situations like this, run away and never look back.

Googler here, not speaking on the behalf of the company, my opinions are my own

People do absolutely NOT get fired over incidents. Making mistakes is human. An incident will prompt a review of the systems and safeguards in place to prevent such an incident, much like an airline incident investigation -

basically "somebody fat-fingered it" is never the answer, postmortems are always blameless

EDIT: now that I think of it, the opposite thing happens after a major incident - a systemic failure should be identified, people are being hired to fix it :)

Re: Google Outage in Europe

#38

Post by rablackburn »

ratww wrote: post #30
1cvmask wrote: post #6

And many people blame their internet provider as google outages are rare.

Now that I think of it, I might have blamed my provider or at least restarted my modem if Hacker News was ever down...

It’s the perfect connection test.

9 times out of 10 if you can’t connect to hn the problem is you, not them. Probably even 99/100

Post reply on HN