Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google Outage in Europe

google.com

Post reply on HN
Print view View original

51–60 of 186 posts

Re: Google Outage in Europe

#51

Post by xcambar »

ddalex wrote: post #37
evercast wrote: post #31

Earlier quoted context omitted.

Comments like this make me wonder if people really expect engineers to be fired because of an outage? I do not work at Google, but none of my workplaces would fire engineers because of a failure. Mistakes happen. As long as they are not repeated, everything is good. If your company fires people in situations like this, run away and never look back.

Googler here, not speaking on the behalf of the company, my opinions are my own People do absolutely NOT get fired over incidents. Making mistakes is human. An incident will prompt a review of the systems and safeguards in place to prevent such an incident, much like an airline incident investigation - basically "somebody fat-fingered it" is never the answer, postmortems are always blameless EDIT: now that I think of…

> Googler here, not speaking on the behalf of the company, my opinions are my own

Why do employees at big tech names (FAANG et al.) are so often so cautious as to include this as a foreword everywhere? Twitter bios are full of that, for instance.

It is crazy to me; who would expect anything else that our opinions being your own and nothing more? Who would expect that your word (with all due respect) is worth anything with regards to the company's PR?

Is there an actual risk in the US? Have there been trials or anything that push people to add such statements?

Re: Google Outage in Europe

#52

Post by hdjjhhvvhga »

moffkalast wrote: post #21
kroolik wrote: post #9

There were outages around the same time last year. Somebody in the HN thread commented back then that the employees evaluation and promotion window ends around december/eoy, thus more releases are made. https://en.m.wikipedia.org/wiki/Google_services_outages

Then they get fired instead, well done ha.

It makes no sense at all. After the outage you have not only a review of the causes and appropriate remedies, but also more experienced people who are now more aware of possible consequences of seemingly unrelated actions and will take extra care not to make these things happen in the future.

Also, such cases are rarely the "fault" of a single person. Or, the direct/immediate cause is often not the main one.

Re: Google Outage in Europe

#53

Post by paganel »

rablackburn wrote: post #38
ratww wrote: post #30

Earlier quoted context omitted.

Now that I think of it, I might have blamed my provider or at least restarted my modem if Hacker News was ever down...

It’s the perfect connection test. 9 times out of 10 if you can’t connect to hn the problem is you, not them. Probably even 99/100

I usually check the HN website to see if my internet is down, both on my mobile and on my work computer. I used to just type "test" in the browser address bar hoping for the related Google SERPs but checking for HN seems way faster nowadays.

Re: Google Outage in Europe

#54

Post by 1ibsq »

chrisseaton wrote: post #47
kroolik wrote: post #22

Earlier quoted context omitted.

Why would you fire an engineer you have just spent millions to train?

What tech company spends millions on training anyone?

The other comments already explained it, but I'm wondering how you haven't come across this 'saying' before. It's so overused and also cheesy in my opinion.

Re: Google Outage in Europe

#55

Post by pfarrell »

chrisseaton wrote: post #47
kroolik wrote: post #22

Earlier quoted context omitted.

Why would you fire an engineer you have just spent millions to train?

What tech company spends millions on training anyone?

The GP is a reference to the anecdote about IBM’s Thomas Watson not firing a executive who had made an error costing the company a substantial amount of money.

Re: Google Outage in Europe

#56

Post by inglor »

kroolik wrote: post #9

There were outages around the same time last year. Somebody in the HN thread commented back then that the employees evaluation and promotion window ends around december/eoy, thus more releases are made. https://en.m.wikipedia.org/wiki/Google_services_outages

I would be very interested in any research about code quality with relation to promotion packets. Ideally acamdemic.

I am not sure where to look for it

Re: Google Outage in Europe

#58

Post by KronisLV »

kreeben wrote: post #41
KronisLV wrote: post #34

Earlier quoted context omitted.

> But now you have someone in your team who will never, ever make that same mistake again and that should be your new go-to guy for all DNS related changes. I'm not entirely sure that's always true. For example, i've seen people introduce N+1 issues into a codebase, spend evenings fixing them and refactoring code to fix production issues... just to later introduce those very same types of issues. Sure, you can learn…

N+1 issues aren't nearly as devastating as N^2. Commend them for not putting your systems to a complete halt, then teach them how to reason about this properly. >> a quality gate Yes, this, also.

> N+1 issues aren't nearly as devastating as N^2.

Depends on the project, i guess: if you're unlucky enough to be working on a monolith and suddenly a page takes 5'000 SQL queries to load as opposed to 100, because someone thought that initializing data through service/DB calls in a loop is "easier" than writing views in the DB, it might still kill the entire system anyways, depending on the count of users.

And once this data initialization is sufficiently complicated and convoluted for you not to be able to rewrite it and them not wanting to rewrite it, all while "the business" is breathing down on your neck, you might either want to introduce caching (and possibly run into cache invalidation problems down the road), or just freshen up your CV.

I guess i'd also like to expand on the previous suggestion and advise others to consider performance/load testing as well, especially when coupled with APM solutions like Skywalking or even Matomo analytics, both of which can allow you to aggregate the historical page load times, CPM and overall performance of your applications, to figure out what went wrong when.

Re: Google Outage in Europe

#59

Post by yunohn »

oblio wrote: post #43
kroolik wrote: post #19

Earlier quoted context omitted.

Nothing wrong with it. People make errors, distributed systems aren't easy. More frequent changes - more likely a bug was introduced. My post is just a speculation, let's wait for the actual technical details doc from Google.

Well, there are various whitepapers that reformulate this exact truth you've highlighted: issues tend to happen more when changes are made. Regarding the technical details doc, Google will never state that outright in individual postmortems. And they will definitely not draw this to the logical conclusion regarding the spiky yearly activity.

> logical conclusion regarding the spiky yearly activity

Why is it logical? There’s tons of changes being deployed at all times, at all large companies.

Across products, verticals, everything - hundreds of changes at any given point. Some of these changes can introduce hard-to-predict bugs in globally distributed systems. Most of the time, external users don’t even notice before they’re fixed.

Like another commenter said, performance reviews do not coincide with with the year-end at Google and other companies.

Re: Google Outage in Europe

#60

Post by b0afc375b5 »

xcambar wrote: post #51
ddalex wrote: post #37

Earlier quoted context omitted.

Googler here, not speaking on the behalf of the company, my opinions are my own People do absolutely NOT get fired over incidents. Making mistakes is human. An incident will prompt a review of the systems and safeguards in place to prevent such an incident, much like an airline incident investigation - basically "somebody fat-fingered it" is never the answer, postmortems are always blameless EDIT: now that I think of…

> Googler here, not speaking on the behalf of the company, my opinions are my own Why do employees at big tech names (FAANG et al.) are so often so cautious as to include this as a foreword everywhere? Twitter bios are full of that, for instance. It is crazy to me; who would expect anything else that our opinions being your own and nothing more? Who would expect that your word (with all due respect) is worth anything…

It's in the spirit of full disclosure, which some, including me, appreciate.
Post reply on HN