Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google Outage in Europe

google.com

Post reply on HN
Print view View original

41–50 of 186 posts

Re: Google Outage in Europe

#41

Post by kreeben »

KronisLV wrote: post #34
kreeben wrote: post #25

Earlier quoted context omitted.

But now you have someone in your team who will never, ever make that same mistake again and should be your new go-to guy for all X related changes (X being DNS or what-have-you). Firing someone with that type of experience does not lead to success. 100% of all devs make huge mistakes, at least once.

> But now you have someone in your team who will never, ever make that same mistake again and that should be your new go-to guy for all DNS related changes. I'm not entirely sure that's always true. For example, i've seen people introduce N+1 issues into a codebase, spend evenings fixing them and refactoring code to fix production issues... just to later introduce those very same types of issues. Sure, you can learn…

N+1 issues aren't nearly as devastating as N^2. Commend them for not putting your systems to a complete halt, then teach them how to reason about this properly.

>> a quality gate

Yes, this, also.

Re: Google Outage in Europe

#42

Post by nudpiedo »

Seems that google is not anymore the company it used to be advertised. The myth prevails though.

It’s a pity as some of its services are frankly among the best, and I depend on them.

Re: Google Outage in Europe

#43

Post by oblio »

kroolik wrote: post #19
oblio wrote: post #18

Earlier quoted context omitted.

He he. It's kind of hilarious. Maybe they should stagger reviews to ensure high availability? :-D

Nothing wrong with it. People make errors, distributed systems aren't easy. More frequent changes - more likely a bug was introduced. My post is just a speculation, let's wait for the actual technical details doc from Google.

Well, there are various whitepapers that reformulate this exact truth you've highlighted: issues tend to happen more when changes are made.

Regarding the technical details doc, Google will never state that outright in individual postmortems. And they will definitely not draw this to the logical conclusion regarding the spiky yearly activity.

Re: Google Outage in Europe

#44

Post by sebow »

Good Maybe these outages will make people realize how dependent they are on a few monopolistic companies.Not only these will make rise of competitors but also hopefully make people's information more independent

Re: Google Outage in Europe

#45

Post by deepstack »

scrollaway wrote: post #10
synchton wrote: post #4

Is it me or are big provider outages getting more common these days?

As time passes, the more big cloud providers there are, and the more complex they individually get (more products). Assuming the chance of an outage is actually static. If there is one (highly reliable and trusted) provider with five products one year, and three providers with ten products the next year, the chance of you seeing an outage has gone way up because of surface area.

More reason to promote for self-host, and decentralised systems like smtp, matrix.org, ActivityPub. The idea of having all the data and server concentrated in a few player such as google, amazon, ccp is not reliable for the digital operation of the planet.

Re: Google Outage in Europe

#46

Post by cranekam »

goldcd wrote: post #23
kroolik wrote: post #22

Earlier quoted context omitted.

Why would you fire an engineer you have just spent millions to train?

I'd guess because Europe-wide outages are costing more than millions

But firing someone doesn't undo an incident. It just introduces other weird incentives. People become afraid to change things for fear of breaking something, or when something does break they try to hide it rather than feeling like they can immediately ask for and receive help to fix it.

The only time someone should be fired for causing an outage is if they're negligent or sloppy or mess things up all the time. This is rare. Almost always outages in large systems are the combination of many factors — latent bugs, design flaws, abnormal load, etc, any one or two of which wouldn't take the site down. But when the combine in a perfect storm that nobody foresaw things fall over.

Re: Google Outage in Europe

#48

Post by l33t2328 »

chrisseaton wrote: post #47
kroolik wrote: post #22

Earlier quoted context omitted.

Why would you fire an engineer you have just spent millions to train?

What tech company spends millions on training anyone?

The implication is that millions were spent training the person who made the mistake when they cost the company millions by making that mistake.

Re: Google Outage in Europe

#49

Post by Ueland »

chrisseaton wrote: post #47
kroolik wrote: post #22

Earlier quoted context omitted.

Why would you fire an engineer you have just spent millions to train?

What tech company spends millions on training anyone?

An outage can be pretty expensive but it is training for those whom triggered it and/or those that fixes it.
Post reply on HN