Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An update on Sunday’s service disruption

cloud.google.com

Post reply on HN
Print view View original

151–160 of 220 posts

Re: An update on Sunday’s service disruption

#151

Post by geofft »

mcny wrote: post #138
kyrra wrote: post #131

Earlier quoted context omitted.

+1 (I'm a Googler, opinions my own) As someone who has been on oncall for 3 year and done a decent amount of production support, this public doc better explains the cause than the internal one if you aren't well versed in the underlying infra. A config change reduced used network capacity by half, and then things started falling over. And pushing the fix took a while due to the now overloaded network.

What can prevent this from happening in the future? Why was the config change required?

I think that's likely to be answered in the post-mortem, which is still to come.

Re: An update on Sunday’s service disruption

#152

Post by mlthoughts2018 »

mcny wrote: post #138
kyrra wrote: post #131

Earlier quoted context omitted.

+1 (I'm a Googler, opinions my own) As someone who has been on oncall for 3 year and done a decent amount of production support, this public doc better explains the cause than the internal one if you aren't well versed in the underlying infra. A config change reduced used network capacity by half, and then things started falling over. And pushing the fix took a while due to the now overloaded network.

What can prevent this from happening in the future? Why was the config change required?

Obviously to prevent things like this, Google needs more binary search tree whiteboard trivia problems in the interview process.

Re: An update on Sunday’s service disruption

#153

Post by [deleted] »

dmix wrote: post #92
theevilsharpie wrote: post #74

Earlier quoted context omitted.

Google also posts detailed postmortems for their more significant outages. Some examples: https://status.cloud.google.com/incident/cloud-networking/18... https://status.cloud.google.com/incident/cloud-pubsub/19001 https://status.cloud.google.com/incident/cloud-networking/18... https://status.cloud.google.com/incident/cloud-networking/18... https://status.cloud.google.com/incident/compute/18012 Given that this was a m…

I hope one of the things Google learns in the post mortem is that the next day summary should clearly include that a full post mortem is coming in the next few days or however long. Half the people in this thread are overlooking that fact and going into outrage mode.

[deleted]

Re: An update on Sunday’s service disruption

#156

Post by human20190310 »

fabledAble wrote: post #124

In 50 years, historians will look back on this as the turning point of AI control of humanity, inevitably leading to the point of no return. The brain trust at Google determined that humans are too prone to error to manage their critical data centers so they trained their AI efforts upon the resiliency of their hardware and software systems (i.e. "to prevent human operators from being able to mess it up"). By the tim…

The author's title is "VP, 24x7", which is already a position not designed for humans who sleep.

Re: An update on Sunday’s service disruption

#157

Post by joncrane »

jrockway wrote: post #143

This answered all the questions I had. I was really racking my brain on what one system at Google could go down to cause this much damage, but it makes perfect sense that bandwidth becoming unavailable and everything in the "default" or "bulk" traffic class being dropped would do it. The real question is whether the fix will be to not reduce bandwidth accidentally, or to upgrade customer traffic to a higher QoS class…

Question since you're an ex-Googler: is part of the scheme is to segment the traffic by revenue generation. I bet that paying customers get the priority, and free services get de-prioritized.

Is there any other analysis as well? For example, among the free services, maybe they rank them based on how much people will notice/how much press it would get if that service slowed down or stopped?

Re: An update on Sunday’s service disruption

#159

Post by thom »

phs318u wrote: post #84
mrunkel wrote: post #35

Earlier quoted context omitted.

> With all services restored to normal operation, Google’s engineering teams are now conducting a thorough post-mortem ... This is not the post-mortem, that is still to come.

Its the pre-post-mortem.

The purgatorium.

Re: An update on Sunday’s service disruption

#160

Post by [deleted] »

fowl2 wrote: post #42
jameslk wrote: post #5

> In essence, the root cause of Sunday’s disruption was a configuration change I feel like I hear about config changes breaking these cloud hosts so often it might as well be a meme. Is there a reason why it's usually configurations to blame vs code, hardware, etc?

In a well designed system I guess this is the only thing that can go wrong.

[deleted]
Post reply on HN