Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Google Cloud support engineer solves a tough DNS case

cloud.google.com

Post reply on HN
Print view View original

181–190 of 283 posts

Re: A Google Cloud support engineer solves a tough DNS case

#181

Post by the-rc »

jeffbee wrote: post #160
obmelvin wrote: post #158

Earlier quoted context omitted.

> particular well-known machine was unracked are you able to comment a bit further on why this machine was well known?

Because of the way engineers habitually run batch jobs with more replicas than there are machines, this one broken computer had crapped up every map/reduce job in that facility for a long time, and it had been sent to repairs many times without benefit. Many people knew instinctively that if their job was stuck it was probably because of the shard on xyz42 (or whatever the node name was).

I still remember the machine name. It starts with an l and ends with a 6. Over the course of a couple of years, pretty much all of its components (CPUs, RAM, drives) were replaced at least once. You could look up its maintenance history and it went on and on. I'm not sure if it was well known across all of engineering; from what I recall, it was in a cluster in Oregon reserved for a specific team. Because it was company property, no matter how doomed, they had to get signoff from upper management, close to Eric Schmidt's level, before they could destroy it.

Re: A Google Cloud support engineer solves a tough DNS case

#182

Post by CobrastanJorji »

erikig wrote: post #54

“...This means that the case will Follow the Sun by default, to provide 24/7 support” I love the concept of “Follow the Sun” to describe 24/7 support - I don’t think I’ve heard it described that way. I wonder how much we’d have to spend to get that tier of service?

"Follow the Sun" is subtly different from 24/7. 24/7 can mean follow the sun, but it can also mean "we are prepared to page someone at 2 AM and wake them up." Follow the sun means "there is an engineer in China, India, France, Boston, and San Francisco, and at least one of them is always at their desk and ready to take work."

The difference for users can be fairly small, but as someone who used to carry a pager for Amazon, the difference is really huge for the support person.

Re: A Google Cloud support engineer solves a tough DNS case

#183

Post by mav3rick »

floatingatoll wrote: post #7

The LKML message described in the post is here: https://lkml.org/lkml/2019/12/19/482

Something I'd like to add here the actual fix is -

+ if (rmem > (size + (unsigned int)sk->sk_rcvbuf))

However in reality this would have worked too - + if (rmem > (unsigned int)(size + sk->sk_rcvbuf)) (The bit pattern of the result remains the same and it's still casted as unsigned int during the comparison)

However, signed integer overflow is undefined behavior in C and unsigned integer overflow isn't. Hence, the submitted patch is the correct solution

Re: A Google Cloud support engineer solves a tough DNS case

#184

Post by foobiekr »

luhn wrote: post #22

This is a fun debugging story, but is a great example why servers should be cattle not pets. Having trouble with a VM? Blow it up and get a fresh one. Still having trouble? The provisioning steps are codified, you can walk through them and find the one that causes the issue.

the cattle-vs-pets thing is not really an excuse to not root cause a persistent issue while you have one.

Re: A Google Cloud support engineer solves a tough DNS case

#185

Post by sleepydog »

ibejoeb wrote: post #104
zimbatm wrote: post #96

Earlier quoted context omitted.

Did you read the article? The customer had set `net.core.rmem_default = 2147483647` on purpose. Which exposed a Kernel bug. The whole herd would be having the same issue.

I really want to know if the telemetrics (or whatever the thing was) pushed enough packets to actually warrant the config. Setting something to the max sounds like preopt to me. This must be a truly exceptional condition if it actually remained undiscovered since linux 3.

I can't think of anything that would warrant a 2GB receive buffer. The buffer should be sized so that the receiving program has a reasonable amount of time to drain it before it becomes full. A large skylake VM in GCP can do 32 Gbps (lowercase b), so assuming worst conditions, a 2GB receive buffer would give the receiver 500ms to call recv(), which is a huge amount of time for something that should take microseconds, especially in the context of a client.

Even if there was a specialized server that needed such a large receive buffer, it doesn't make sense to set the system-wide default so high.

Re: A Google Cloud support engineer solves a tough DNS case

#186

Post by layoutIfNeeded »

jeffbee wrote: post #95
storyinmemo wrote: post #74

Earlier quoted context omitted.

My fleet of machines I was the owner of at Facebook was around 10,000. I still remember the odd JVM crash that prompted me to reimage a machine. I wouldn't have remembered it except there were a few that month and it was the 3rd time I reimaged the same machine that I thought, "That's odd... I think I know that machine name." Checked history, saw the 3 repair jobs I had submitted... RAM was reset, CPU was eventually…

It is indeed folly to assume that cattle have no identity. There's an internally famous video inside Google in which a particular well-known machine was unracked, dragged out into a field, and ceremonially smashed to pieces by some hardware techs. Sometimes a machine just takes an arrow to the knee and it's never the same again. Then there are all the uncontrolled or unrecorded differences between machines: the ones…

I don’t think they make hard disks in Hungary.

Re: A Google Cloud support engineer solves a tough DNS case

#187

Post by the-rc »

layoutIfNeeded wrote: post #186
jeffbee wrote: post #95

Earlier quoted context omitted.

It is indeed folly to assume that cattle have no identity. There's an internally famous video inside Google in which a particular well-known machine was unracked, dragged out into a field, and ceremonially smashed to pieces by some hardware techs. Sometimes a machine just takes an arrow to the knee and it's never the same again. Then there are all the uncontrolled or unrecorded differences between machines: the ones…

I don’t think they make hard disks in Hungary.

The infamous IBM Deskstars aka Deathstars were made there.

Re: A Google Cloud support engineer solves a tough DNS case

#188

Post by walshemj »

jeffbee wrote: post #148
walshemj wrote: post #144

Earlier quoted context omitted.

back when we where buying hardware for big (at the time) intranet. All the servers where brought of the same batch of suns production line, I recall our sysadmin saying he rely wanted to do the same for the disks ie case of identical drives

A super-bad idea because drives made on the same week in the same facility will all fail at the same moment.

Unlikely that's not how statistics works in production engineering

Re: A Google Cloud support engineer solves a tough DNS case

#189

Post by grawlinson »

jeffbee wrote: post #95
storyinmemo wrote: post #74

Earlier quoted context omitted.

My fleet of machines I was the owner of at Facebook was around 10,000. I still remember the odd JVM crash that prompted me to reimage a machine. I wouldn't have remembered it except there were a few that month and it was the 3rd time I reimaged the same machine that I thought, "That's odd... I think I know that machine name." Checked history, saw the 3 repair jobs I had submitted... RAM was reset, CPU was eventually…

It is indeed folly to assume that cattle have no identity. There's an internally famous video inside Google in which a particular well-known machine was unracked, dragged out into a field, and ceremonially smashed to pieces by some hardware techs. Sometimes a machine just takes an arrow to the knee and it's never the same again. Then there are all the uncontrolled or unrecorded differences between machines: the ones…

One of my computers has been aptly named 'THESEUS' due to what was replaced on it. By the time it was repaired to an acceptable level, the only original component remaining was the chassis.

Re: A Google Cloud support engineer solves a tough DNS case

#190

Post by jeffbee »

walshemj wrote: post #188
jeffbee wrote: post #148

Earlier quoted context omitted.

A super-bad idea because drives made on the same week in the same facility will all fail at the same moment.

Unlikely that's not how statistics works in production engineering

With respect, I just went through a "code red" at a large, well-known cloud storage company caused by synchronized late-life death of hard disks all manufactured in the same batch. That's the second time in my career that I've been through the same phenomenon. Hard disks that are made together wear out together.
Post reply on HN