Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Google Cloud support engineer solves a tough DNS case

cloud.google.com

Post reply on HN
Print view View original

211–220 of 283 posts

Re: A Google Cloud support engineer solves a tough DNS case

#211

Post by bluedino »

GABeech wrote: post #36
TwoBit wrote: post #18

The customer had set an extremely large buffer size and nobody thought to mention that? Perhaps the individual reporting the problem was different and was unaware of that unusual change.

There are many things here that concern me from a system view. 1. They need guaranteed delivery, but chose to use UDP 2. They jacked up the default rmem buffer to ~2GB which is insane. Also, applies to all sockets not just UDP, so I wouldn't be surprised if they where also running into issues with memory pressure especially under load 3. Support didn't seem to let them know that's a pretty unconventional configuratio…

I would have told the customer to reduce the number by 1000 or whatever and closed the case.

Re: A Google Cloud support engineer solves a tough DNS case

#212

Post by catlifeonmars »

storyinmemo wrote: post #74
luhn wrote: post #22

This is a fun debugging story, but is a great example why servers should be cattle not pets. Having trouble with a VM? Blow it up and get a fresh one. Still having trouble? The provisioning steps are codified, you can walk through them and find the one that causes the issue.

My fleet of machines I was the owner of at Facebook was around 10,000. I still remember the odd JVM crash that prompted me to reimage a machine. I wouldn't have remembered it except there were a few that month and it was the 3rd time I reimaged the same machine that I thought, "That's odd... I think I know that machine name." Checked history, saw the 3 repair jobs I had submitted... RAM was reset, CPU was eventually…

I’m confused as to why the solution wasn’t to just replace the machine (preferably automatically) the first time it failed.

Re: A Google Cloud support engineer solves a tough DNS case

#213

Post by bluedino »

atombender wrote: post #23
TwoBit wrote: post #18

The customer had set an extremely large buffer size and nobody thought to mention that? Perhaps the individual reporting the problem was different and was unaware of that unusual change.

To be fair, there are a lot of sysctl settings. To be sure, it's one of the first places I would look for networking weirdness, but it's also often hard to tell what impact those settings have on anything.

Anyone that messes with them, better know what they are doing. 99% of the time they read about or heard about the setting somewhere and are blindly following along. I see this a lot with "performance improvements" when they should be looking other places like their web server configuration. Like why would you tweak those on a low end wordpress server?!

Re: A Google Cloud support engineer solves a tough DNS case

#215

Post by drudru11 »

erikig wrote: post #54

“...This means that the case will Follow the Sun by default, to provide 24/7 support” I love the concept of “Follow the Sun” to describe 24/7 support - I don’t think I’ve heard it described that way. I wonder how much we’d have to spend to get that tier of service?

I know that this phrase was used way back in the early 90s. I suspect it was used before then as well.

Re: A Google Cloud support engineer solves a tough DNS case

#216

Post by killjoywashere »

dekhn wrote: post #142
jeffbee wrote: post #95

Earlier quoted context omitted.

It is indeed folly to assume that cattle have no identity. There's an internally famous video inside Google in which a particular well-known machine was unracked, dragged out into a field, and ceremonially smashed to pieces by some hardware techs. Sometimes a machine just takes an arrow to the knee and it's never the same again. Then there are all the uncontrolled or unrecorded differences between machines: the ones…

Once, the head of IT of a company I used to work for was touring the datacenter, passing some new racks filled with blade servers. He stopped, said "why are all the fans running full blast on this rack?" and the admins checked and they were running some test workload at scale somebody had forgotten about a few weeks before. Everybody was embarassed because no monitoring caught it, but the VP of IT did by walking past…

Admiral Rickover was known for walking into the engineering spaces of nuclear ships and just throwing a valve handle that would force a reactor scram. Not infrequently on a submerged submarine. Just to make sure the team was on their toes.

Re: A Google Cloud support engineer solves a tough DNS case

#217

Post by frogpelt »

grawlinson wrote: post #189
jeffbee wrote: post #95

Earlier quoted context omitted.

It is indeed folly to assume that cattle have no identity. There's an internally famous video inside Google in which a particular well-known machine was unracked, dragged out into a field, and ceremonially smashed to pieces by some hardware techs. Sometimes a machine just takes an arrow to the knee and it's never the same again. Then there are all the uncontrolled or unrecorded differences between machines: the ones…

One of my computers has been aptly named 'THESEUS' due to what was replaced on it. By the time it was repaired to an acceptable level, the only original component remaining was the chassis.

I was troubleshooting a computer once that would randomly shut off during boot and, one component at a time, I replaced everything on it including the motherboard to no avail.

Finally I took all the parts out of the original computer and put them in a different chassis and it worked! Put them back in the old chassis and back to the old problem.

Eventually I noticed that there was an extra stand-off in the first computer case and it was shorting out the motherboard.

It was literally the chassis causing the problem.

Re: A Google Cloud support engineer solves a tough DNS case

#219

Post by the-rc »

dmoy wrote: post #206
fizwhiz wrote: post #200

Earlier quoted context omitted.

> There's an internally famous video inside Google in which a particular well-known machine was unracked, dragged out into a field, and ceremonially smashed to pieces by some hardware techs. Eyy what would I search on moma to find this video?

Huh, I can't find it. I can confirm I've read the same thing though, years back.

There used to be a go link for it (same as the machine name), but knowing Google, it might be stale. There's a good chance you can find more on the internal folklore site. If that one is still around, too.

Re: A Google Cloud support engineer solves a tough DNS case

#220

Post by PudgePacket »

storyinmemo wrote: post #74
luhn wrote: post #22

This is a fun debugging story, but is a great example why servers should be cattle not pets. Having trouble with a VM? Blow it up and get a fresh one. Still having trouble? The provisioning steps are codified, you can walk through them and find the one that causes the issue.

My fleet of machines I was the owner of at Facebook was around 10,000. I still remember the odd JVM crash that prompted me to reimage a machine. I wouldn't have remembered it except there were a few that month and it was the 3rd time I reimaged the same machine that I thought, "That's odd... I think I know that machine name." Checked history, saw the 3 repair jobs I had submitted... RAM was reset, CPU was eventually…

Had a similar issue. Spend weeks debugging a process, adding logging metrics, making graphs to figure out where the performance cliff was in an email sending service. The results were so weird they were unexplainable.

Someone suggested just nuke it and bring it back up on a fresh instance. Problem was gone! Everything running smoothly again..

Post reply on HN