Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Google Cloud support engineer solves a tough DNS case

cloud.google.com

Post reply on HN
Print view View original

241–250 of 283 posts

Re: A Google Cloud support engineer solves a tough DNS case

#241

Post by exikyut »

the-rc wrote: post #181
jeffbee wrote: post #160

Earlier quoted context omitted.

Because of the way engineers habitually run batch jobs with more replicas than there are machines, this one broken computer had crapped up every map/reduce job in that facility for a long time, and it had been sent to repairs many times without benefit. Many people knew instinctively that if their job was stuck it was probably because of the shard on xyz42 (or whatever the node name was).

I still remember the machine name. It starts with an l and ends with a 6. Over the course of a couple of years, pretty much all of its components (CPUs, RAM, drives) were replaced at least once. You could look up its maintenance history and it went on and on. I'm not sure if it was well known across all of engineering; from what I recall, it was in a cluster in Oregon reserved for a specific team. Because it was comp…

I wonder if MR could integrate with a fuzzing engine that jumbles random combinations of real inputs into garbage but runnable jobs that cause reproducible crashes above some threshold (eg at least once per day, or if things are bad enough, once per month or something).

Regarding this system: the motherboard was never swapped?

Re: A Google Cloud support engineer solves a tough DNS case

#242

Post by exikyut »

ahupp wrote: post #117
storyinmemo wrote: post #74

Earlier quoted context omitted.

My fleet of machines I was the owner of at Facebook was around 10,000. I still remember the odd JVM crash that prompted me to reimage a machine. I wouldn't have remembered it except there were a few that month and it was the 3rd time I reimaged the same machine that I thought, "That's odd... I think I know that machine name." Checked history, saw the 3 repair jobs I had submitted... RAM was reset, CPU was eventually…

Also at FB: one day we got a huge spike in measured site-wide cpu usage. After the terror subsided, we found that a single request on a single machine had reported an improbably huge number of cycles (like, a billion years of cpu time). We figured a hardware problem and sent it to repair. A month later the same thing happened to the same machine; it had just been reimaged and sent back into the fleet. There was some…

Out of curiosity, do these reproducibly-broken components ever make it into an upstream testing environment?

Re: A Google Cloud support engineer solves a tough DNS case

#243

Post by exikyut »

jeffbee wrote: post #190
walshemj wrote: post #188

Earlier quoted context omitted.

Unlikely that's not how statistics works in production engineering

With respect, I just went through a "code red" at a large, well-known cloud storage company caused by synchronized late-life death of hard disks all manufactured in the same batch. That's the second time in my career that I've been through the same phenomenon. Hard disks that are made together wear out together.

I'm curious how wide the failure window was (timespan, ramp-up/down, etc), relative to how many devices were involved.

And I wonder how well the signal in that ratio might scale down to hundreds or tens of disks.

Re: A Google Cloud support engineer solves a tough DNS case

#246

Post by MaxBarraclough »

killjoywashere wrote: post #216
dekhn wrote: post #142

Earlier quoted context omitted.

Once, the head of IT of a company I used to work for was touring the datacenter, passing some new racks filled with blade servers. He stopped, said "why are all the fans running full blast on this rack?" and the admins checked and they were running some test workload at scale somebody had forgotten about a few weeks before. Everybody was embarassed because no monitoring caught it, but the VP of IT did by walking past…

Admiral Rickover was known for walking into the engineering spaces of nuclear ships and just throwing a valve handle that would force a reactor scram. Not infrequently on a submerged submarine. Just to make sure the team was on their toes.

I'm not a sailor or a nuclear engineer, but that doesn't sound like a great idea. Should the Chaos Monkey approach really be used on nuclear systems?

Aside: was it even legal for him to do that?

Re: A Google Cloud support engineer solves a tough DNS case

#247

Post by killjoywashere »

MaxBarraclough wrote: post #246
killjoywashere wrote: post #216

Earlier quoted context omitted.

Admiral Rickover was known for walking into the engineering spaces of nuclear ships and just throwing a valve handle that would force a reactor scram. Not infrequently on a submerged submarine. Just to make sure the team was on their toes.

I'm not a sailor or a nuclear engineer, but that doesn't sound like a great idea. Should the Chaos Monkey approach really be used on nuclear systems? Aside: was it even legal for him to do that?

If you are responsible for building the industry that designs and builds nuclear submarines that carry nuclear missiles, you had better make sure that those submarines and their crews can handle chaos monkeys.

Also, Rickover was Congress's favorite admiral. They forced the Navy to promote him. I'm pretty sure they made sure that the laws were to his liking.

Re: A Google Cloud support engineer solves a tough DNS case

#248

Post by masonic »

jeffbee wrote: post #95
storyinmemo wrote: post #74

Earlier quoted context omitted.

My fleet of machines I was the owner of at Facebook was around 10,000. I still remember the odd JVM crash that prompted me to reimage a machine. I wouldn't have remembered it except there were a few that month and it was the 3rd time I reimaged the same machine that I thought, "That's odd... I think I know that machine name." Checked history, saw the 3 repair jobs I had submitted... RAM was reset, CPU was eventually…

It is indeed folly to assume that cattle have no identity. There's an internally famous video inside Google in which a particular well-known machine was unracked, dragged out into a field, and ceremonially smashed to pieces by some hardware techs. Sometimes a machine just takes an arrow to the knee and it's never the same again. Then there are all the uncontrolled or unrecorded differences between machines: the ones…

"There's an internally famous video inside Google in which a particular well-known machine was unracked, dragged out into a field, and ceremonially smashed to pieces by some hardware techs..."

Are you perhaps thinking of the printer execution scene from "Office Space"?

https://www.youtube.com/watch?v=N9wsjroVlu8

Re: A Google Cloud support engineer solves a tough DNS case

#249

Post by xi- »

luhn wrote: post #22

This is a fun debugging story, but is a great example why servers should be cattle not pets. Having trouble with a VM? Blow it up and get a fresh one. Still having trouble? The provisioning steps are codified, you can walk through them and find the one that causes the issue.

They had a perfectly valid configuration and it uncovered a bug in the Linux kernel. That wouldn't have happened had they ignored the issue and tried again.

Of course, anything necessary to get your production box producing, but a well-engineered server is worth the debugging time.

Re: A Google Cloud support engineer solves a tough DNS case

#250

Post by jwandborg »

bluedino wrote: post #210
mcguire wrote: post #129

" When sk_rcvbuf gets close to 2^31, adding the size of the packet can cause an integer overflow. And since it’s an int it becomes a negative number, therefore the condition is true when it should be false (for more, also check out this discussion of signed magnitude representation). " And this is why you don't generally use signed numbers in systems code, unless you specifically need negative numbers. And why you gr…

What happens when some dumbledork sets it to 2^32+1?

From the blog post:

> (if you try and set it to 2^31 the kernel returns “INVALID ARGUMENT”).

I would expect the same to happen for 2^31+1.

Post reply on HN