Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Google Cloud support engineer solves a tough DNS case

cloud.google.com

Post reply on HN
Print view View original

251–260 of 283 posts

Re: A Google Cloud support engineer solves a tough DNS case

#251

Post by the-rc »

gwillen wrote: post #233
the-rc wrote: post #181

Earlier quoted context omitted.

I still remember the machine name. It starts with an l and ends with a 6. Over the course of a couple of years, pretty much all of its components (CPUs, RAM, drives) were replaced at least once. You could look up its maintenance history and it went on and on. I'm not sure if it was well known across all of engineering; from what I recall, it was in a cluster in Oregon reserved for a specific team. Because it was comp…

My recollection, assuming it's the same machine I'm thinking of, is that it wasn't reserved for our team; rather, we left a do-nothing job permanently allocated to it, in order to prevent some poor other sucker from getting their job scheduled on it. (Because we, through painful experience, were well aware the machine had hardware problems; but we had long since given up on convincing the responsible parties to take…

Maybe a different machine? I meant that it was not in one of the general-purpose clusters: the entire pool was dedicated and a random team couldn't request Borg quota in it. For years, though, half of the Oregon datacenter was special for one reason or another.

The infamous machine did go through repairs and part swaps many times, as you could see from its long and troubled hwops history.

The worst machines were the zombies with NICs bad enough to break Stubby RPCs, but still passing heartbeat checks. Or breaking connections only when (re)using specific ports. Fun times!

Re: A Google Cloud support engineer solves a tough DNS case

#252

Post by techntoke »

storyinmemo wrote: post #74
luhn wrote: post #22

This is a fun debugging story, but is a great example why servers should be cattle not pets. Having trouble with a VM? Blow it up and get a fresh one. Still having trouble? The provisioning steps are codified, you can walk through them and find the one that causes the issue.

My fleet of machines I was the owner of at Facebook was around 10,000. I still remember the odd JVM crash that prompted me to reimage a machine. I wouldn't have remembered it except there were a few that month and it was the 3rd time I reimaged the same machine that I thought, "That's odd... I think I know that machine name." Checked history, saw the 3 repair jobs I had submitted... RAM was reset, CPU was eventually…

> Cattle is a threshold, but when the same problem keeps coming up it's time to call the vet.

If CPU was bad, then that means that you kept running the instance on the same node. Quick was to test if it was "cattle" would have been to try on a different node

Additionally, if CPU was bad how was it not affecting other services?

Re: A Google Cloud support engineer solves a tough DNS case

#253

Post by nexuist »

MaxBarraclough wrote: post #246
killjoywashere wrote: post #216

Earlier quoted context omitted.

Admiral Rickover was known for walking into the engineering spaces of nuclear ships and just throwing a valve handle that would force a reactor scram. Not infrequently on a submerged submarine. Just to make sure the team was on their toes.

I'm not a sailor or a nuclear engineer, but that doesn't sound like a great idea. Should the Chaos Monkey approach really be used on nuclear systems? Aside: was it even legal for him to do that?

In general, one would hope that a nuclear system is designed such that the problem can easily be corrected if a single button or lever is accidentally pressed. It would be quite a terrible system if you could e.g. trigger a meltdown with just one action.

Re: A Google Cloud support engineer solves a tough DNS case

#254

Post by bogomipz »

I enjoyed reading this but wouldn't have running either "netstat -s" or "ss -s" to show protocol statistic have shown either receive buffer errors/receive packet errors statistics? It seems like this basic tool was noticeably absent from the early troubleshooting steps and other standard troubleshooting tools used.

I understand the importance of the ultimate fix but wouldn't seeing an incrementing error counter for UDP have shortened some of the troubleshooting done to identify and resolve the customer's immediate issue?

Re: A Google Cloud support engineer solves a tough DNS case

#255

Post by clubdorothe »

storyinmemo wrote: post #74
luhn wrote: post #22

This is a fun debugging story, but is a great example why servers should be cattle not pets. Having trouble with a VM? Blow it up and get a fresh one. Still having trouble? The provisioning steps are codified, you can walk through them and find the one that causes the issue.

My fleet of machines I was the owner of at Facebook was around 10,000. I still remember the odd JVM crash that prompted me to reimage a machine. I wouldn't have remembered it except there were a few that month and it was the 3rd time I reimaged the same machine that I thought, "That's odd... I think I know that machine name." Checked history, saw the 3 repair jobs I had submitted... RAM was reset, CPU was eventually…

I'm curious, how many machines have FB?

Re: A Google Cloud support engineer solves a tough DNS case

#256

Post by jfk13 »

simook wrote: post #225
magicalhippo wrote: post #152

Earlier quoted context omitted.

> It is indeed folly to assume that cattle have no identity. My dad had a book over almost all the cow names in Norway[1], per 1988. As a kid I found it rather fun to just flip through it and read some names, often wondering how they came up with them. However since then it seems the tradition of naming cattle has dropped[2] to less than 30%. [1]: "Gullhorn og dei andre : kunamn i Noreg" https://urn.nb.no/URN:NBN:no-…

I only know of small farmers who name their livestock, but that was back in the States.

Compared to US-scale farming, I'd guess most Norwegian farmers are "small".

Re: A Google Cloud support engineer solves a tough DNS case

#257

Post by bovermyer »

MaxBarraclough wrote: post #246
killjoywashere wrote: post #216

Earlier quoted context omitted.

Admiral Rickover was known for walking into the engineering spaces of nuclear ships and just throwing a valve handle that would force a reactor scram. Not infrequently on a submerged submarine. Just to make sure the team was on their toes.

I'm not a sailor or a nuclear engineer, but that doesn't sound like a great idea. Should the Chaos Monkey approach really be used on nuclear systems? Aside: was it even legal for him to do that?

A reactor scram is basically just an emergency shutdown. If I were a nuclear engineer, it might give me a heart attack to hear the scram alarms, but I would be plenty happy knowing the scram works.

Re: A Google Cloud support engineer solves a tough DNS case

#258

Post by magicalhippo »

jfk13 wrote: post #256
simook wrote: post #225

Earlier quoted context omitted.

I only know of small farmers who name their livestock, but that was back in the States.

Compared to US-scale farming, I'd guess most Norwegian farmers are "small".

Indeed. When the 1988 study was done, which resulted in the book of cow names among other things, there were about 360k cows in Norway total, and the average number of cows per farm was ~5.6.

These days the number has risen, IIRC around 35, though that's still quite a low number compared to larger countries I imagine. I'm pretty sure the variance is quite high however, with a fair number of farms with just a few cows dragging down the mean.

Re: A Google Cloud support engineer solves a tough DNS case

#259

Post by erikig »

amessina1 wrote: post #68
erikig wrote: post #54

“...This means that the case will Follow the Sun by default, to provide 24/7 support” I love the concept of “Follow the Sun” to describe 24/7 support - I don’t think I’ve heard it described that way. I wonder how much we’d have to spend to get that tier of service?

For Google Cloud, 250$ per month per user: https://cloud.google.com/support

Thank you.

Re: A Google Cloud support engineer solves a tough DNS case

#260

Post by peterwwillis »

kccqzy wrote: post #123
peterwwillis wrote: post #108

Note: You shouldn't use int , unsigned int , char , short , long . Use int16_t , uint16_t , uint8_t , etc (or their _fast equivalents) from stdint.h . The former's sizes change based on platform, cpu, and compiler; the latter are fixed-width (or flexible, where _fast may use a larger size if it's faster). I started brushing up on my C recently and have been collecting these little nuggets: https://gist.github.com/pet…

Not when you are writing the Linux kernel, when you know exactly which sizes the integers are. Or even when you are writing low-level code on a known platform (e.g. an LP64 platform).

All the typical sizes depend on the compiler and the flags you provide. If you provide the wrong flags, the sizes of each type may change, but you won't see any warnings about it because it's expected behavior, and now you've got different binaries with different behavior. Or you could use fixed-width types and if the wrong flags get passed (no C99 support) your code just doesn't compile.

I just checked the Linux kernel style guide, and they explicitly suggest you can use fixed-width types from C99 when it makes sense. I get that they want a balance for their project, but for general programming, it's just safer to be explicit. https://www.kernel.org/doc/html/v5.1/process/coding-style.ht...

Post reply on HN