Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Google Cloud support engineer solves a tough DNS case

cloud.google.com

Post reply on HN
Print view View original

231–240 of 283 posts

Re: A Google Cloud support engineer solves a tough DNS case

#231

Post by amessina1 »

programmertote wrote: post #169
amessina1 wrote: post #84

Earlier quoted context omitted.

Author of the article here. I only thought of the possibility of making a blog post after the case was closed and I started telling my colleagues about it, and realized I would have loved to read about this. The case was indeed fun to work with, but the main reason why it had such a fast and happy resolution was because the customer was very responsive and very cooperative. I cannot talk for every Technical Solution…

I'm inspired by how much you seem to know about the details of computer network stuff. Is that a required knowledge to become a Google Tech Support person or you are just above average in terms of that among your peers? Also, I wonder how you learn all these knowledge (that is, asking for recommendation on a few books/resources for learning) if you don't mind sharing. Thanks in advance!

I don't have deep knowledge of details of compute networks, there is a team of TSE who deal with network cases who know more than me. But the whole point of troubleshooting is not knowing what is wrong, but being able to find what is wrong. In order to do that you need good basis, and those you can make by studying how networks and linux systems work (someone here posted some titles) and with experience (I have some grey hair myself). But every time you troubleshoot something you end up touching something you don't know, and that's where you learn something new you might use next time. For example I didn't know about dropwatch, a colleague suggested it to me.

During the interview process at Google we don't expect candidates to be able to get to this level of depth, but we try to hire candidates that could, over time and depending on their skill set, potentially reach a similar level of depth and ability to troubleshoot cases.

Re: A Google Cloud support engineer solves a tough DNS case

#232

Post by mjevans »

fizwhiz wrote: post #200
jeffbee wrote: post #95

Earlier quoted context omitted.

It is indeed folly to assume that cattle have no identity. There's an internally famous video inside Google in which a particular well-known machine was unracked, dragged out into a field, and ceremonially smashed to pieces by some hardware techs. Sometimes a machine just takes an arrow to the knee and it's never the same again. Then there are all the uncontrolled or unrecorded differences between machines: the ones…

> There's an internally famous video inside Google in which a particular well-known machine was unracked, dragged out into a field, and ceremonially smashed to pieces by some hardware techs. Eyy what would I search on moma to find this video?

Sounds like they were re-enacting Office Space. If you haven't seen that movie... It's culturally relevant even today. Has a bit of profanity and such though.

Re: A Google Cloud support engineer solves a tough DNS case

#233

Post by gwillen »

the-rc wrote: post #181
jeffbee wrote: post #160

Earlier quoted context omitted.

Because of the way engineers habitually run batch jobs with more replicas than there are machines, this one broken computer had crapped up every map/reduce job in that facility for a long time, and it had been sent to repairs many times without benefit. Many people knew instinctively that if their job was stuck it was probably because of the shard on xyz42 (or whatever the node name was).

I still remember the machine name. It starts with an l and ends with a 6. Over the course of a couple of years, pretty much all of its components (CPUs, RAM, drives) were replaced at least once. You could look up its maintenance history and it went on and on. I'm not sure if it was well known across all of engineering; from what I recall, it was in a cluster in Oregon reserved for a specific team. Because it was comp…

My recollection, assuming it's the same machine I'm thinking of, is that it wasn't reserved for our team; rather, we left a do-nothing job permanently allocated to it, in order to prevent some poor other sucker from getting their job scheduled on it. (Because we, through painful experience, were well aware the machine had hardware problems; but we had long since given up on convincing the responsible parties to take it out of the pool, since it passed all their internal tests every time we complained. I don't remember how long this situation existed before someone finally took it out back and shot it.)

Could be a different incident and a different machine, though. I'm sure this story happened more than once.

Re: A Google Cloud support engineer solves a tough DNS case

#234

Post by bostik »

atombender wrote: post #197
lanstin wrote: post #179

Earlier quoted context omitted.

I have seen so much random behavior from puppet runs. It's basically a big fancy wrapper around a bunch of shell commands (much better than the raw shell commands) but subject to all the bizarre race conditions and so on. We had to wait 30 minutes to use a newly created VM so that puppet had run three times, and it was >0.99 likely to be good now. (If it wasn't, it was killed and we retried; 30 minutes was chosen to…

Puppet can be difficult to get right. Dependencies are _very_ hard to get right, despite the fact that Puppet is virtually designed around the idea of dependencies. I'm a fan of the concept, less a fan of the execution. Unfortunately, the competition (Salt, Ansible, Chef) aren't really any better here. These days, I run Kubernetes whenever possible, and keep the base OS light, which makes the configuration management…

After years of pain, I've come to appreciate what was once relayed to me. All configuration management software is broken. They are equally terrible, each in their own merry way. The only thing you get to do is to choose the one that sucks the least for your use-case, and two years down the line hope that you made the right choice.

Which is why I have come to believe that the very concept of host configuration management is broken. We should do it as little as possible, preferably NONE AT ALL. Sure, use something like Ansible to run the image creation steps, and provision the necessary first-boot scripts in place. Only leave the steps in that absolutely can not be done during image pre-bake.

Cycle your hosts without mercy, so that new ones are brought up from fresh pre-baked images, continuously.

And even for the few unavoidable snowflake hosts (eg. those that have to live outside the K8S cluster), follow the same strategy. Make them disposable, so that you can bring up a new one from their own pre-baked images on demand. Try to keep the delta between the snowflake base and your cattle base as small as possible.

Configuring live hosts should be considered an anti-pattern - if you find yourself doing it at all, take a step back and consider how to get rid of the need.

Re: A Google Cloud support engineer solves a tough DNS case

#235

Post by koheripbal »

FreeFull wrote: post #130
mcguire wrote: post #129

" When sk_rcvbuf gets close to 2^31, adding the size of the packet can cause an integer overflow. And since it’s an int it becomes a negative number, therefore the condition is true when it should be false (for more, also check out this discussion of signed magnitude representation). " And this is why you don't generally use signed numbers in systems code, unless you specifically need negative numbers. And why you gr…

I'm not sure how using an unsigned number would help, given that when it overflows you're still going to have some code do unexpected stuff anyway.

well... at least it takes twice as long to overflow.

Re: A Google Cloud support engineer solves a tough DNS case

#236

Post by bigiain »

walshemj wrote: post #188
jeffbee wrote: post #148

Earlier quoted context omitted.

A super-bad idea because drives made on the same week in the same facility will all fail at the same moment.

Unlikely that's not how statistics works in production engineering

Not once, but twice:

https://www.techradar.com/news/new-bug-destroys-hpe-ssds-aft...

"Hewlett Packard Enterprise (HPE) has once again issued a warning to its customers that some of its Serial-Attached SCSI solid-state drives will fail after 40,000 hours of operation unless a critical patch is applied.

Back in November of last year, the company sent out a similar message to its customers after a firmware defect in its SSDs caused them to fail after running for 32,768 hours."

Can you imagine provisioning and deploying a rack or 3 full of shiny new identical drives, all in RAID6 or RAID10, so you couldn't possibly lose any data without multiple drives all failing at once...

(Evidence that the universe can and does invent better idiots...)

Re: A Google Cloud support engineer solves a tough DNS case

#237

Post by tallanvor »

schoolornot wrote: post #29

I've been supporting AWS environments almost from the beginning but can't ever remember a case where I was asked or even considered offering Support a copy of a VM's storage volume. Is this common on Google/Azure/etc.?

It's not common, but it certainly happens. Sometimes you just can't reproduce a problem without specific data.

Re: A Google Cloud support engineer solves a tough DNS case

#239

Post by rswail »

frogpelt wrote: post #217
grawlinson wrote: post #189

Earlier quoted context omitted.

One of my computers has been aptly named 'THESEUS' due to what was replaced on it. By the time it was repaired to an acceptable level, the only original component remaining was the chassis.

I was troubleshooting a computer once that would randomly shut off during boot and, one component at a time, I replaced everything on it including the motherboard to no avail. Finally I took all the parts out of the original computer and put them in a different chassis and it worked! Put them back in the old chassis and back to the old problem. Eventually I noticed that there was an extra stand-off in the first compu…

Back in the 90s we had a faulty DELL server that someone decided needed to have its BIOS upgraded. They didn't read the specs and upgraded to a BIOS not supported by the CPU.

Motherboard is bricked. Ring DELL for support. After going through the rigmarole of explaining what had happened and that we had a bricked motherboard, the person on the phone said "Have you tried taking out the CPU and rebooting?"

To avoid further delay in getting a replacement sent (we had 4 hour on-site at the time), we went through the motions. Not surprisingly, the motherboard was substantially bricked without a CPU.

The DELL engineer that came on-site was suitably amused.

Re: A Google Cloud support engineer solves a tough DNS case

#240

Post by exikyut »

jeffbee wrote: post #160
obmelvin wrote: post #158

Earlier quoted context omitted.

> particular well-known machine was unracked are you able to comment a bit further on why this machine was well known?

Because of the way engineers habitually run batch jobs with more replicas than there are machines, this one broken computer had crapped up every map/reduce job in that facility for a long time, and it had been sent to repairs many times without benefit. Many people knew instinctively that if their job was stuck it was probably because of the shard on xyz42 (or whatever the node name was).

In what way had the jobs failed? Very open-ended question :) but just coming from a hardware-diagnosis standpoint. (I guess the canonical answer is "here's the repair history," but yeah, duh.)

> engineers habitually run batch jobs with more replicas than there are machines

Idly curious, how do I parse parse this? It sounds like the same jobs are replicated to multiple machines as a sort of asynchronous, eventually-consistent lockstep arrangement?

Post reply on HN