A Google Cloud support engineer solves a tough DNS case • Hacker News Board

Re: A Google Cloud support engineer solves a tough DNS case

#191

Post by [deleted] » Tue, May 19, 2020, 11:22 PM UTC

sleepydog wrote: post #185

ibejoeb wrote: post #104

Earlier quoted context omitted.

I really want to know if the telemetrics (or whatever the thing was) pushed enough packets to actually warrant the config. Setting something to the max sounds like preopt to me. This must be a truly exceptional condition if it actually remained undiscovered since linux 3.

I can't think of anything that would warrant a 2GB receive buffer. The buffer should be sized so that the receiving program has a reasonable amount of time to drain it before it becomes full. A large skylake VM in GCP can do 32 Gbps (lowercase b), so assuming worst conditions, a 2GB receive buffer would give the receiver 500ms to call recv(), which is a huge amount of time for something that should take microseconds,…

[deleted]

Re: A Google Cloud support engineer solves a tough DNS case

#192

Post by jiggawatts » Tue, May 19, 2020, 11:39 PM UTC

grawlinson wrote: post #189

jeffbee wrote: post #95

Earlier quoted context omitted.

It is indeed folly to assume that cattle have no identity. There's an internally famous video inside Google in which a particular well-known machine was unracked, dragged out into a field, and ceremonially smashed to pieces by some hardware techs. Sometimes a machine just takes an arrow to the knee and it's never the same again. Then there are all the uncontrolled or unrecorded differences between machines: the ones…

One of my computers has been aptly named 'THESEUS' due to what was replaced on it. By the time it was repaired to an acceptable level, the only original component remaining was the chassis.

Heh... that reminds me of my favourite troubleshooting story!

We had a customer with regularly failing tape backups. CRC errors, verify pass failures, even failed writes, and so forth.

We replaced the tapes with new ones. Same issues.

We replaced the tape drive with a new one. Still the same problems.

We replaced the internal ribbon cable and the SCSI controller. No luck.

Firmware flashed everything. Didn't help.

New server chassis, wiped the OS and reinstalled everything from scratch. Changed the backup software just in case. The backups still failed!

Literally no part was the same. I went on site to start looking into things like the power cables, the UPS, or vibration issues. Basically were getting desperate and grasping at straws.

I was sitting down in an office, casually chatting with the IT guy while we were waiting for 5pm so we could reboot the server. He's leaning back in is office chair, and he casually picks up one of the tape cartridges and throws it up in the air and then catches it before it hits the ground. Just playing. Over and over.

I asked him if he does that a lot.

"Yes, it's fun!" he answered.

ಠ_ಠ

Re: A Google Cloud support engineer solves a tough DNS case

#193

Post by floatingatoll » Wed, May 20, 2020, 12:09 AM UTC

mav3rick wrote: post #183

floatingatoll wrote: post #7

The LKML message described in the post is here: https://lkml.org/lkml/2019/12/19/482

Something I'd like to add here the actual fix is - + if (rmem > (size + (unsigned int)sk->sk_rcvbuf)) However in reality this would have worked too - + if (rmem > (unsigned int)(size + sk->sk_rcvbuf)) (The bit pattern of the result remains the same and it's still casted as unsigned int during the comparison) However, signed integer overflow is undefined behavior in C and unsigned integer overflow isn't. Hence, the su…

Those are not safe to treat as equivalent, even if it might work in theory. You should always cast as narrowly as possible, and when you see code doing otherwise, look very carefully for bugs.

If A + (cast)B is a correct form, then (cast)(A + B) is generally an inappropriate form. As you note, it’s possible it will happen to work, but it’s not good form.

Re: A Google Cloud support engineer solves a tough DNS case

#194

Post by karmakaze » Wed, May 20, 2020, 12:09 AM UTC

jeffbee wrote: post #95

storyinmemo wrote: post #74

Earlier quoted context omitted.

My fleet of machines I was the owner of at Facebook was around 10,000. I still remember the odd JVM crash that prompted me to reimage a machine. I wouldn't have remembered it except there were a few that month and it was the 3rd time I reimaged the same machine that I thought, "That's odd... I think I know that machine name." Checked history, saw the 3 repair jobs I had submitted... RAM was reset, CPU was eventually…

It is indeed folly to assume that cattle have no identity. There's an internally famous video inside Google in which a particular well-known machine was unracked, dragged out into a field, and ceremonially smashed to pieces by some hardware techs. Sometimes a machine just takes an arrow to the knee and it's never the same again. Then there are all the uncontrolled or unrecorded differences between machines: the ones…

There was a time when I managed a few clusters of MySQL servers provisioned on OVH bare-metal. [Yes, we know "they're the worst" at least used to be.] Everything was running well until we upgraded to their largest offerings. Turned out they had NUMA memory so had to retune the MySQL parameters. After sorting that out they all worked fine except one machine. Note that we already had a pretty good relationship so I'd already requested the exact same motherboards and BIOS on these machines. Somehow one machine always came up different. I couldn't trust OVH to set the CMOS the same so I carefully checked against one of the 'normal' machines. The cores reported differently and had different affinity characteristics. Never did find out why that was, perhaps a CPU microcode/stepping that I never investigated. Anyway I just worked around the issue (with a different tuning for this one machine) because asking for a replacement resulted in a similar situation.

Re: A Google Cloud support engineer solves a tough DNS case

#195

Post by [deleted] » Wed, May 20, 2020, 12:16 AM UTC

gerdesj wrote: post #180

dekhn wrote: post #142

Earlier quoted context omitted.

Once, the head of IT of a company I used to work for was touring the datacenter, passing some new racks filled with blade servers. He stopped, said "why are all the fans running full blast on this rack?" and the admins checked and they were running some test workload at scale somebody had forgotten about a few weeks before. Everybody was embarassed because no monitoring caught it, but the VP of IT did by walking past…

I walk the server room daily, every morning. I've tended our monitoring system for 15 years now and I don't trust myself to be infallible. I'm also the MD ...

[deleted]

Re: A Google Cloud support engineer solves a tough DNS case

#196

Post by mav3rick » Wed, May 20, 2020, 12:36 AM UTC

floatingatoll wrote: post #193

mav3rick wrote: post #183

Earlier quoted context omitted.

Something I'd like to add here the actual fix is - + if (rmem > (size + (unsigned int)sk->sk_rcvbuf)) However in reality this would have worked too - + if (rmem > (unsigned int)(size + sk->sk_rcvbuf)) (The bit pattern of the result remains the same and it's still casted as unsigned int during the comparison) However, signed integer overflow is undefined behavior in C and unsigned integer overflow isn't. Hence, the su…

Those are not safe to treat as equivalent, even if it might work in theory. You should always cast as narrowly as possible, and when you see code doing otherwise, look very carefully for bugs. If A + (cast)B is a correct form, then (cast)(A + B) is generally an inappropriate form. As you note, it’s possible it will happen to work, but it’s not good form.

Yes I qualified it at the end why the former is the correct solution and not the latter.

Re: A Google Cloud support engineer solves a tough DNS case

#197

Post by atombender » Wed, May 20, 2020, 12:37 AM UTC

lanstin wrote: post #179

atombender wrote: post #98

Earlier quoted context omitted.

If you use an automated configuration management system such as Puppet, you don't ever run sysctl manually in a shell. Instead, everything is controlled by the configuration management system. sysctl is a bit problematic in terms of exhaustiveness. That is, how do you ensure that the kernel only has its original values plus whatever you put in sysctl.conf, and nobody actually ran sysctl manually at some point? But it…

I have seen so much random behavior from puppet runs. It's basically a big fancy wrapper around a bunch of shell commands (much better than the raw shell commands) but subject to all the bizarre race conditions and so on. We had to wait 30 minutes to use a newly created VM so that puppet had run three times, and it was >0.99 likely to be good now. (If it wasn't, it was killed and we retried; 30 minutes was chosen to…

Puppet can be difficult to get right. Dependencies are _very_ hard to get right, despite the fact that Puppet is virtually designed around the idea of dependencies. I'm a fan of the concept, less a fan of the execution.

Unfortunately, the competition (Salt, Ansible, Chef) aren't really any better here.

These days, I run Kubernetes whenever possible, and keep the base OS light, which makes the configuration management surface extremely small.

Re: A Google Cloud support engineer solves a tough DNS case

#198

Post by resonantjacket5 » Wed, May 20, 2020, 12:40 AM UTC

walshemj wrote: post #188

jeffbee wrote: post #148

Earlier quoted context omitted.

A super-bad idea because drives made on the same week in the same facility will all fail at the same moment.

Unlikely that's not how statistics works in production engineering

Your default assumption only works if every disk has an independent probability of failing from each other. Which is definitely not true if you buy all the disks from the same batch.

Re: A Google Cloud support engineer solves a tough DNS case

#199

Post by solarengineer » Wed, May 20, 2020, 12:44 AM UTC

jeffbee wrote: post #190

walshemj wrote: post #188

Earlier quoted context omitted.

Unlikely that's not how statistics works in production engineering

With respect, I just went through a "code red" at a large, well-known cloud storage company caused by synchronized late-life death of hard disks all manufactured in the same batch. That's the second time in my career that I've been through the same phenomenon. Hard disks that are made together wear out together.

I can confirm this. I learned the hard way to buy hard disk drives of the same model but from different batches.

Re: A Google Cloud support engineer solves a tough DNS case

#200

Post by fizwhiz » Wed, May 20, 2020, 12:47 AM UTC

jeffbee wrote: post #95

storyinmemo wrote: post #74

Earlier quoted context omitted.

My fleet of machines I was the owner of at Facebook was around 10,000. I still remember the odd JVM crash that prompted me to reimage a machine. I wouldn't have remembered it except there were a few that month and it was the 3rd time I reimaged the same machine that I thought, "That's odd... I think I know that machine name." Checked history, saw the 3 repair jobs I had submitted... RAM was reset, CPU was eventually…

It is indeed folly to assume that cattle have no identity. There's an internally famous video inside Google in which a particular well-known machine was unracked, dragged out into a field, and ceremonially smashed to pieces by some hardware techs. Sometimes a machine just takes an arrow to the knee and it's never the same again. Then there are all the uncontrolled or unrecorded differences between machines: the ones…

> There's an internally famous video inside Google in which a particular well-known machine was unracked, dragged out into a field, and ceremonially smashed to pieces by some hardware techs.

Eyy what would I search on moma to find this video?