A Google Cloud support engineer solves a tough DNS case • Hacker News Board

Re: A Google Cloud support engineer solves a tough DNS case

#261

Post by GoodJokes » Wed, May 20, 2020, 2:31 PM UTC

killjoywashere wrote: post #247

MaxBarraclough wrote: post #246

Earlier quoted context omitted.

I'm not a sailor or a nuclear engineer, but that doesn't sound like a great idea. Should the Chaos Monkey approach really be used on nuclear systems? Aside: was it even legal for him to do that?

If you are responsible for building the industry that designs and builds nuclear submarines that carry nuclear missiles, you had better make sure that those submarines and their crews can handle chaos monkeys. Also, Rickover was Congress's favorite admiral. They forced the Navy to promote him. I'm pretty sure they made sure that the laws were to his liking.

Lol, as software developers it should be obvious that you can test stuff like this without endangering, uh, the world. This general sounds like a pompous tool. Like most millitary higher ups.

Re: A Google Cloud support engineer solves a tough DNS case

#262

Post by skywhopper » Wed, May 20, 2020, 2:41 PM UTC

luhn wrote: post #22

This is a fun debugging story, but is a great example why servers should be cattle not pets. Having trouble with a VM? Blow it up and get a fresh one. Still having trouble? The provisioning steps are codified, you can walk through them and find the one that causes the issue.

You don't have any information about where the customer was on the cattle/pet divide, nor if the general advice to "treat servers as cattle" even makes sense in their case. Regardless, the whole point of the exercise was to find the root cause and prevent it from happening in the future. Sometimes you gotta dig in and do the work.

Re: A Google Cloud support engineer solves a tough DNS case

#263

Post by fizwhiz » Wed, May 20, 2020, 3:47 PM UTC

the-rc wrote: post #219

dmoy wrote: post #206

Earlier quoted context omitted.

Huh, I can't find it. I can confirm I've read the same thing though, years back.

There used to be a go link for it (same as the machine name), but knowing Google, it might be stale. There's a good chance you can find more on the internal folklore site. If that one is still around, too.

Confirmed: the go link with the machine name works but I don't want to post it on HN to be safe :)

If any Googlers are reading this: just goto go/legends and search for officespace. The first link that pops up has context as to why the video exists.

Re: A Google Cloud support engineer solves a tough DNS case

#264

Post by karatestomp » Wed, May 20, 2020, 4:03 PM UTC

cameronbrown wrote: post #100

josefresco wrote: post #40

Earlier quoted context omitted.

Hey thanks! I've never heard it but given that I'm old and it (the phrase) was coined in 2011-2012 I'm not surprised.

Awesome! Everyone learns something new every day :)

We just pray we don't learn the "obvious thing everyone who's any good knows" in an interview.

Re: A Google Cloud support engineer solves a tough DNS case

#265

Post by ahupp » Wed, May 20, 2020, 4:12 PM UTC

exikyut wrote: post #242

ahupp wrote: post #117

Earlier quoted context omitted.

Also at FB: one day we got a huge spike in measured site-wide cpu usage. After the terror subsided, we found that a single request on a single machine had reported an improbably huge number of cycles (like, a billion years of cpu time). We figured a hardware problem and sent it to repair. A month later the same thing happened to the same machine; it had just been reimaged and sent back into the fleet. There was some…

Out of curiosity, do these reproducibly-broken components ever make it into an upstream testing environment?

Yes, in a few cases the hardware would go back to its manufacturer for more investigation. The ones I'm aware of were more subtly bad, and more reproducible than this one though.

Re: A Google Cloud support engineer solves a tough DNS case

#266

Post by karagenit » Wed, May 20, 2020, 4:25 PM UTC

sleepydog wrote: post #139

mcguire wrote: post #136

Earlier quoted context omitted.

For one thing, it's a clue you need to step back and think, "What happens when this overflows?" rather than "Oh, it's just a number." For another, that's why you get paranoid. (For a third, I strongly recommend something like Frama-C with the Weakest-Precondition module---it's very good at finding issues like these.)

I'm not convinced that it would have been any more obvious to the person who made the error that the variable could overflow if it were unsigned. It's also much easier, IMO, to accidentally underflow an unsigned integer; it's so much more common to work with 0 than it is to work with +/- 2 billion.

I'm being pedantic, but technically wrapping from 0 to MAX_INT is still considered overflow. Underflow refers to decimal truncation e.g. by integer division.

Re: A Google Cloud support engineer solves a tough DNS case

#267

Post by dekhn » Wed, May 20, 2020, 4:32 PM UTC

MaxBarraclough wrote: post #246

killjoywashere wrote: post #216

Earlier quoted context omitted.

Admiral Rickover was known for walking into the engineering spaces of nuclear ships and just throwing a valve handle that would force a reactor scram. Not infrequently on a submerged submarine. Just to make sure the team was on their toes.

I'm not a sailor or a nuclear engineer, but that doesn't sound like a great idea. Should the Chaos Monkey approach really be used on nuclear systems? Aside: was it even legal for him to do that?

Rickover was the sort of person who had the technical expertise, gumption, and charisma to get away with this. It's worth reading up on this amazing individual, who made navy's nuclear reactors so safe, and led the creation of nuclear reactor expertise within the navy.

Also, if you're not confident enough in your nuclear reactor to apply chaos monkey techniques, you shouldn't be engineering nuclear reactors.

Re: A Google Cloud support engineer solves a tough DNS case

#268

Post by Corrado » Wed, May 20, 2020, 5:43 PM UTC

rswail wrote: post #239

frogpelt wrote: post #217

Earlier quoted context omitted.

I was troubleshooting a computer once that would randomly shut off during boot and, one component at a time, I replaced everything on it including the motherboard to no avail. Finally I took all the parts out of the original computer and put them in a different chassis and it worked! Put them back in the old chassis and back to the old problem. Eventually I noticed that there was an extra stand-off in the first compu…

Back in the 90s we had a faulty DELL server that someone decided needed to have its BIOS upgraded. They didn't read the specs and upgraded to a BIOS not supported by the CPU. Motherboard is bricked. Ring DELL for support. After going through the rigmarole of explaining what had happened and that we had a bricked motherboard, the person on the phone said "Have you tried taking out the CPU and rebooting?" To avoid furt…

My neighbour's computer stopped working after a lightning storm and asked me to take a look at it. It wouldn't boot so I started taking things out of it (hard drive, video card, modem, etc.) and trying again.

Nothing worked. Finally, I removed the processor from the motherboard, looked at it, and reinstalled it. The computer booted right up and never had another problem. Weird.

Re: A Google Cloud support engineer solves a tough DNS case

#269

Post by lanstin » Wed, May 20, 2020, 7:00 PM UTC

dilyevsky wrote: post #102

bpodgursky wrote: post #89

Earlier quoted context omitted.

If you're spending that much money (this isn't GCP specific -- this is any cloud) you should be establishing a 1-3 year min-commit contract, and in practice, this will get negotiated through the CFO. This will get you massive discounts -- 20-30% under list price, in exchange for spending $X million/year over Y years. It will also get you a dedicated sales rep and sales team, and they will absolutely crack the whip on…

Yeah not talking about discount. The discount was nice (or so i heard, but if you do 3y commit you can get that anyway). > and they will absolutely crack the whip on internal teams to get issues resolved Not in my experience. Although we did get an ever-rotating rep. I think they changed three of them in like a year or so

Google only gives you good service if they respect you as engineers. We'd say stupid stuff and get the cold-shoulder, and then later would find some cool bug with encrypted VPNs dropping packets (with no monitoring in GCP, only our tcpdump from various places) and got some very skilled network engineers looking at the data and making code changes. They still muted us for long periods of time while talking amongst themselves, but did deliver.

Re: A Google Cloud support engineer solves a tough DNS case

#270

Post by walshemj » Wed, May 20, 2020, 8:41 PM UTC

jeffbee wrote: post #190

walshemj wrote: post #188

Earlier quoted context omitted.

Unlikely that's not how statistics works in production engineering

With respect, I just went through a "code red" at a large, well-known cloud storage company caused by synchronized late-life death of hard disks all manufactured in the same batch. That's the second time in my career that I've been through the same phenomenon. Hard disks that are made together wear out together.

Shockingly bad production engineering then.