Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Google Cloud support engineer solves a tough DNS case

cloud.google.com

Post reply on HN
Print view View original

91–100 of 283 posts

Re: A Google Cloud support engineer solves a tough DNS case

#91

Post by cortesoft »

kelnos wrote: post #56
Axsuul wrote: post #28

Earlier quoted context omitted.

Do you have a support plan?

Why should you need a support plan for a product you're paying for? "Ok, you can pay us $X/mo for the service, but if something goes wrong, we won't help you unless you also pay an additional $Y/mo." It's absolute garbage that this is where the industry is.

Why is it bad to give the option to not pay for support you don't want? If they didn't charge separately for it, that means the cost is distributed to everyone in terms of higher costs for the service itself.

If you think support is always necessary, then just do the math yourself and add in the cost of support for every product, and use that price to determine if it is worth it or not.

Re: A Google Cloud support engineer solves a tough DNS case

#92

Post by jeffbee »

atombender wrote: post #53
throwaway2048 wrote: post #46

Earlier quoted context omitted.

plenty of issues are not deterministic, even with 100% of everything managed by configuration management software.

But /etc/sysctl.conf is deterministic.

What is a reason why that file would correspond to the actual sysctls in effect?

Re: A Google Cloud support engineer solves a tough DNS case

#93

Post by user5994461 »

itsmemattchung wrote: post #2

> they use raw sockets! Raw sockets are different than normal sockets: they bypass iptables, and they are not buffered! Can someone elaborate on the above statement from the article? Does this imply that raw sockets have unbounded buffer?

In short, raw sockets can push bytes into the network card.

Whereas the commonly used socket functions recv/send construct the required headers for TCP, UDP and whatnot, they handle encapsulation/buffering/connection/etc so they're easy to use for developers, just read and write application data.

By nature raw sockets skip TCP/UDP libraries and a good chunk of the network kernel code. Including the place where the bug was located.

Re: A Google Cloud support engineer solves a tough DNS case

#94

Post by x86_64Ubuntu »

kelnos wrote: post #56
Axsuul wrote: post #28

Earlier quoted context omitted.

Do you have a support plan?

Why should you need a support plan for a product you're paying for? "Ok, you can pay us $X/mo for the service, but if something goes wrong, we won't help you unless you also pay an additional $Y/mo." It's absolute garbage that this is where the industry is.

I don't know, lot's of tech enterprises will give you the product for free and make it up on "support". Fortunately, a lot of these environments have forums and such which means you will eventually get help, but not soon enough if your hair is on fire.

Re: A Google Cloud support engineer solves a tough DNS case

#95

Post by jeffbee »

storyinmemo wrote: post #74
luhn wrote: post #22

This is a fun debugging story, but is a great example why servers should be cattle not pets. Having trouble with a VM? Blow it up and get a fresh one. Still having trouble? The provisioning steps are codified, you can walk through them and find the one that causes the issue.

My fleet of machines I was the owner of at Facebook was around 10,000. I still remember the odd JVM crash that prompted me to reimage a machine. I wouldn't have remembered it except there were a few that month and it was the 3rd time I reimaged the same machine that I thought, "That's odd... I think I know that machine name." Checked history, saw the 3 repair jobs I had submitted... RAM was reset, CPU was eventually…

It is indeed folly to assume that cattle have no identity. There's an internally famous video inside Google in which a particular well-known machine was unracked, dragged out into a field, and ceremonially smashed to pieces by some hardware techs. Sometimes a machine just takes an arrow to the knee and it's never the same again. Then there are all the uncontrolled or unrecorded differences between machines: the ones at the tops of the racks, or the ends of the rows, are hotter (or colder); there's some difference between the same model of hard disk made in Hungary compared to the ones made in Mexico; at some date the BIOS vendor made an undocumented firmware revision that changes an obscure energy/performance register in your CPU; you have a machine with a dead CMOS battery that worked normally until it was rebooted.

Cattle is a good philosophy but it takes a huge amount of work to approach perfection.

Re: A Google Cloud support engineer solves a tough DNS case

#96

Post by zimbatm »

luhn wrote: post #22

This is a fun debugging story, but is a great example why servers should be cattle not pets. Having trouble with a VM? Blow it up and get a fresh one. Still having trouble? The provisioning steps are codified, you can walk through them and find the one that causes the issue.

Did you read the article?

The customer had set `net.core.rmem_default = 2147483647` on purpose. Which exposed a Kernel bug. The whole herd would be having the same issue.

Re: A Google Cloud support engineer solves a tough DNS case

#97

Post by mushishi »

amessina1 wrote: post #84
kelnos wrote: post #61

Earlier quoted context omitted.

The cynical part of me expects that this case was handled so well because a) the support people found the issue fascinating and fun to work on, b) the post-mort on it would make an excellent blog post. On the flip side, it's encouraging that they have people somewhere in the support chain who are capable enough to read Linux kernel code and submit fixes upstream.

Author of the article here. I only thought of the possibility of making a blog post after the case was closed and I started telling my colleagues about it, and realized I would have loved to read about this. The case was indeed fun to work with, but the main reason why it had such a fast and happy resolution was because the customer was very responsive and very cooperative. I cannot talk for every Technical Solution…

Thank you for writing it. Never really had a glimpse of networking debugging process.

Re: A Google Cloud support engineer solves a tough DNS case

#98

Post by atombender »

jeffbee wrote: post #92
atombender wrote: post #53

Earlier quoted context omitted.

But /etc/sysctl.conf is deterministic.

What is a reason why that file would correspond to the actual sysctls in effect?

If you use an automated configuration management system such as Puppet, you don't ever run sysctl manually in a shell. Instead, everything is controlled by the configuration management system.

sysctl is a bit problematic in terms of exhaustiveness. That is, how do you ensure that the kernel only has its original values plus whatever you put in sysctl.conf, and nobody actually ran sysctl manually at some point? But it's possible to do.

Re: A Google Cloud support engineer solves a tough DNS case

#99

Post by brightball »

zimbatm wrote: post #96
luhn wrote: post #22

This is a fun debugging story, but is a great example why servers should be cattle not pets. Having trouble with a VM? Blow it up and get a fresh one. Still having trouble? The provisioning steps are codified, you can walk through them and find the one that causes the issue.

Did you read the article? The customer had set `net.core.rmem_default = 2147483647` on purpose. Which exposed a Kernel bug. The whole herd would be having the same issue.

I think what he's trying to suggest is that the customer may have been able to isolate the issue faster by walking through the provisioning settings for the machine to identify core changes.

The bug report resulted in a core fix, which is a better result than if the customer had fixed it themselves of course.

Re: A Google Cloud support engineer solves a tough DNS case

#100

Post by cameronbrown »

josefresco wrote: post #40
cameronbrown wrote: post #37

Earlier quoted context omitted.

It's a well known phrase: https://devops.stackexchange.com/questions/653/what-is-the-d...

Hey thanks! I've never heard it but given that I'm old and it (the phrase) was coined in 2011-2012 I'm not surprised.

Awesome! Everyone learns something new every day :)
Post reply on HN