Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Google Cloud support engineer solves a tough DNS case

cloud.google.com

Post reply on HN
Print view View original

141–150 of 283 posts

Re: A Google Cloud support engineer solves a tough DNS case

#141

Post by DethNinja »

mcguire wrote: post #129

" When sk_rcvbuf gets close to 2^31, adding the size of the packet can cause an integer overflow. And since it’s an int it becomes a negative number, therefore the condition is true when it should be false (for more, also check out this discussion of signed magnitude representation). " And this is why you don't generally use signed numbers in systems code, unless you specifically need negative numbers. And why you gr…

Signed or unsigned doesn’t matter much both can overflow.

I actually started putting assertion checks about overflow issues almost everywhere but it requires great discipline. I wonder if there is a better solution available.

Re: A Google Cloud support engineer solves a tough DNS case

#142

Post by dekhn »

jeffbee wrote: post #95
storyinmemo wrote: post #74

Earlier quoted context omitted.

My fleet of machines I was the owner of at Facebook was around 10,000. I still remember the odd JVM crash that prompted me to reimage a machine. I wouldn't have remembered it except there were a few that month and it was the 3rd time I reimaged the same machine that I thought, "That's odd... I think I know that machine name." Checked history, saw the 3 repair jobs I had submitted... RAM was reset, CPU was eventually…

It is indeed folly to assume that cattle have no identity. There's an internally famous video inside Google in which a particular well-known machine was unracked, dragged out into a field, and ceremonially smashed to pieces by some hardware techs. Sometimes a machine just takes an arrow to the knee and it's never the same again. Then there are all the uncontrolled or unrecorded differences between machines: the ones…

Once, the head of IT of a company I used to work for was touring the datacenter, passing some new racks filled with blade servers. He stopped, said "why are all the fans running full blast on this rack?" and the admins checked and they were running some test workload at scale somebody had forgotten about a few weeks before.

Everybody was embarassed because no monitoring caught it, but the VP of IT did by walking past the cage.

Re: A Google Cloud support engineer solves a tough DNS case

#143

Post by mcguire »

auspex wrote: post #58
hidiegomariani wrote: post #45

amazing. I wish I knew where to go learn the basics to navigate that many layers of knowledge (kernel/os/network)..

These books should help! Computer Networks and Internet Internet working with TCP/IP Volume III The Linux Programming interface

And...

Computer Architecture: A Quantitative Approach

Operating System Concepts

Computer Networking: A Top-Down Approach

W. Richard Stevens (Somewhat out of date, but I haven't seen anything to beat them.)

Doesn't look horrible: http://intronetworks.cs.luc.edu/

Re: A Google Cloud support engineer solves a tough DNS case

#144

Post by walshemj »

jeffbee wrote: post #95
storyinmemo wrote: post #74

Earlier quoted context omitted.

My fleet of machines I was the owner of at Facebook was around 10,000. I still remember the odd JVM crash that prompted me to reimage a machine. I wouldn't have remembered it except there were a few that month and it was the 3rd time I reimaged the same machine that I thought, "That's odd... I think I know that machine name." Checked history, saw the 3 repair jobs I had submitted... RAM was reset, CPU was eventually…

It is indeed folly to assume that cattle have no identity. There's an internally famous video inside Google in which a particular well-known machine was unracked, dragged out into a field, and ceremonially smashed to pieces by some hardware techs. Sometimes a machine just takes an arrow to the knee and it's never the same again. Then there are all the uncontrolled or unrecorded differences between machines: the ones…

back when we where buying hardware for big (at the time) intranet. All the servers where brought of the same batch of suns production line, I recall our sysadmin saying he rely wanted to do the same for the disks ie case of identical drives

Re: A Google Cloud support engineer solves a tough DNS case

#145

Post by tobykimmel »

bauerd wrote: post #125
tobykimmel wrote: post #124

The big news here is Google support engineer solves ANY problem. I can never get through to them. That’s the downside to a fully automated support system, no humans.

There are lots of support engineers around, you just need to pay Google to use up their time.

That’s not true for all services. Google Voice for example has become unusable, and this is confirmed by thousands of support forum posts. There’s no way to reach a human. All the support forum posts are eventually closed with no resolution.

Google should just shut down Google Voice instead of keeping it around for free, but broken and with no support.

Re: A Google Cloud support engineer solves a tough DNS case

#146

Post by ehsankia »

andyljones wrote: post #66
supernova87a wrote: post #57

I'm not an expert with AWS or Google Cloud, so I'm interested in knowing: What "level" of customer or SLA do you have to be to get a certain quantity or guarantee of support and troubleshooting? Or is it that if even a free-tier customer points out something that is fundamentally a problem, it will receive attention by certain solutions engineers? Are there $ spending, 20 x (c3.4x.large), or I-pay-you-for-certain-upt…

Here you go: https://cloud.google.com/support#support-plans $250/month/dev is the minimal for phone calls on technical issues, $150k + 4% of GCP spend for 'come running' support. There're more details here https://cloud.google.com/support/docs/procedures#additional_... though they use the old names for the support tiers.

It seems like the blog post talks about a written case report, which the 100$ tier has access to, albeit with 4 hour first response instead of 1 hour. So it is possible that you could get your case escalated to such an in depth debugging with that tier?

Re: A Google Cloud support engineer solves a tough DNS case

#147

Post by raziel2p »

luhn wrote: post #22

This is a fun debugging story, but is a great example why servers should be cattle not pets. Having trouble with a VM? Blow it up and get a fresh one. Still having trouble? The provisioning steps are codified, you can walk through them and find the one that causes the issue.

I can't agree with this. In the case that was debugged, the configuration change was probably legitimately changed for a valid business reason and was codified. If you have traditional VM infrastructure your provisioning may consist of 100s of small configuration changes. Are you really going to step through every single one of them manually? Configuration management tools don't exactly have a `git bisect` equivalent, and even if you did, you'd have to re-image the VM every time because VMs are stateful.

And even if you could somehow bisect every single configuration change in your configuration management, there's the added complexity of how many configuration changes are actually needed to test if the problem is still present. In this case it would probably be fairly easy because DNS is such a core "feature", but if this is something more application-level, you're really going to be lost.

Re: A Google Cloud support engineer solves a tough DNS case

#148

Post by jeffbee »

walshemj wrote: post #144
jeffbee wrote: post #95

Earlier quoted context omitted.

It is indeed folly to assume that cattle have no identity. There's an internally famous video inside Google in which a particular well-known machine was unracked, dragged out into a field, and ceremonially smashed to pieces by some hardware techs. Sometimes a machine just takes an arrow to the knee and it's never the same again. Then there are all the uncontrolled or unrecorded differences between machines: the ones…

back when we where buying hardware for big (at the time) intranet. All the servers where brought of the same batch of suns production line, I recall our sysadmin saying he rely wanted to do the same for the disks ie case of identical drives

A super-bad idea because drives made on the same week in the same facility will all fail at the same moment.

Re: A Google Cloud support engineer solves a tough DNS case

#149

Post by tedunangst »

jeffbee wrote: post #140
cesarb wrote: post #109

Earlier quoted context omitted.

AFAIK, Rust can panic on overflow even in release builds if you want it to, at a somewhat heavy performance cost (which is why this is not enabled by default in release builds). In this case, it would convert the issue from "some packets are unexpectedly being discarded" into an immediate crash within the kernel.

How heavy is the performance cost really? On x86, wouldn't a JO to a higher address take care of it? It would be a never-taken branch with perfect predictability.

At what cost to code size?

Re: A Google Cloud support engineer solves a tough DNS case

#150

Post by totaldex »

luhn wrote: post #22

This is a fun debugging story, but is a great example why servers should be cattle not pets. Having trouble with a VM? Blow it up and get a fresh one. Still having trouble? The provisioning steps are codified, you can walk through them and find the one that causes the issue.

While I agree with your premise, its important to stress that finding solutions by 'blowing up your server' and starting fresh are rarely sustainable.

It's often a useful exercise to dig into the root cause - a lot of times the problem you're seeing is just the tip of the iceberg.

Post reply on HN