Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Google Cloud support engineer solves a tough DNS case

cloud.google.com

Post reply on HN
Print view View original

111–120 of 283 posts

Re: A Google Cloud support engineer solves a tough DNS case

#111

Post by londons_explore »

cmeacham98 wrote: post #43
dralley wrote: post #41

Earlier quoted context omitted.

That's awfully convenient, and I can't deny having done this, but it's also a great way to never understand what went wrong.

Re-provisioning a failed server to solve the problem and taking a deep dive to find the root cause are not mutually exclusive. Essentially all VM software will allow you to snapshot/backup/clone the VM for later analysis, while also fixing your production environment _now_.

When you have a service which is acting weirdly, ideally I'd like to snapshot the VM, then do whatever I need to do to repair the service urgently.

That might involve making a new VM from scratch, but it also might twiddling some settings, or other emergency changes.

Afterwards, I want to be able to restore the VM state, probably in a firewalled off environment, so I can debug exactly what was wrong.

Sometimes I'd like to do it to a set of VM's - for example, if there is some DNS wierdness, I might want to snapshot an application server and a DNS server.

So far, no cloud provider seems to offer functionality to make that easy, which is a bit disappointing.

Re: A Google Cloud support engineer solves a tough DNS case

#112

Post by unbeli »

econcon wrote: post #17

Google support is basically non existent for customers even spending like 5-10K a month on their platform. Even questions you raise on their Reddit sub go unnoticed. They don't have bunch of people actively trying to solve their customers problem. That said the only time my problems were actually listed to were from BigQuery team. Other than this, I don't think it's possible to get any explanation on a feature from a…

Perhaps Reddit isn't a preferred support channel for them?

Re: A Google Cloud support engineer solves a tough DNS case

#113

Post by steveklabnik »

cesarb wrote: post #109
steveklabnik wrote: post #77

Earlier quoted context omitted.

I think this person is trolling. However, for the purposes of discussion, two things here: So, in Rust, overflow panics in debug builds, but does wrap around in release builds. So, it is possible this bug would have been caught in testing, but if it wasn't, it still would have slipped into production. However, that being said, Rust does not do implicit casting between numeric types. So it's very likely that this code…

AFAIK, Rust can panic on overflow even in release builds if you want it to, at a somewhat heavy performance cost (which is why this is not enabled by default in release builds). In this case, it would convert the issue from "some packets are unexpectedly being discarded" into an immediate crash within the kernel.

Yes, there is a flag to change the default behavior. I am not sure how many folks actually use this flag.

Re: A Google Cloud support engineer solves a tough DNS case

#114

Post by bb611 »

londons_explore wrote: post #81
supernova87a wrote: post #57

I'm not an expert with AWS or Google Cloud, so I'm interested in knowing: What "level" of customer or SLA do you have to be to get a certain quantity or guarantee of support and troubleshooting? Or is it that if even a free-tier customer points out something that is fundamentally a problem, it will receive attention by certain solutions engineers? Are there $ spending, 20 x (c3.4x.large), or I-pay-you-for-certain-upt…

When purchasing support, you should consider you are really buying an expert who really knows google cloud, but doesn't have special buttons to click to do things you couldn't do. If a service you depend on is down, your support agent will be able to tell you it's down, but not speed up the fix. Cloud support will have more information about performance black holes and limitations that the documents don't describe. T…

This sounds a lot like my experience with AWS support.

They really deeply understand AWS, are very responsive to calls/emails, but often have no tools to solve the problem we're having right now.

This even happened with some of their high end hardware, we did an upgrade for a critical set of instances to some extremely pricey dedicated hosts and ended up in a runaround for over a week due to a hardware issue on their side.

Re: A Google Cloud support engineer solves a tough DNS case

#115

Post by sitkack »

jeffbee wrote: post #95
storyinmemo wrote: post #74

Earlier quoted context omitted.

My fleet of machines I was the owner of at Facebook was around 10,000. I still remember the odd JVM crash that prompted me to reimage a machine. I wouldn't have remembered it except there were a few that month and it was the 3rd time I reimaged the same machine that I thought, "That's odd... I think I know that machine name." Checked history, saw the 3 repair jobs I had submitted... RAM was reset, CPU was eventually…

It is indeed folly to assume that cattle have no identity. There's an internally famous video inside Google in which a particular well-known machine was unracked, dragged out into a field, and ceremonially smashed to pieces by some hardware techs. Sometimes a machine just takes an arrow to the knee and it's never the same again. Then there are all the uncontrolled or unrecorded differences between machines: the ones…

There needs to be some level of conformity between instances, they stop being a heard and more of a zoo if the skew is too large. The workloads running on the instances shouldn't be able to tell which instance type they are running on, or your workloads should be written such that it doesn't matter (but at some point it will). Things that grow and move together, wear together, so you will end up with a system that is designed against the empirical contract, not the stated one.

Re: A Google Cloud support engineer solves a tough DNS case

#116

Post by MrMorden »

steveklabnik wrote: post #113
cesarb wrote: post #109

Earlier quoted context omitted.

AFAIK, Rust can panic on overflow even in release builds if you want it to, at a somewhat heavy performance cost (which is why this is not enabled by default in release builds). In this case, it would convert the issue from "some packets are unexpectedly being discarded" into an immediate crash within the kernel.

Yes, there is a flag to change the default behavior. I am not sure how many folks actually use this flag.

If you want flags, gcc has -Wall -Wextra -Werror which seems like it would have caught this bug. (Of course, if you weren't using -Wall -Wextra from the beginning you'll have a lot of catching up to do before you can build with -Werror.)

Re: A Google Cloud support engineer solves a tough DNS case

#117

Post by ahupp »

storyinmemo wrote: post #74
luhn wrote: post #22

This is a fun debugging story, but is a great example why servers should be cattle not pets. Having trouble with a VM? Blow it up and get a fresh one. Still having trouble? The provisioning steps are codified, you can walk through them and find the one that causes the issue.

My fleet of machines I was the owner of at Facebook was around 10,000. I still remember the odd JVM crash that prompted me to reimage a machine. I wouldn't have remembered it except there were a few that month and it was the 3rd time I reimaged the same machine that I thought, "That's odd... I think I know that machine name." Checked history, saw the 3 repair jobs I had submitted... RAM was reset, CPU was eventually…

Also at FB: one day we got a huge spike in measured site-wide cpu usage. After the terror subsided, we found that a single request on a single machine had reported an improbably huge number of cycles (like, a billion years of cpu time). We figured a hardware problem and sent it to repair. A month later the same thing happened to the same machine; it had just been reimaged and sent back into the fleet. There was some problem with the hardware performance counter where it randomly returned zero, but after that we made sure it was removed permanently.

Re: A Google Cloud support engineer solves a tough DNS case

#119

Post by anonymousisme »

brightball wrote: post #99
zimbatm wrote: post #96

Earlier quoted context omitted.

Did you read the article? The customer had set `net.core.rmem_default = 2147483647` on purpose. Which exposed a Kernel bug. The whole herd would be having the same issue.

I think what he's trying to suggest is that the customer may have been able to isolate the issue faster by walking through the provisioning settings for the machine to identify core changes. The bug report resulted in a core fix, which is a better result than if the customer had fixed it themselves of course.

Also, it was very nice of Google to follow up and submit the patch to LKML. IMHO this goes beyond the scope of their role. They could have taken a more selfish approach and accepted the bug as "normal" behavior, and advised their customer to not configure the buffer to such an enormous size.

Re: A Google Cloud support engineer solves a tough DNS case

#120

Post by indigodaddy »

anonymousisme wrote: post #119
brightball wrote: post #99

Earlier quoted context omitted.

I think what he's trying to suggest is that the customer may have been able to isolate the issue faster by walking through the provisioning settings for the machine to identify core changes. The bug report resulted in a core fix, which is a better result than if the customer had fixed it themselves of course.

Also, it was very nice of Google to follow up and submit the patch to LKML. IMHO this goes beyond the scope of their role. They could have taken a more selfish approach and accepted the bug as "normal" behavior, and advised their customer to not configure the buffer to such an enormous size.

Was thinking the exact same thought as I finished up the article...
Post reply on HN