Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Google Cloud support engineer solves a tough DNS case

cloud.google.com

Post reply on HN
Print view View original

21–30 of 283 posts

Re: A Google Cloud support engineer solves a tough DNS case

#21

Post by rantwasp »

donalhunt wrote: post #12
rantwasp wrote: post #4

that’s a pretty good and detailed explanation. 2 things: 1) i hope they have a runbook for situations like this (ie the support engineer does not have to figure all this on the fly) 2) the customer should have provides more details and maybe should have thought of the tweaks they made (classic solution is to compare 2 instances - one works one does not)

You can't runbook these sort of issues... You can ensure you have the knowledge and tools to root cause the problem.

you most definitely can and you should. there are steps that you can do to gather the info and the linked example shows basic things to try. when you exhaust the run-book is when you start digging

Re: A Google Cloud support engineer solves a tough DNS case

#22

Post by luhn »

This is a fun debugging story, but is a great example why servers should be cattle not pets. Having trouble with a VM? Blow it up and get a fresh one. Still having trouble? The provisioning steps are codified, you can walk through them and find the one that causes the issue.

Re: A Google Cloud support engineer solves a tough DNS case

#23

Post by atombender »

TwoBit wrote: post #18

The customer had set an extremely large buffer size and nobody thought to mention that? Perhaps the individual reporting the problem was different and was unaware of that unusual change.

To be fair, there are a lot of sysctl settings. To be sure, it's one of the first places I would look for networking weirdness, but it's also often hard to tell what impact those settings have on anything.

Re: A Google Cloud support engineer solves a tough DNS case

#24

Post by ser_tyrion »

> they use raw sockets! Raw sockets are different than normal sockets: they bypass iptables

But this bugreport says raw sockets would be filtered by the OUTPUT chain of iptables: https://bugzilla.redhat.com/show_bug.cgi?id=1269914#c4

Is that accurate across distros? It does make sense for some socket types, like device sockets, to not be routed through iptables.

Re: A Google Cloud support engineer solves a tough DNS case

#25

Post by dtech »

jldugger wrote: post #10

> After another spin around the world the case comes back to our team. So basically, follow the sun doesn't work for hard problems? Can you really say the people are working on this 24/7 if progress is only made in one time zone?

By the time his team had it, numerous things had been investigated and tried, and the customer already had a workaround. Presumably these things all happened in the other time zones.

By the time it stopped transferring, it was already diagnosed as a hard problem and the time pressure was off.

Re: A Google Cloud support engineer solves a tough DNS case

#26

Post by daenz »

luhn wrote: post #22

This is a fun debugging story, but is a great example why servers should be cattle not pets. Having trouble with a VM? Blow it up and get a fresh one. Still having trouble? The provisioning steps are codified, you can walk through them and find the one that causes the issue.

The end result was a kernel patch to LKML, so I for one am happy they solved this problem at the source.

Re: A Google Cloud support engineer solves a tough DNS case

#27

Post by atombender »

luhn wrote: post #22

This is a fun debugging story, but is a great example why servers should be cattle not pets. Having trouble with a VM? Blow it up and get a fresh one. Still having trouble? The provisioning steps are codified, you can walk through them and find the one that causes the issue.

If a freshly provisioned VM doesn't have the same issue, then they're not using automated configuration management (Puppet, Chef, or similar) for these settings, and then they have a more serious problem, as nothing in their runtime environment is "codified" or predictable.

Re: A Google Cloud support engineer solves a tough DNS case

#28

Post by Axsuul »

econcon wrote: post #17

Google support is basically non existent for customers even spending like 5-10K a month on their platform. Even questions you raise on their Reddit sub go unnoticed. They don't have bunch of people actively trying to solve their customers problem. That said the only time my problems were actually listed to were from BigQuery team. Other than this, I don't think it's possible to get any explanation on a feature from a…

Do you have a support plan?

Re: A Google Cloud support engineer solves a tough DNS case

#30

Post by ellard »

Axsuul wrote: post #28
econcon wrote: post #17

Google support is basically non existent for customers even spending like 5-10K a month on their platform. Even questions you raise on their Reddit sub go unnoticed. They don't have bunch of people actively trying to solve their customers problem. That said the only time my problems were actually listed to were from BigQuery team. Other than this, I don't think it's possible to get any explanation on a feature from a…

Do you have a support plan?

Support plans make sense for products/software that are sold and expected to operate independently (cars as always are a good example). Heck, even those products generally come with some sort of free limited time support which come in the form of warranties. Services requiring a separate support plan is... not intuitive.
Post reply on HN