if (rmem > (size + sk->sk_rcvbuf))
goto uncharge_drop;
A Google Cloud support engineer solves a tough DNS case
31–40 of 283 posts
Re: A Google Cloud support engineer solves a tough DNS case
#32I've been supporting AWS environments almost from the beginning but can't ever remember a case where I was asked or even considered offering Support a copy of a VM's storage volume. Is this common on Google/Azure/etc.?
It reminds me of my friend's hosting company that failed. They got a big customer and created a VM and the customer asked to fix a problem that involved getting a shell in the VM. Friend does it and the customer is gone next day.
This aside, even though we had too many support cases so far with AWS, and having highest support level, they mostly cannot access user data, just the metadata. We had a major problem with RDS once, and they specifically requested to load that snapshot to an internal instance to reproduce. It can happen in AWS, but not very common, in my experience.
Re: A Google Cloud support engineer solves a tough DNS case
#33The customer had set an extremely large buffer size and nobody thought to mention that? Perhaps the individual reporting the problem was different and was unaware of that unusual change.
I spend a decent amount of time investigating trouble reports, and in my experience it's quite uncommon to even get as much information as was provided in what google showed. It's also fairly uncommon to get any of these sorts of rare configurations in trouble reports, and usually takes some probing.
When I was a new engineer working in telco, one of the longest investigations I worked was when connectivity broke between one of our regional roaming partners and 1/3 of our nodes (I'm summarizing to try and keep the story brief). We called them and asked if they changed anything, reviewed the configuration and secrets used on the tunnels, etc. And were working with the vendor to go through any problems with the implementation. Saturday morning and probably 20 hours of investigation later, a new engineer at the regional partner see's there is a work order for changes to the connectivity to our nodes (we were adding some new ones) that was supposed to be executed that week. A typo in the change overwrote the secrets used by an existing tunnel instead of creating a new secret for the new peer. The person we were working with to investigate, was the person who implemented that change and told us several times nothing changed. He was also the one we worked with and read through all the secrets for typos or issues and didn't notice anything. Saturday morning he get's into the office, is shown the work order, and goes, oh yea, I did that at exactly the time the tunnel went down. Fix of typo'd secret later and everything comes right back up.
So just in my experience, I find it quite plausible that buffer size was not mentioned. And even besides this story, I know I've personally missed connecting causes with potential effects when investigating a problem, it's very easy to dismiss some setting, like the buffer size, as being connected specifically to DNS behaviours, especially if they are not noticed together or with a strong change management system that helps connect the timelines together.
Re: A Google Cloud support engineer solves a tough DNS case
#34This is a fun debugging story, but is a great example why servers should be cattle not pets. Having trouble with a VM? Blow it up and get a fresh one. Still having trouble? The provisioning steps are codified, you can walk through them and find the one that causes the issue.
> servers should be cattle not pets
I'm stealing this.
Re: A Google Cloud support engineer solves a tough DNS case
#35that’s a pretty good and detailed explanation. 2 things: 1) i hope they have a runbook for situations like this (ie the support engineer does not have to figure all this on the fly) 2) the customer should have provides more details and maybe should have thought of the tweaks they made (classic solution is to compare 2 instances - one works one does not)
1) you cannot have a runbook for everything, and even if you have a runbook you the best you could have found in this case is that something weird was happening in the VM. The setting had an insanely big value but it was accepted by the kernel, so you would assume it was a valid one.
2) the customer provided a huge amount of details, but it is usually very hard to explain what did you change from the base image. Most customers might not be willing to provide the full spec of their running system, as they might contain information they don't want to disclose. It is easier when the issue starts right after a change has been made, but this was not the case.
Re: A Google Cloud support engineer solves a tough DNS case
#36The customer had set an extremely large buffer size and nobody thought to mention that? Perhaps the individual reporting the problem was different and was unaware of that unusual change.
There are many things here that concern me from a system view.
1. They need guaranteed delivery, but chose to use UDP 2. They jacked up the default rmem buffer to ~2GB which is insane. Also, applies to all sockets not just UDP, so I wouldn't be surprised if they where also running into issues with memory pressure especially under load 3. Support didn't seem to let them know that's a pretty unconventional configuration
That was an interesting debugging story, and catching a bug like this is always good IMO. But, there is just so much WTF in this setup.
Re: A Google Cloud support engineer solves a tough DNS case
#37This is a fun debugging story, but is a great example why servers should be cattle not pets. Having trouble with a VM? Blow it up and get a fresh one. Still having trouble? The provisioning steps are codified, you can walk through them and find the one that causes the issue.
> servers should be cattle not pets I'm stealing this.
It's a well known phrase:
https://devops.stackexchange.com/questions/653/what-is-the-d...
Re: A Google Cloud support engineer solves a tough DNS case
#38I've been supporting AWS environments almost from the beginning but can't ever remember a case where I was asked or even considered offering Support a copy of a VM's storage volume. Is this common on Google/Azure/etc.?
It reminds me of my friend's hosting company that failed. They got a big customer and created a VM and the customer asked to fix a problem that involved getting a shell in the VM. Friend does it and the customer is gone next day. This aside, even though we had too many support cases so far with AWS, and having highest support level, they mostly cannot access user data, just the metadata. We had a major problem with R…
> It reminds me of my friend's hosting company that failed. They got a big customer and created a VM and the customer asked to fix a problem that involved getting a shell in the VM. Friend does it and the customer is gone next day.
The customer asked the support people to access a shell on their VM and they then quit because...? - or the support people accessed a shell without the customer’s express permission?
Re: A Google Cloud support engineer solves a tough DNS case
#39Earlier quoted context omitted.
You can't runbook these sort of issues... You can ensure you have the knowledge and tools to root cause the problem.
you most definitely can and you should. there are steps that you can do to gather the info and the linked example shows basic things to try. when you exhaust the run-book is when you start digging
I agree with the point around information gathering. Good technical teams have guides for endusers to collect relevant information. In this case the engineering team may have shared information with the TSCs to help narrow down the root cause.
This also highlights why support agents shouldn't be 100 engaged with customers. They need time to review and amend runbooks, consult with engineering teams, etc.
Re: A Google Cloud support engineer solves a tough DNS case
#40Earlier quoted context omitted.
> servers should be cattle not pets I'm stealing this.
It's a well known phrase: https://devops.stackexchange.com/questions/653/what-is-the-d...
Hey thanks! I've never heard it but given that I'm old and it (the phrase) was coined in 2011-2012 I'm not surprised.