Live data from Hacker News

A Google Cloud support engineer solves a tough DNS case

cloud.google.com

161–170 of 283 posts

Re: A Google Cloud support engineer solves a tough DNS case

#161
post #122

Earlier quoted context omitted.

Also, it was very nice of Google to follow up and submit the patch to LKML. IMHO this goes beyond the scope of their role. They could have taken a more selfish approach and accepted the bug as "normal" behavior, and advised their customer to not configure the buffer to such an enormous size.

Any decent engineer would smile at the fact that they just found a bug in this type of open source stack and happily submit it. Feel this is more of a side effect of individual behavior rather than company policy.

Unfortunately, not every engineer has an employer that would let them do this…

Re: A Google Cloud support engineer solves a tough DNS case

#162

Earlier quoted context omitted.

How heavy is the performance cost really? On x86, wouldn't a JO to a higher address take care of it? It would be a never-taken branch with perfect predictability.

http://www.cs.utah.edu/~regehr/papers/overflow12.pdf > For undefined behavior checking using precondition checks, slowdown relative to the baseline ranged from −0.5%–191%. In other words, from a tiny accidental speedup to a 3X increase in runtime.

Very interesting, thank you. I question the application of this result to kernel performance, though. Specint has hot arithmetic and UDP packet handling pretty much does not.

Re: A Google Cloud support engineer solves a tough DNS case

#163
post #154

He forgot mention that the kernel is Linux. It's almost like linux has become the standard OS. Linux is the new windows.

That being said, as you have noticed, Linux is the go-to OS for servers, and the post has a number of Linux-isms.

Re: A Google Cloud support engineer solves a tough DNS case

#165

Earlier quoted context omitted.

There needs to be some level of conformity between instances, they stop being a heard and more of a zoo if the skew is too large. The workloads running on the instances shouldn't be able to tell which instance type they are running on, or your workloads should be written such that it doesn't matter (but at some point it will). Things that grow and move together, wear together, so you will end up with a system that is…

My approach has always been that somewhere in my fleet there is a heat sink that fell off and a CPU running at 400MHz. My last two jobs have started with me sitting down at my desk on day 1 and demonstrating this fact. After concluding that the zoo is unavoidable, the only thing left to do is write the software accordingly.

> After concluding that the zoo is unavoidable, the only thing left to do is write the software accordingly.

What do you do to write software accordingly? Make it detect when it's running on a dud? Have it run as best as it can anyways?

Re: A Google Cloud support engineer solves a tough DNS case

#166

I'm not an expert with AWS or Google Cloud, so I'm interested in knowing: What "level" of customer or SLA do you have to be to get a certain quantity or guarantee of support and troubleshooting? Or is it that if even a free-tier customer points out something that is fundamentally a problem, it will receive attention by certain solutions engineers? Are there $ spending, 20 x (c3.4x.large), or I-pay-you-for-certain-upt…

At my previous employer, a large network services provider, everyone got the same depth of support, even people that were on the free tier! Granted, the people paying us $ENTERPRISE usually got responses in minutes, whereas free users might be waiting around for a week on more difficult cases, but it was the same set of engineers working on each.

Free customers did report bugs, and we would replicate, triage, and fix them as usual. These tended to be more obscure bugs (more severe ones would usually surface in the paid queues first), but we didn't discard them immediately.

Whether we effectively ignored bugs depended more on the ability of the customer to provide an actionable report. Some users provide exactly what we need up front, but there is a lot of "it doesn't work!" white noise from users that aren't able to or aren't willing to put in the work to accurately describe their issue and/or action feedback from us. There's usually not a whole lot support can do in that scenario if we don't see any obvious issues, but we'd go a bit further to placate paying customers--I fondly remember joining a call between some very technically inept user and their ISP who was adamant that either we or the ISP were at fault, after we guided their network team through taking local packet captures showing unanswered SYNs past their network border.

Re: A Google Cloud support engineer solves a tough DNS case

#167
post #117

Earlier quoted context omitted.

My fleet of machines I was the owner of at Facebook was around 10,000. I still remember the odd JVM crash that prompted me to reimage a machine. I wouldn't have remembered it except there were a few that month and it was the 3rd time I reimaged the same machine that I thought, "That's odd... I think I know that machine name." Checked history, saw the 3 repair jobs I had submitted... RAM was reset, CPU was eventually…

Also at FB: one day we got a huge spike in measured site-wide cpu usage. After the terror subsided, we found that a single request on a single machine had reported an improbably huge number of cycles (like, a billion years of cpu time). We figured a hardware problem and sent it to repair. A month later the same thing happened to the same machine; it had just been reimaged and sent back into the fleet. There was some…

Alternatively, you found a machine from the future :-)

Re: A Google Cloud support engineer solves a tough DNS case

#168

I'm not an expert with AWS or Google Cloud, so I'm interested in knowing: What "level" of customer or SLA do you have to be to get a certain quantity or guarantee of support and troubleshooting? Or is it that if even a free-tier customer points out something that is fundamentally a problem, it will receive attention by certain solutions engineers? Are there $ spending, 20 x (c3.4x.large), or I-pay-you-for-certain-upt…

This question brings to mind a recent experience I had with Scaleway support. I pay maybe 25 euro a month to host my k8s based application on their managed k8s offering.I did not pay the extra 2 euro a month for an upgraded support tier. I encountered an issue when deploying istio to the cluster, pinged the scaleway support chat on a saturday, and they had figured out the bug on their end and had a fix eta estimate w…

Support should be judged on how they perform under load, and how they perform consistently, not how they perform on random events.

We don't really have any way to know from your story whether Scaleway's support was under normal or extra load and delivered an excellent experience, or whether they had a bunch of bored support reps just waiting for something to work on because it was abnormally slow. The latter is nice, at the moment it happens, but doesn't really help you if 2 days later for a different issue you're left in a lurch for days on end because they're busy. The former would be good for maybe indicating that.

That's the whole point of service level guarantees. They provide a lower bound on the support you'll receive, which is often much more important and useful to track.

Re: A Google Cloud support engineer solves a tough DNS case

#169
post #61

Earlier quoted context omitted.

The cynical part of me expects that this case was handled so well because a) the support people found the issue fascinating and fun to work on, b) the post-mort on it would make an excellent blog post. On the flip side, it's encouraging that they have people somewhere in the support chain who are capable enough to read Linux kernel code and submit fixes upstream.

Author of the article here. I only thought of the possibility of making a blog post after the case was closed and I started telling my colleagues about it, and realized I would have loved to read about this. The case was indeed fun to work with, but the main reason why it had such a fast and happy resolution was because the customer was very responsive and very cooperative. I cannot talk for every Technical Solution…

I'm inspired by how much you seem to know about the details of computer network stuff. Is that a required knowledge to become a Google Tech Support person or you are just above average in terms of that among your peers?

Also, I wonder how you learn all these knowledge (that is, asking for recommendation on a few books/resources for learning) if you don't mind sharing. Thanks in advance!

Re: A Google Cloud support engineer solves a tough DNS case

#170
post #78
post #71

Earlier quoted context omitted.

Using unsigned does not generally fix overflow flaws. It just moves the threshold.

Sure. I was not suggesting it that it will eliminate overflows but would eliminate one source of them. Also mostly because there are probably few use cases that warrant signed values.

Using unsigned numbers doesn't really fix anything here, because the Linux kernel defines overflow of signed numbers. In both cases, you have generally surprising behavior when the number gets large enough: changing the type doesn't help; it just hides the issue in one of the cases.
Post reply on HN