Earlier quoted context omitted.
Also, it was very nice of Google to follow up and submit the patch to LKML. IMHO this goes beyond the scope of their role. They could have taken a more selfish approach and accepted the bug as "normal" behavior, and advised their customer to not configure the buffer to such an enormous size.
Any decent engineer would smile at the fact that they just found a bug in this type of open source stack and happily submit it. Feel this is more of a side effect of individual behavior rather than company policy.
A Google Cloud support engineer solves a tough DNS case
161–170 of 283 posts
Re: A Google Cloud support engineer solves a tough DNS case
#162Earlier quoted context omitted.
How heavy is the performance cost really? On x86, wouldn't a JO to a higher address take care of it? It would be a never-taken branch with perfect predictability.
http://www.cs.utah.edu/~regehr/papers/overflow12.pdf > For undefined behavior checking using precondition checks, slowdown relative to the baseline ranged from −0.5%–191%. In other words, from a tiny accidental speedup to a 3X increase in runtime.
Re: A Google Cloud support engineer solves a tough DNS case
#163He forgot mention that the kernel is Linux. It's almost like linux has become the standard OS. Linux is the new windows.
Re: A Google Cloud support engineer solves a tough DNS case
#164Oh look another C overflow bug.
Re: A Google Cloud support engineer solves a tough DNS case
#165Earlier quoted context omitted.
There needs to be some level of conformity between instances, they stop being a heard and more of a zoo if the skew is too large. The workloads running on the instances shouldn't be able to tell which instance type they are running on, or your workloads should be written such that it doesn't matter (but at some point it will). Things that grow and move together, wear together, so you will end up with a system that is…
My approach has always been that somewhere in my fleet there is a heat sink that fell off and a CPU running at 400MHz. My last two jobs have started with me sitting down at my desk on day 1 and demonstrating this fact. After concluding that the zoo is unavoidable, the only thing left to do is write the software accordingly.
What do you do to write software accordingly? Make it detect when it's running on a dud? Have it run as best as it can anyways?
Re: A Google Cloud support engineer solves a tough DNS case
#166I'm not an expert with AWS or Google Cloud, so I'm interested in knowing: What "level" of customer or SLA do you have to be to get a certain quantity or guarantee of support and troubleshooting? Or is it that if even a free-tier customer points out something that is fundamentally a problem, it will receive attention by certain solutions engineers? Are there $ spending, 20 x (c3.4x.large), or I-pay-you-for-certain-upt…
Free customers did report bugs, and we would replicate, triage, and fix them as usual. These tended to be more obscure bugs (more severe ones would usually surface in the paid queues first), but we didn't discard them immediately.
Whether we effectively ignored bugs depended more on the ability of the customer to provide an actionable report. Some users provide exactly what we need up front, but there is a lot of "it doesn't work!" white noise from users that aren't able to or aren't willing to put in the work to accurately describe their issue and/or action feedback from us. There's usually not a whole lot support can do in that scenario if we don't see any obvious issues, but we'd go a bit further to placate paying customers--I fondly remember joining a call between some very technically inept user and their ISP who was adamant that either we or the ISP were at fault, after we guided their network team through taking local packet captures showing unanswered SYNs past their network border.
Re: A Google Cloud support engineer solves a tough DNS case
#167Earlier quoted context omitted.
My fleet of machines I was the owner of at Facebook was around 10,000. I still remember the odd JVM crash that prompted me to reimage a machine. I wouldn't have remembered it except there were a few that month and it was the 3rd time I reimaged the same machine that I thought, "That's odd... I think I know that machine name." Checked history, saw the 3 repair jobs I had submitted... RAM was reset, CPU was eventually…
Also at FB: one day we got a huge spike in measured site-wide cpu usage. After the terror subsided, we found that a single request on a single machine had reported an improbably huge number of cycles (like, a billion years of cpu time). We figured a hardware problem and sent it to repair. A month later the same thing happened to the same machine; it had just been reimaged and sent back into the fleet. There was some…
Re: A Google Cloud support engineer solves a tough DNS case
#168I'm not an expert with AWS or Google Cloud, so I'm interested in knowing: What "level" of customer or SLA do you have to be to get a certain quantity or guarantee of support and troubleshooting? Or is it that if even a free-tier customer points out something that is fundamentally a problem, it will receive attention by certain solutions engineers? Are there $ spending, 20 x (c3.4x.large), or I-pay-you-for-certain-upt…
This question brings to mind a recent experience I had with Scaleway support. I pay maybe 25 euro a month to host my k8s based application on their managed k8s offering.I did not pay the extra 2 euro a month for an upgraded support tier. I encountered an issue when deploying istio to the cluster, pinged the scaleway support chat on a saturday, and they had figured out the bug on their end and had a fix eta estimate w…
We don't really have any way to know from your story whether Scaleway's support was under normal or extra load and delivered an excellent experience, or whether they had a bunch of bored support reps just waiting for something to work on because it was abnormally slow. The latter is nice, at the moment it happens, but doesn't really help you if 2 days later for a different issue you're left in a lurch for days on end because they're busy. The former would be good for maybe indicating that.
That's the whole point of service level guarantees. They provide a lower bound on the support you'll receive, which is often much more important and useful to track.
Re: A Google Cloud support engineer solves a tough DNS case
#169Earlier quoted context omitted.
The cynical part of me expects that this case was handled so well because a) the support people found the issue fascinating and fun to work on, b) the post-mort on it would make an excellent blog post. On the flip side, it's encouraging that they have people somewhere in the support chain who are capable enough to read Linux kernel code and submit fixes upstream.
Author of the article here. I only thought of the possibility of making a blog post after the case was closed and I started telling my colleagues about it, and realized I would have loved to read about this. The case was indeed fun to work with, but the main reason why it had such a fast and happy resolution was because the customer was very responsive and very cooperative. I cannot talk for every Technical Solution…
Also, I wonder how you learn all these knowledge (that is, asking for recommendation on a few books/resources for learning) if you don't mind sharing. Thanks in advance!
Re: A Google Cloud support engineer solves a tough DNS case
#170Earlier quoted context omitted.
Using unsigned does not generally fix overflow flaws. It just moves the threshold.
Sure. I was not suggesting it that it will eliminate overflows but would eliminate one source of them. Also mostly because there are probably few use cases that warrant signed values.