Live data from Hacker News

A Google Cloud support engineer solves a tough DNS case

cloud.google.com

151–160 of 283 posts

Re: A Google Cloud support engineer solves a tough DNS case

#151
post #109

Earlier quoted context omitted.

AFAIK, Rust can panic on overflow even in release builds if you want it to, at a somewhat heavy performance cost (which is why this is not enabled by default in release builds). In this case, it would convert the issue from "some packets are unexpectedly being discarded" into an immediate crash within the kernel.

How heavy is the performance cost really? On x86, wouldn't a JO to a higher address take care of it? It would be a never-taken branch with perfect predictability.

http://www.cs.utah.edu/~regehr/papers/overflow12.pdf

> For undefined behavior checking using precondition checks, slowdown relative to the baseline ranged from −0.5%–191%. In other words, from a tiny accidental speedup to a 3X increase in runtime.

Re: A Google Cloud support engineer solves a tough DNS case

#152
post #95

Earlier quoted context omitted.

My fleet of machines I was the owner of at Facebook was around 10,000. I still remember the odd JVM crash that prompted me to reimage a machine. I wouldn't have remembered it except there were a few that month and it was the 3rd time I reimaged the same machine that I thought, "That's odd... I think I know that machine name." Checked history, saw the 3 repair jobs I had submitted... RAM was reset, CPU was eventually…

It is indeed folly to assume that cattle have no identity. There's an internally famous video inside Google in which a particular well-known machine was unracked, dragged out into a field, and ceremonially smashed to pieces by some hardware techs. Sometimes a machine just takes an arrow to the knee and it's never the same again. Then there are all the uncontrolled or unrecorded differences between machines: the ones…

> It is indeed folly to assume that cattle have no identity.

My dad had a book over almost all the cow names in Norway[1], per 1988. As a kid I found it rather fun to just flip through it and read some names, often wondering how they came up with them.

However since then it seems the tradition of naming cattle has dropped[2] to less than 30%.

[1]: "Gullhorn og dei andre : kunamn i Noreg" https://urn.nb.no/URN:NBN:no-nb_digibok_2010111708049

[2]: https://www.nrk.no/nordland/kyrne-far-ikke-lenger-navn-1.835...

Re: A Google Cloud support engineer solves a tough DNS case

#153

Earlier quoted context omitted.

How heavy is the performance cost really? On x86, wouldn't a JO to a higher address take care of it? It would be a never-taken branch with perfect predictability.

At what cost to code size?

That would be the thing: two bytes in the hot path. But it wouldn't need to be universally applied to every place the + operator appears. Couldn't it just be wrapped around sensitive computations, or generated for the x > y + z idiom? It would be cheaper or at least as cheap as the machine code generated for the manual overflow check (if int_max - z > y ...)

Re: A Google Cloud support engineer solves a tough DNS case

#156
post #31

if (rmem > (size + sk->sk_rcvbuf)) goto uncharge_drop; What is rmem in this case? I'm a bit confused as to why it is written that way. This drops the packet right when it overflows the buffer?

Basically rmem - size was the number of bytes that were consumed before this current packet. In the line before, we have thread-atomically added size to rmem immediately prior and read the current value in one single step. Call this "staking our claim" to part of the buffer, and the meaning of "uncharge" here is discharging this claim by atomically decrementing the counter, before dropping the packet.

Probably this bug would not have happened if this comparison were written as `rmem - size > sk->sk_rcvbuf`?

So it is saying that the simplest sanity check is "if the buffer was already full before we staked our claim we should drop this packet immediately." As the "goto" indicates, there are then a bunch more checks on other circumstances where we should also drop the packet. Due to the quirks of multi-threading it is of course possible that some packets get unnecessarily dropped between when we stake the claim and when we discharge it, which the code just accepts -- the thinking is presumably "yeah if the buffer is full a lot of packets are gonna get dropped and that's just life -- it's much less important that we dropped some extra packets when we were already dropping packets, and much more important that we don't mismanage the buffer's memory when it's nearly full."

A comment suggests that part of the reason for this awkward phrasing is that it is possible for rmem = size, in other words the buffer was empty when we staked our claim--and in this case we don't want to drop this packet even if it would overflow a small max-buffer-size. I think the idea there is "we already have the socket buffer allocated, obviously this thing fits in memory, so let's just handle it if the queue is empty rather than dropping every single packet that is larger than the queue size."

Re: A Google Cloud support engineer solves a tough DNS case

#157
post #56
post #28

Earlier quoted context omitted.

Do you have a support plan?

Why should you need a support plan for a product you're paying for? "Ok, you can pay us $X/mo for the service, but if something goes wrong, we won't help you unless you also pay an additional $Y/mo." It's absolute garbage that this is where the industry is.

Because the product that you're paying for is an extremely efficient automated cloud infrastructure with little manual support, unless you're willing to pay more for it. If you want hand holding there are other less automated infrastructure providers that cost a lot more.

Re: A Google Cloud support engineer solves a tough DNS case

#158
post #95

Earlier quoted context omitted.

My fleet of machines I was the owner of at Facebook was around 10,000. I still remember the odd JVM crash that prompted me to reimage a machine. I wouldn't have remembered it except there were a few that month and it was the 3rd time I reimaged the same machine that I thought, "That's odd... I think I know that machine name." Checked history, saw the 3 repair jobs I had submitted... RAM was reset, CPU was eventually…

It is indeed folly to assume that cattle have no identity. There's an internally famous video inside Google in which a particular well-known machine was unracked, dragged out into a field, and ceremonially smashed to pieces by some hardware techs. Sometimes a machine just takes an arrow to the knee and it's never the same again. Then there are all the uncontrolled or unrecorded differences between machines: the ones…

> particular well-known machine was unracked

are you able to comment a bit further on why this machine was well known?

Re: A Google Cloud support engineer solves a tough DNS case

#160
post #95

Earlier quoted context omitted.

It is indeed folly to assume that cattle have no identity. There's an internally famous video inside Google in which a particular well-known machine was unracked, dragged out into a field, and ceremonially smashed to pieces by some hardware techs. Sometimes a machine just takes an arrow to the knee and it's never the same again. Then there are all the uncontrolled or unrecorded differences between machines: the ones…

> particular well-known machine was unracked are you able to comment a bit further on why this machine was well known?

Because of the way engineers habitually run batch jobs with more replicas than there are machines, this one broken computer had crapped up every map/reduce job in that facility for a long time, and it had been sent to repairs many times without benefit. Many people knew instinctively that if their job was stuck it was probably because of the shard on xyz42 (or whatever the node name was).
Post reply on HN