Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Google Cloud support engineer solves a tough DNS case

cloud.google.com

Post reply on HN
Print view View original

131–140 of 283 posts

Re: A Google Cloud support engineer solves a tough DNS case

#131

Post by mcguire »

luhn wrote: post #22

This is a fun debugging story, but is a great example why servers should be cattle not pets. Having trouble with a VM? Blow it up and get a fresh one. Still having trouble? The provisioning steps are codified, you can walk through them and find the one that causes the issue.

Once upon a time I got pulled off a project in the middle; the project was given to a new, but experienced, developer to finish. I was talking to a friend who is one of our cloud guys the other day and he tells me "my" app is a bit of a problem child. It seems it crashes regularly but the problem is mitigated by the server restarting, so no one has any urge to fix it. (As a professional, I'm offended.)

Servers as cattle can cover a variety of sins.

Re: A Google Cloud support engineer solves a tough DNS case

#132

Post by ser_tyrion »

barbegal wrote: post #48
ser_tyrion wrote: post #24

> they use raw sockets! Raw sockets are different than normal sockets: they bypass iptables But this bugreport says raw sockets would be filtered by the OUTPUT chain of iptables: https://bugzilla.redhat.com/show_bug.cgi?id=1269914#c4 Is that accurate across distros? It does make sense for some socket types, like device sockets, to not be routed through iptables.

I think that bug report is misleading. Raw sockets do bypass iptables but they still go through ebtables. They hook in at the ebtables NAT OUTPUT chain. See the diagram here https://erlerobotics.gitbooks.io/erle-robotics-introduction-...

Thanks for that great link

Re: A Google Cloud support engineer solves a tough DNS case

#133

Post by mcguire »

lalos wrote: post #122
anonymousisme wrote: post #119

Earlier quoted context omitted.

Also, it was very nice of Google to follow up and submit the patch to LKML. IMHO this goes beyond the scope of their role. They could have taken a more selfish approach and accepted the bug as "normal" behavior, and advised their customer to not configure the buffer to such an enormous size.

Any decent engineer would smile at the fact that they just found a bug in this type of open source stack and happily submit it. Feel this is more of a side effect of individual behavior rather than company policy.

"Professional responsibility", if nothing else.

Re: A Google Cloud support engineer solves a tough DNS case

#134

Post by jeffbee »

sitkack wrote: post #115
jeffbee wrote: post #95

Earlier quoted context omitted.

It is indeed folly to assume that cattle have no identity. There's an internally famous video inside Google in which a particular well-known machine was unracked, dragged out into a field, and ceremonially smashed to pieces by some hardware techs. Sometimes a machine just takes an arrow to the knee and it's never the same again. Then there are all the uncontrolled or unrecorded differences between machines: the ones…

There needs to be some level of conformity between instances, they stop being a heard and more of a zoo if the skew is too large. The workloads running on the instances shouldn't be able to tell which instance type they are running on, or your workloads should be written such that it doesn't matter (but at some point it will). Things that grow and move together, wear together, so you will end up with a system that is…

My approach has always been that somewhere in my fleet there is a heat sink that fell off and a CPU running at 400MHz. My last two jobs have started with me sitting down at my desk on day 1 and demonstrating this fact. After concluding that the zoo is unavoidable, the only thing left to do is write the software accordingly.

Re: A Google Cloud support engineer solves a tough DNS case

#135

Post by bpodgursky »

dilyevsky wrote: post #102
bpodgursky wrote: post #89

Earlier quoted context omitted.

If you're spending that much money (this isn't GCP specific -- this is any cloud) you should be establishing a 1-3 year min-commit contract, and in practice, this will get negotiated through the CFO. This will get you massive discounts -- 20-30% under list price, in exchange for spending $X million/year over Y years. It will also get you a dedicated sales rep and sales team, and they will absolutely crack the whip on…

Yeah not talking about discount. The discount was nice (or so i heard, but if you do 3y commit you can get that anyway). > and they will absolutely crack the whip on internal teams to get issues resolved Not in my experience. Although we did get an ever-rotating rep. I think they changed three of them in like a year or so

Contract-level commits will generally work on top of Committed Use Discounts, not instead of (but obviously this comes down to your own SKU by SKU negotiating).

Re: A Google Cloud support engineer solves a tough DNS case

#136

Post by mcguire »

FreeFull wrote: post #130
mcguire wrote: post #129

" When sk_rcvbuf gets close to 2^31, adding the size of the packet can cause an integer overflow. And since it’s an int it becomes a negative number, therefore the condition is true when it should be false (for more, also check out this discussion of signed magnitude representation). " And this is why you don't generally use signed numbers in systems code, unless you specifically need negative numbers. And why you gr…

I'm not sure how using an unsigned number would help, given that when it overflows you're still going to have some code do unexpected stuff anyway.

For one thing, it's a clue you need to step back and think, "What happens when this overflows?" rather than "Oh, it's just a number."

For another, that's why you get paranoid.

(For a third, I strongly recommend something like Frama-C with the Weakest-Precondition module---it's very good at finding issues like these.)

Re: A Google Cloud support engineer solves a tough DNS case

#138

Post by calvinmorrison »

lalos wrote: post #122
anonymousisme wrote: post #119

Earlier quoted context omitted.

Also, it was very nice of Google to follow up and submit the patch to LKML. IMHO this goes beyond the scope of their role. They could have taken a more selfish approach and accepted the bug as "normal" behavior, and advised their customer to not configure the buffer to such an enormous size.

Any decent engineer would smile at the fact that they just found a bug in this type of open source stack and happily submit it. Feel this is more of a side effect of individual behavior rather than company policy.

until you hit something like "bug reports should be subitted to here, not there and filled out with appropriate information, matching triplicate documents and make sure to CC the grand puba, also CLOSED-WONTFIX" enough times and you just stop submitting requests.

Re: A Google Cloud support engineer solves a tough DNS case

#139

Post by sleepydog »

mcguire wrote: post #136
FreeFull wrote: post #130

Earlier quoted context omitted.

I'm not sure how using an unsigned number would help, given that when it overflows you're still going to have some code do unexpected stuff anyway.

For one thing, it's a clue you need to step back and think, "What happens when this overflows?" rather than "Oh, it's just a number." For another, that's why you get paranoid. (For a third, I strongly recommend something like Frama-C with the Weakest-Precondition module---it's very good at finding issues like these.)

I'm not convinced that it would have been any more obvious to the person who made the error that the variable could overflow if it were unsigned.

It's also much easier, IMO, to accidentally underflow an unsigned integer; it's so much more common to work with 0 than it is to work with +/- 2 billion.

Re: A Google Cloud support engineer solves a tough DNS case

#140

Post by jeffbee »

cesarb wrote: post #109
steveklabnik wrote: post #77

Earlier quoted context omitted.

I think this person is trolling. However, for the purposes of discussion, two things here: So, in Rust, overflow panics in debug builds, but does wrap around in release builds. So, it is possible this bug would have been caught in testing, but if it wasn't, it still would have slipped into production. However, that being said, Rust does not do implicit casting between numeric types. So it's very likely that this code…

AFAIK, Rust can panic on overflow even in release builds if you want it to, at a somewhat heavy performance cost (which is why this is not enabled by default in release builds). In this case, it would convert the issue from "some packets are unexpectedly being discarded" into an immediate crash within the kernel.

How heavy is the performance cost really? On x86, wouldn't a JO to a higher address take care of it? It would be a never-taken branch with perfect predictability.
Post reply on HN