Live data from Hacker News

A Google Cloud support engineer solves a tough DNS case

cloud.google.com

101–110 of 283 posts

Re: A Google Cloud support engineer solves a tough DNS case

#101
post #56
post #28

Earlier quoted context omitted.

Do you have a support plan?

Why should you need a support plan for a product you're paying for? "Ok, you can pay us $X/mo for the service, but if something goes wrong, we won't help you unless you also pay an additional $Y/mo." It's absolute garbage that this is where the industry is.

Yes, my 100$ monthly spend deserves the same support as a company with a $250,000 monthly spend.

Re: A Google Cloud support engineer solves a tough DNS case

#102

Earlier quoted context omitted.

Wait what does it have to do with the finance team? We did get their “platinum” tier or whatever.

If you're spending that much money (this isn't GCP specific -- this is any cloud) you should be establishing a 1-3 year min-commit contract, and in practice, this will get negotiated through the CFO. This will get you massive discounts -- 20-30% under list price, in exchange for spending $X million/year over Y years. It will also get you a dedicated sales rep and sales team, and they will absolutely crack the whip on…

Yeah not talking about discount. The discount was nice (or so i heard, but if you do 3y commit you can get that anyway).

> and they will absolutely crack the whip on internal teams to get issues resolved

Not in my experience. Although we did get an ever-rotating rep. I think they changed three of them in like a year or so

Re: A Google Cloud support engineer solves a tough DNS case

#103
post #55

That brings up another point. Should the kernel standardize on unsigned scalars completely? How many legitimate use cases are there to use signed scalars in the kernel?

The Linux kernel internally uses the common C convention in which negative numbers are errors (-ENOMEM and so on) while positive numbers (and zero) are successful values. (Some parts of the kernel use a related convention in which values within the "last page" of the address space, that is, -4096 to -1 inclusive, are errors, while other values are valid memory addresses; there are macros to convert between both conventions.)

Re: A Google Cloud support engineer solves a tough DNS case

#104
post #96
post #22

This is a fun debugging story, but is a great example why servers should be cattle not pets. Having trouble with a VM? Blow it up and get a fresh one. Still having trouble? The provisioning steps are codified, you can walk through them and find the one that causes the issue.

Did you read the article? The customer had set `net.core.rmem_default = 2147483647` on purpose. Which exposed a Kernel bug. The whole herd would be having the same issue.

I really want to know if the telemetrics (or whatever the thing was) pushed enough packets to actually warrant the config. Setting something to the max sounds like preopt to me. This must be a truly exceptional condition if it actually remained undiscovered since linux 3.

Re: A Google Cloud support engineer solves a tough DNS case

#105

Earlier quoted context omitted.

plenty of issues are not deterministic, even with 100% of everything managed by configuration management software.

But /etc/sysctl.conf is deterministic.

Not all issues stem from sysctl.conf

Re: A Google Cloud support engineer solves a tough DNS case

#107
post #92

Earlier quoted context omitted.

What is a reason why that file would correspond to the actual sysctls in effect?

If you use an automated configuration management system such as Puppet, you don't ever run sysctl manually in a shell. Instead, everything is controlled by the configuration management system. sysctl is a bit problematic in terms of exhaustiveness. That is, how do you ensure that the kernel only has its original values plus whatever you put in sysctl.conf, and nobody actually ran sysctl manually at some point? But it…

OK, so a reason why this file might reflect reality is that some automatic system wrote the file and subsequently successfully ran sysctl -p, but there are dozens of reasons why the file and reality would differ. The only source of truth is sysctl(8) or reading files in /proc/sys, and these are the values that need to propagate to observability systems and decision-making.

Re: A Google Cloud support engineer solves a tough DNS case

#108
Note: You shouldn't use int, unsigned int, char, short, long. Use int16_t, uint16_t, uint8_t, etc (or their _fast equivalents) from stdint.h.

The former's sizes change based on platform, cpu, and compiler; the latter are fixed-width (or flexible, where _fast may use a larger size if it's faster).

I started brushing up on my C recently and have been collecting these little nuggets: https://gist.github.com/peterwwillis/53cd9d34d8755784e483790...

Re: A Google Cloud support engineer solves a tough DNS case

#109
post #62

Earlier quoted context omitted.

Can you expand on your answer. How does Rust resolve this issue? Does it solve by disallowing type casts thereby preventing overflow when the number becomes negative?

I think this person is trolling. However, for the purposes of discussion, two things here: So, in Rust, overflow panics in debug builds, but does wrap around in release builds. So, it is possible this bug would have been caught in testing, but if it wasn't, it still would have slipped into production. However, that being said, Rust does not do implicit casting between numeric types. So it's very likely that this code…

AFAIK, Rust can panic on overflow even in release builds if you want it to, at a somewhat heavy performance cost (which is why this is not enabled by default in release builds). In this case, it would convert the issue from "some packets are unexpectedly being discarded" into an immediate crash within the kernel.

Re: A Google Cloud support engineer solves a tough DNS case

#110
post #22

This is a fun debugging story, but is a great example why servers should be cattle not pets. Having trouble with a VM? Blow it up and get a fresh one. Still having trouble? The provisioning steps are codified, you can walk through them and find the one that causes the issue.

We really need a new analogy - I assure you that ranchers care when they lose a cow.
Post reply on HN