why can't you reach Google just over the phone. Support is very bad
A Google Cloud support engineer solves a tough DNS case
171–180 of 283 posts
Re: A Google Cloud support engineer solves a tough DNS case
#172Earlier quoted context omitted.
AFAIK, Rust can panic on overflow even in release builds if you want it to, at a somewhat heavy performance cost (which is why this is not enabled by default in release builds). In this case, it would convert the issue from "some packets are unexpectedly being discarded" into an immediate crash within the kernel.
How heavy is the performance cost really? On x86, wouldn't a JO to a higher address take care of it? It would be a never-taken branch with perfect predictability.
In this case, it'd probably be fine. In general I can see it interfering with loop vectorization and bloating code size, so it might not be low-cost in general.
Re: A Google Cloud support engineer solves a tough DNS case
#173The customer had set an extremely large buffer size and nobody thought to mention that? Perhaps the individual reporting the problem was different and was unaware of that unusual change.
I used to do work kind of like this stuff on enterprise storage arrays that ran a modified BSD. We didn't lock the system down much, so customers could go and set whatever system guts stuff they wanted. We had a tool on-system that would basically tar up all the system configs and phone home with them when the customer hit a button. You'd better believe that one of my first steps investigating anything was diffing the crap out of any relevant configs against a clean base version.
As another commenter mentioned, this was the result of customers never actually mentioning their weird sysctl tuning in the original issue description. It's not like they're trying to screw you over or anything - there's just an awful lot of config options in an entire system that does anything interesting, and in the case of a big enterprise appliance, it's likely that dozens of people have had admin on it at one point or another.
Re: A Google Cloud support engineer solves a tough DNS case
#174Is Google Cloud support this good in general, or only for certain tiers or plans?
Whenever the engineering support steps in, the problem gets solved fast or the issue gets acknowledged. However, over the last 15 months, ~50% of times our GCP support cases end up in frustration with no resolution. We have more success finding issues in the corresponding GitHub repo and opening an issue there.
We have the expensive off-the-shelve support option (I think 450/seat/month) for 1h response.
In most cases, we spend more time back and forth with support than it would take you to figure it out. I'm talking about issues that span over weeks with tens of hours spent. We end up reiterating the original support case problem (i.e. the support engineer doesn't bother reading the actual problem) whenever the engineer changes.
We've had: P1s where the support engineer told us we'll get an update the following day, only to figure out that there was a breaking release on their side that exactly matched our description. While investigating a load balancer issue, the support engineer looked at the LB logs, saw a ton of logs coming from penetration scans (e.g. GET /phpmyadmin), and suggested that the solution was to open up those addresses.
Re: A Google Cloud support engineer solves a tough DNS case
#175Earlier quoted context omitted.
My approach has always been that somewhere in my fleet there is a heat sink that fell off and a CPU running at 400MHz. My last two jobs have started with me sitting down at my desk on day 1 and demonstrating this fact. After concluding that the zoo is unavoidable, the only thing left to do is write the software accordingly.
> After concluding that the zoo is unavoidable, the only thing left to do is write the software accordingly. What do you do to write software accordingly? Make it detect when it's running on a dud? Have it run as best as it can anyways?
Suicide is a good solution, if some higher-level thing will notice and move the task to another machine. Batch frameworks can kill slow shards, or re-assign their work to faster shards. Clients of online services can direct more traffic to working shards and less or none to slow or broken ones.
Re: A Google Cloud support engineer solves a tough DNS case
#176Earlier quoted context omitted.
My fleet of machines I was the owner of at Facebook was around 10,000. I still remember the odd JVM crash that prompted me to reimage a machine. I wouldn't have remembered it except there were a few that month and it was the 3rd time I reimaged the same machine that I thought, "That's odd... I think I know that machine name." Checked history, saw the 3 repair jobs I had submitted... RAM was reset, CPU was eventually…
It is indeed folly to assume that cattle have no identity. There's an internally famous video inside Google in which a particular well-known machine was unracked, dragged out into a field, and ceremonially smashed to pieces by some hardware techs. Sometimes a machine just takes an arrow to the knee and it's never the same again. Then there are all the uncontrolled or unrecorded differences between machines: the ones…
> There's an internally famous video inside Google in which a particular well-known machine was unracked, dragged out into a field
I really hope the video is edited to strongly resemble this scene: https://www.youtube.com/watch?v=N9wsjroVlu8
Re: A Google Cloud support engineer solves a tough DNS case
#177Earlier quoted context omitted.
i’m not saying a runbook will catch everything. but it will give you a chance to solve the problem quicker.
I totally agree, but keep in mind that it is an iterative process: you have a case, you apply the runbook/playbook you have, if they are not enough you use your skill/knowledge to solve the case, then you update the playbooks.
iteration is my middle name :) the point I was trying to make is you should be prepared - if only for the 95% of the cases that the runbook can solve. you also don't start with a complete runbook - you build it as you operate the service.
Re: A Google Cloud support engineer solves a tough DNS case
#178Earlier quoted context omitted.
It is indeed folly to assume that cattle have no identity. There's an internally famous video inside Google in which a particular well-known machine was unracked, dragged out into a field, and ceremonially smashed to pieces by some hardware techs. Sometimes a machine just takes an arrow to the knee and it's never the same again. Then there are all the uncontrolled or unrecorded differences between machines: the ones…
> There's an internally famous video inside Google in which a particular well-known machine was unracked, dragged out into a field I really hope the video is edited to strongly resemble this scene: https://www.youtube.com/watch?v=N9wsjroVlu8
It was made to reference exactly that, down to the attire.
Re: A Google Cloud support engineer solves a tough DNS case
#179Earlier quoted context omitted.
What is a reason why that file would correspond to the actual sysctls in effect?
If you use an automated configuration management system such as Puppet, you don't ever run sysctl manually in a shell. Instead, everything is controlled by the configuration management system. sysctl is a bit problematic in terms of exhaustiveness. That is, how do you ensure that the kernel only has its original values plus whatever you put in sysctl.conf, and nobody actually ran sysctl manually at some point? But it…
I have seen so much random behavior from puppet runs. It's basically a big fancy wrapper around a bunch of shell commands (much better than the raw shell commands) but subject to all the bizarre race conditions and so on. We had to wait 30 minutes to use a newly created VM so that puppet had run three times, and it was >0.99 likely to be good now. (If it wasn't, it was killed and we retried; 30 minutes was chosen to minimize the expected time; the puppet config was migrated from cfengine and was based on a lot of host-name based regular expressions and very dangerous to debug/refactor).
Re: A Google Cloud support engineer solves a tough DNS case
#180Earlier quoted context omitted.
It is indeed folly to assume that cattle have no identity. There's an internally famous video inside Google in which a particular well-known machine was unracked, dragged out into a field, and ceremonially smashed to pieces by some hardware techs. Sometimes a machine just takes an arrow to the knee and it's never the same again. Then there are all the uncontrolled or unrecorded differences between machines: the ones…
Once, the head of IT of a company I used to work for was touring the datacenter, passing some new racks filled with blade servers. He stopped, said "why are all the fans running full blast on this rack?" and the admins checked and they were running some test workload at scale somebody had forgotten about a few weeks before. Everybody was embarassed because no monitoring caught it, but the VP of IT did by walking past…
I walk the server room daily, every morning. I've tended our monitoring system for 15 years now and I don't trust myself to be infallible. I'm also the MD ...