A Google Cloud support engineer solves a tough DNS case • Hacker News Board

Re: A Google Cloud support engineer solves a tough DNS case

#81

Post by londons_explore » Tue, May 19, 2020, 5:43 PM UTC

supernova87a wrote: post #57

I'm not an expert with AWS or Google Cloud, so I'm interested in knowing: What "level" of customer or SLA do you have to be to get a certain quantity or guarantee of support and troubleshooting? Or is it that if even a free-tier customer points out something that is fundamentally a problem, it will receive attention by certain solutions engineers? Are there $ spending, 20 x (c3.4x.large), or I-pay-you-for-certain-upt…

When purchasing support, you should consider you are really buying an expert who really knows google cloud, but doesn't have special buttons to click to do things you couldn't do.

If a service you depend on is down, your support agent will be able to tell you it's down, but not speed up the fix.

Cloud support will have more information about performance black holes and limitations that the documents don't describe. They also will be able to advise on "is this design or that design likely better". They generally know how the backends of GCP services work, and their common failure modes, which is pretty hard knowledge to get from the outside.

Re: A Google Cloud support engineer solves a tough DNS case

#82

Post by teej » Tue, May 19, 2020, 5:44 PM UTC

kelnos wrote: post #56

Axsuul wrote: post #28

Earlier quoted context omitted.

Do you have a support plan?

Why should you need a support plan for a product you're paying for? "Ok, you can pay us $X/mo for the service, but if something goes wrong, we won't help you unless you also pay an additional $Y/mo." It's absolute garbage that this is where the industry is.

This is consumer thinking, not business thinking. Every high ticket item on the planet comes with the option of a support contract.

Re: A Google Cloud support engineer solves a tough DNS case

#83

Post by dilyevsky » Tue, May 19, 2020, 5:44 PM UTC

bpodgursky wrote: post #72

dilyevsky wrote: post #64

Earlier quoted context omitted.

How much is “substantial” for gcp to get decent support? Tens of millions/y certainly wasn’t cutting it...

If you weren't getting substantial support at tens of millions/y, your finance team did something deeply wrong when negotiating the contract.

Wait what does it have to do with the finance team? We did get their “platinum” tier or whatever.

Re: A Google Cloud support engineer solves a tough DNS case

#84

Post by amessina1 » Tue, May 19, 2020, 5:47 PM UTC

kelnos wrote: post #61

xvector wrote: post #13

Is Google Cloud support this good in general, or only for certain tiers or plans?

The cynical part of me expects that this case was handled so well because a) the support people found the issue fascinating and fun to work on, b) the post-mort on it would make an excellent blog post. On the flip side, it's encouraging that they have people somewhere in the support chain who are capable enough to read Linux kernel code and submit fixes upstream.

Author of the article here. I only thought of the possibility of making a blog post after the case was closed and I started telling my colleagues about it, and realized I would have loved to read about this.

The case was indeed fun to work with, but the main reason why it had such a fast and happy resolution was because the customer was very responsive and very cooperative.

I cannot talk for every Technical Solution Engineer, but I can tell you that I have no particular interest in simply closing a ticket: I want to go down the rabbit hole and solve technical issues, and I know many of my colleagues feel the same.

I am also far from being the most senior or skilled TSE in Google Cloud Support, I just wrote an article about one of most interesting cases I had.

Re: A Google Cloud support engineer solves a tough DNS case

#85

Post by gonehome » Tue, May 19, 2020, 5:48 PM UTC

josefresco wrote: post #34

luhn wrote: post #22

This is a fun debugging story, but is a great example why servers should be cattle not pets. Having trouble with a VM? Blow it up and get a fresh one. Still having trouble? The provisioning steps are codified, you can walk through them and find the one that causes the issue.

> servers should be cattle not pets I'm stealing this.

If you're interested there are a couple of great books that dig more into this kind of thing.

The Phoenix Project: https://www.amazon.com/Phoenix-Project-DevOps-Helping-Busine...

Its related Dev Ops Handbook: https://www.amazon.com/DevOps-Handbook-World-Class-Reliabili...

Re: A Google Cloud support engineer solves a tough DNS case

#86

Post by joshuamorton » Tue, May 19, 2020, 5:48 PM UTC

kelnos wrote: post #56

Axsuul wrote: post #28

Earlier quoted context omitted.

Do you have a support plan?

Why should you need a support plan for a product you're paying for? "Ok, you can pay us $X/mo for the service, but if something goes wrong, we won't help you unless you also pay an additional $Y/mo." It's absolute garbage that this is where the industry is.

Because some businesses want support, and others don't. Forcing the people who don't want support to pay for support seems rude, does it not?

Re: A Google Cloud support engineer solves a tough DNS case

#87

Post by fenesiistvan » Tue, May 19, 2020, 5:50 PM UTC

rantwasp wrote: post #4

that’s a pretty good and detailed explanation. 2 things: 1) i hope they have a runbook for situations like this (ie the support engineer does not have to figure all this on the fly) 2) the customer should have provides more details and maybe should have thought of the tweaks they made (classic solution is to compare 2 instances - one works one does not)

I don't think that this was a "support engineer". He was a god mode developer!

Re: A Google Cloud support engineer solves a tough DNS case

#88

Post by banana_giraffe » Tue, May 19, 2020, 5:52 PM UTC

econcon wrote: post #17

Google support is basically non existent for customers even spending like 5-10K a month on their platform. Even questions you raise on their Reddit sub go unnoticed. They don't have bunch of people actively trying to solve their customers problem. That said the only time my problems were actually listed to were from BigQuery team. Other than this, I don't think it's possible to get any explanation on a feature from a…

The difference of the support plan is amazing. I had a long standing issue where I couldn't get any help, at best I get stuffed to a forum where some random other user would invariably tell me I'm doing things the wrong way and standard non-answers.

We got put on their best support plan for a few months for reasons. The difference was insane. I gave them permission/creds to log into the problem box, along with steps to repo things. In a few days they had reverse engineered what was going on in our code, without having our code, figured out where the bottleneck was in the kernel, and gave me detailed steps to build a tweaked kernel that wouldn't be a problem.

Re: A Google Cloud support engineer solves a tough DNS case

#89

Post by bpodgursky » Tue, May 19, 2020, 5:52 PM UTC

dilyevsky wrote: post #83

bpodgursky wrote: post #72

Earlier quoted context omitted.

If you weren't getting substantial support at tens of millions/y, your finance team did something deeply wrong when negotiating the contract.

Wait what does it have to do with the finance team? We did get their “platinum” tier or whatever.

If you're spending that much money (this isn't GCP specific -- this is any cloud) you should be establishing a 1-3 year min-commit contract, and in practice, this will get negotiated through the CFO. This will get you massive discounts -- 20-30% under list price, in exchange for spending $X million/year over Y years.

It will also get you a dedicated sales rep and sales team, and they will absolutely crack the whip on internal teams to get issues resolved. At those spends, you can almost get an in-house support team of PSOs to bounce problems off of.

Re: A Google Cloud support engineer solves a tough DNS case

#90

Post by cryptoquick » Tue, May 19, 2020, 5:53 PM UTC

econcon wrote: post #17

Google support is basically non existent for customers even spending like 5-10K a month on their platform. Even questions you raise on their Reddit sub go unnoticed. They don't have bunch of people actively trying to solve their customers problem. That said the only time my problems were actually listed to were from BigQuery team. Other than this, I don't think it's possible to get any explanation on a feature from a…

I was about to say similar. My experiences with Google Support, even on their $500/mo. Gold plan, have been infuriating at best.

Even their product teams are just... Depressing. There was a support ticket open for TEN YEARS for people asking for WebSockets support on Google App Engine.

I often wind up with DevOps responsibilities, and I'd never recommend building more on them, and I'd help in every conceivable way to minimize money given to Google, in addition to aiding the transition to more reliable providers.

And on a more personal note... My husband recently tried recovering his email. He no longer had the password, the phone number he had it registered with was no longer owned by him (a major security issue, btw), so he tried his recovery email. And even after clicking the link from his recovery email, he was denied access, and sent to the same help page that couldn't be more unhelpful if someone actively tried. And there was no apparent way to contact support from that screen.

I hear tell of Gmail users who could reach Google support, but... We don't put much stock in those stories round these parts.

Moral of the story? Never trust Google with your emails, or other important information. Always make backups if you must continue using them, and forward your email to an email provider you trust (ideally one that you pay for, own, and has a decent support department of any kind).