Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tell HN: Google Cloud suspended our production projects at 1am on Saturday

news.ycombinator.com

Post reply on HN
Print view View original

281–290 of 531 posts

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#281

Post by RA_Fisher »

jaywalk wrote: post #170
klohto wrote: post #142

Earlier quoted context omitted.

What is this fud? Both AWS and Azure won't suspend your account for several months, even with charges failing. Especially not if you're communicating with them. It's incredibly easy to get a hold of human support in case of the other clouds.

My company isn't very big (we spend in the very low five-figures with Azure per month) and yet we were assigned an "Azure Success Manager" (or something close to that) who I have a standing monthly phone call with, and can reach out to at any time to oversee any issues that may come up. We actually did have a billing issue come up a while ago (it was Microsoft's fault) and although our account wasn't in any real dang…

That’s awesome customer service! Thanks for sharing.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#282

Post by le-mark »

cft wrote: post #264
hazbo wrote: post #124

A few years ago I was working at a startup and we had just started moving some stuff over from AWS to GCP. Things were going pretty good with GCP until one day they pulled the plug on everything with no explanation. It turns out that our company credit card details had been fraudulently used without our knowledge. The criminal had decided to use the card to buy Google Ad Words or something like that - this is the sam…

For Google a customer is a data point in their machine learning set (hopefully weighted by revenue). The decisions are made by a ML model. If you treat your relationship with Google as such from the beginning, you can extract some value if you use Google for near zero value use cases - spam/marketing emails, disposable non-real time computations, Adwords/AdSense on spammy sites. Not sure if having an Android phone is…

This, either be big enough for a dedicated account representative (do they even offer that?) or use for noncritical workloads.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#283

Post by andix »

vbezhenar wrote: post #183
b112 wrote: post #157

Earlier quoted context omitted.

Google is just horrible, when it comes to any form of support, paid or not. They make paypal look helpful by comparison. I really can't understand why anyone would use any of their services. Support counts. It's one of the first things I look at, when evaluating any external partner. What does it cost, how long does it take for it to happen, etc? Whatever anyone else says in this thread, that's what you need to deter…

Google tech is the best. Google invented k8s, their k8s cloud is the best in the world. There's nothing similar to GKE Autopilot. Similar things could be said about some other services. People want the best cloud and they hope that those issues won't affect them.

K8S runs well on all major cloud providers. You can run it on premise without licensing costs quite well (for example k3s, rancher, …). Or fully managed with Open Shift.

AWS, Azure and GC are all on the same level. Any of them gives you „the best“ technology.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#284

Post by Nebasuke »

black_puppydog wrote: post #266
Nebasuke wrote: post #252

Earlier quoted context omitted.

I got hit by a car in a situation similar to this. They made eye contact (I thought), were slowing down, and still hit me anyway by accelerating, despite me having the right of way on a bike. Turns out it was the first time they used the car, 1 minute into the drive and their focus was somewhere else. I would say the advice here is, wait until the car is basically stopped before you cross the street rather than trust…

> wait until the car is basically stopped you mean like an email literally saying they acknowledge the situation and won't suspend your account?

I think an email from support is more like "confirmation that they checked whether the car has seen me".

I'm not sure how successfully you've dealt with support with companies, but I have almost never been able to rely on the promises of a support agent, as they are simply not the ones actually making the decisions (in this case on whether your account gets suspended or not).

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#285

Post by majewsky »

smileybarry wrote: post #234
madeofpalk wrote: post #154

Earlier quoted context omitted.

It's not really the payment method that's key here, it's having a relationship with sales / account manager.

Right, but Hetzner also just doesn’t auto-charge you (from my experience, at least). You get invoiced and pay it manually (there’s a link in the invoice), then the invoice is settled immediately in your account panel. I’ve even been late once by forgetting about it and they didn’t cut me off in any way.

> Hetzner also just doesn’t auto-charge you (from my experience, at least).

I have my Hetzner account set up so they pull the money automatically (through SEPA Direct Debit). Not sure how it works with credit cards, but it's not like pull payments are not possible with Hetzner.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#286

Post by marcinzm »

vidarh wrote: post #237
silisili wrote: post #36

Seeing all these horror stories, I can't help but wonder when people are going to switch back to on prem, or at least some hybrid. It's pretty apparent we can't trust cloud providers...

The pendulum will swing over cost, not over outages like these, because the cost challenges are real, while these outages are rare. It won't go all back to on prem because cloud providers will slash their margins once they struggle to maintain growth, and doing so they'll diminish the advantages of on-prem/hybrid setups for a lot of people, and so we'll find some equilibrium or other. I'd expect more emphasis on hybr…

Full on prem is a pretty bad for a large company because your company now depends on a single monopolistic service provider that is run as a cost center (ie: the department managing your on prem stuff). Eventually that leads to a very bad engineer experience which leads to a massive opportunity cost for any tech company. It works for a while and then over time becomes more and more dysfunctional. Hybrid at least includes competition with cloud providers which makes things less dysfunctional.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#287

Post by rendaw »

jamespwilliams wrote: post #19

The same is true of DigitalOcean, fwiw. Shutting down machines used in production with no notice, outside working hours. I don’t know why these companies don’t care - they’re massively shooting themselves in their feet. Incidents like this turn customers off for life.

At a previous employer we were hosting with DO via credit balance. Our finance department couldn't get things together and kept forgetting (almost every month) to add new credits to the account. Despite this DO never stopped or terminated our servers, and we were often days to weeks behind on the payments.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#288

Post by latexr »

doix wrote: post #163
baobabKoodaa wrote: post #133

Earlier quoted context omitted.

Ah yes, it was the customer's fault. Stupid customer using method recommended by service provider.

It's like looking both ways when crossing the road despite the light telling you that you can go. Sure you have right of way and if a car hits you, it's their fault. But at the end of the day, you are the one getting hit by the car. Google is at fault for how they handled it, but the parents comment is still good advice for anyone reading this thread.

According to the OP, Google assured them a suspension wouldn’t happen then did it anyway.

In the car analogy it’s akin to looking the driver in the eyes and them signalling to you it’s safe to cross, only to hit the gas and run you over as you do. No amount of looking both ways before crossing would have saved you.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#290

Post by newaccount74 »

Beltalowda wrote: post #235
helsinkiandrew wrote: post #158

Earlier quoted context omitted.

Hang on - what would you do if you were running a business with hundreds of thousands of users and someone doesn't pay? Was it an accident? has the customer gone bust, or died, or are they trying to defraud you and get a month or two of free service. Credit cards are exceptionally finicky, very susceptible to fraud, and payment can often fail. They're great for one off purchases and services that don't matter if they…

I actually had some payment troubles with Linode a few months back; I had cancelled one of my bank accounts and forgot the CC there was tied to Linode, and because credit cards are not very common where I live and it can take a bit to request one so I couldn't "just" pay. Oops! 100% a mistake on my part of course; I emailed Linode, explained the above, and they said "sure, is 3 weeks time enough to sort it out? I've…

Linode stopped my services because of 20$ outstanding balance that I ignored for two weeks (was on vacation). When I tried to update the payment method, the card wasn't accepted, and after a few tries I got an error message that I made too many attempts and wouldn't be allowed to change it for the next 24h.

Fortunately they have customer support, and 5 minutes after emailing them they resumed my services again.

Post reply on HN