Live data from Hacker News

Tell HN: Google Cloud suspended our production projects at 1am on Saturday

news.ycombinator.com

101–110 of 531 posts

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#101
post #12

It sucks that they gave you conflicting information that resulted in this but I wouldn't trust having my cloud infrastructure payments be handled using a single credit card. If you have a spend of over $2500/mo you can apply for invoiced billing directly from Google or if you have less than that you can contact a GCP reseller partner to get invoiced by them. Some GCP reseller partners also have a close relationship w…

What's the benefit of invoiced billing?

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#102
post #16
post #6

I wonder what it would take to bring a culture change to Google so they can finally start having some empathy for customers and stop the casual cruelty. A new CEO?

Regulation, full stop. The reason Big tech makes so much profits is because they have no accountability and at their scale, social shaming doesn’t work.

I'm a fan of regulating big tech, but in this case reputation will also make a big difference. I know lots of people who actively avoid google's cloud products because they don't trust google not to pull stunts like this.

There have been too many stories on HN and twitter over the years of business critical google accounts being suspended for no reason or google cloud products massively increasing their price overnight.

I don't trust google as a provider for business-critical infrastructure. I'm far from alone.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#103
Murfie died because of Google cloud costs.

It's a bit of a long story, but from what I was able to piece together, a Google cloud service provider recommended some changes to the way the storage was done to save money. Mostly, because of those changes they got hit by a surprise $30,000 bill for data egress. It was basically the last straw for a company that wasn't profitable due mostly to cloud costs. It killed the company. That's why I had to step in to try to save them.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#104
post #33

The law should come hard on this sort of shenanigans, but hitting on Big Tech has already become a political issue in the US, Europe is, in the end, too afraid to do it (bar some relatively minor extra taxes that Big Tech is all to happy to pay as long as it's not made legally liable in other ways), and here we are, with crazy stories like this one (one of many).

>Europe is, in the end, too afraid to do it (bar some relatively minor extra taxes that Big Tech is all to happy to pay as long as it's not made legally liable in other ways),

https://en.m.wikipedia.org/wiki/Antitrust_cases_against_Goog...

$8 billion afraid. If you consider that 'minor', i rather suspect you are being anti-EU just for the hell of it.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#105
post #37

Earlier quoted context omitted.

If you are paying for the load/pressure, what's the problem?

Backblaze is very cheap and has several "don't overload our system" clauses in their terms and conditions. They're cheap and reliable, that generally comes at the risk of early termination. I don't think you can have a cheap, reliable, unlimited service; pick two at most.

Isn't everything metered with Backblaze (ie. no unlimited)? I think only things such as delete aren't metered. But in order to delete something you must have first uploaded it (which is metered).

If everything is metered it seems kind of odd to complain about a lot of activity. Since it's literally the thing you make money with.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#106

Not exclusive to GCP. Backblaze has previously locked our account on a saturday. They claim we were putting too much pressure on their cluster. They could've locked the 1! token that was causing 90% of the load, but they locked the whole account which was used by 10 different apps for a veriety of purposes causing unecessary downtime in some production workloads. Also the workload has been quite consistent for days/w…

I don't see this as related - your account triggered some abuse detection - did you ever go to them and validate what you're doing is covered by the terms you're paying for ?

I mean it sounds like a dick move by Backblaze, but comparing it to what OP is describing is downplaying how bad this looks for GCP IMO.

Going through with official contact at GCP, being assured he did everything right and that it was taken care of - and still getting shut down. In my book that's just inexcusable - if you can't count on that kind of support to work then how can you rely on that platform for anything ?

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#109
post #79

The same is true of DigitalOcean, fwiw. Shutting down machines used in production with no notice, outside working hours. I don’t know why these companies don’t care - they’re massively shooting themselves in their feet. Incidents like this turn customers off for life.

I actually managed to get my account suspended one day due to nonpayment. It was just a personal toy project and it was completely my fault: I was automatically filtering the mails into a folder that I didn't read until it was too late. If you look at the sheer amount of emails they sent me before suspending the account, Digital Ocean did everything right: https://i.imgur.com/ah13bEA.png They gave lots of warnings an…

Just like you setup multiple instances across different availability zones for redundancy, you also need multiple billing contacts in case one of them fails and causes an outage.

Not sure how to mitigate these risks for providers that only support one "project owner" with a single billing address :/

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#110
post #6

I wonder what it would take to bring a culture change to Google so they can finally start having some empathy for customers and stop the casual cruelty. A new CEO?

I'm kinda surprised they don't voluntarily allow courts to make these tricky decisions for them.

Perhaps even offer to pay court costs.

They can say:

> We suspended your account for violation of XYZ rule. You can appeal internally, and if that is denied, you can ask a court to rule on if you broke this rule. If the court finds you did not violate the rule, we will reinstate your account. We will provide any data from your account to the court to make its decision, including proprietary data that we cannot reveal directly to you, for example data about other linked accounts.

Post reply on HN