Live data from Hacker News

Tell HN: Google Cloud suspended our production projects at 1am on Saturday

news.ycombinator.com

1–10 of 531 posts

Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#1
TLDR; never use google cloud systems for production.

Google cloud suspended all our projects due to the billing issue in their system they had.

Despite reassurances "your account will not be suspended" while communicating with billing support, all the projects were suspended at 1am on Saturday.

All the account payments were made and the billing cards are valid. There are no outstanding bills.

Never use GCP for production.

---- Edit: full story by request, long read: ----

Previous month billing didn't went through. Not sure if this was due to the billing outage google had (https://status.cloud.google.com/products/oLCqDYkE9NFWQVgctQTL/history) or financial transaction issue, however we went ahead and made a manual payment covering all the outstanding amount + extra. Despite the payment made, about a week+ later we suddenly started receiving threatening emails "Your Projects are at risk of suspension". Edited and updated the billing cards. Opened a billing support request clearly mentioning this is a production environment and all the bills are paid. They were "investigating" the issue and assured the project will not be suspended.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#3
Not exclusive to GCP.

Backblaze has previously locked our account on a saturday.

They claim we were putting too much pressure on their cluster.

They could've locked the 1! token that was causing 90% of the load, but they locked the whole account which was used by 10 different apps for a veriety of purposes causing unecessary downtime in some production workloads.

Also the workload has been quite consistent for days/weeks, so they could've detected it earlier and/or address it in a better way.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#4
Sorry, not really sorry.

It has been well documented that any of the big tech companies will thankfully flush you and all you've build down the drain whenever they feel like that. You'll also probably never be able to talk to someone (high enough up the ladder) to figure things out.

Nevertheless lots of people are building their tech and their lives based on the services provided by these companies, when a maliciously flagged YouTube comment for a linked account can kill _everything_ in seconds.

Stop building stuff on their platforms. Now.

edit: Yeah, probably not very "helpful" but how many of the "Google killed my X for no reason" posts will we have to see until either regulatory bodies step in (doesn't look like that) or people will just stop using those services. Yeah, it's cheap and it's easy to start with, but so are some drugs, too.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#5

Not exclusive to GCP. Backblaze has previously locked our account on a saturday. They claim we were putting too much pressure on their cluster. They could've locked the 1! token that was causing 90% of the load, but they locked the whole account which was used by 10 different apps for a veriety of purposes causing unecessary downtime in some production workloads. Also the workload has been quite consistent for days/w…

Whoa - how much traffic were you generating?

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#8
post #6

I wonder what it would take to bring a culture change to Google so they can finally start having some empathy for customers and stop the casual cruelty. A new CEO?

It’s probably too big for one strong personality to get it back to the Don’t Be Evil days.

Regulation, litigation, or split it into pieces.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#9

Out of curiosity, what level of support did you have? I'm referring to these https://cloud.google.com/support What I hear is that unless you have at least standard support, they treat you like garbage.

The only entities Google doesn't treat like garbage are advertisers. All others are part of the product

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#10
Google Cloud is a weird thing. It's got Google quality:

- technically amazing: BigQuery and GKE are great

- really half assed as a product: console is somehow slower than AWS despite the latter being the most tech debt of the lot for having invented everything, support half witted, and honestly I had a card declined and even Alibaba Cloud didn't suspend my shit

If you have one of their premium support things you'll get faster service but I think I'm not hyper keen on them for general stuff.

Post reply on HN