Live data from Hacker News

Tell HN: Google Cloud suspended our production projects at 1am on Saturday

news.ycombinator.com

1–10 of 24 posts

Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#1
TLDR; never use google cloud systems for production.

Google cloud suspended all our projects due to the billing issue in their system they had.

Despite reassurances "your account will not be suspended" while communicating with billing support, all the projects were suspended at 1am on Saturday.

All the account payments were made and the billing cards are valid. There are no outstanding bills.

Never use GCP for production.

---- Edit full story by request, long read: ----

Previous month billing didn't went through. Not sure if this was due to the billing outage google had (https://status.cloud.google.com/products/oLCqDYkE9NFWQVgctQTL/history) or financial transaction issue, however we went ahead and made a manual payment covering all the outstanding amount + extra. Despite the payment made, about a week+ later we suddenly started receiving threatening emails "Your Projects are at risk of suspension". Edited and updated the billing cards. Opened a billing support request clearly mentioning this is a production environment and all the bills are paid. They were "investigating" the issue and assured the project will not be suspended.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#2
Not a good look for Google.

I still don't think Google understands customer service yet. They think they can solve everything by being more technical than everyone else.

Google's engineering led culture is both a blessing and a curse.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#5
I’m sorry but without more context I won’t upvote this. We hear your side of the story but we won’t hear Google’s.

You say payments were made and billing cards were valid. Your account wouldn’t be suspended for that. I’m convinced there are important nuances you aren’t mentioning.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#6
post #5

I’m sorry but without more context I won’t upvote this. We hear your side of the story but we won’t hear Google’s. You say payments were made and billing cards were valid. Your account wouldn’t be suspended for that. I’m convinced there are important nuances you aren’t mentioning.

Yup. What exactly was the earlier problem? Why did you need assurances that projects would not be suspended in the first place?

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#7
Google are either completely delusional in thinking that at this point ML-based solutions are capable of replacing human decision making or they are fully aware that it's not working well and getting worse over time. It won't change though as savings made on humans are driving their YoY.

Pichai sacrificed (what was left of) Google's good press in favor of YoY. Google is proving time and again that they are unstable and unreliable as a business partner and can't be trusted with anything but being a growing stock market product.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#9

Had a similar experience with GCP. I don’t remember the exactly how, but somehow we ended up with a $0.01 bill. This bill was apparently too small for Google to deduct, instead all our api services were disabled :/

How weird, we get charged 0.01 all the time, and I laugh thinking that it cost them more to charge us a penny than it would've been to just roll it over.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#10
post #7

Google are either completely delusional in thinking that at this point ML-based solutions are capable of replacing human decision making or they are fully aware that it's not working well and getting worse over time. It won't change though as savings made on humans are driving their YoY. Pichai sacrificed (what was left of) Google's good press in favor of YoY. Google is proving time and again that they are unstable a…

[deleted]
Post reply on HN