Tell HN: Google Cloud suspended our production projects at 1am on Saturday • Hacker News Board

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#401

Post by shadowgovt » Mon, Aug 22, 2022, 2:23 PM UTC

andrewinardeer wrote: post #88

I know I'm going to come off as a Google apologist, however, I'm leaning towards there being more to the story than what you are saying. The previous month's billing didn't 'went through' and you are unable to zero in on whether this is a Google issue or a you issue. I get that it is easier to complain on the internet and often cathartic do so as a coping mechanism. Having an account suspended because of a billing is…

Google is the outlier here. Relative to their competition, they're both new at public-facing corporate Cloud offerings(1) and new at non-scalable customer service issues.

They're playing catch-up, and customers are at risk while they do so.

(1) AWS had a head-start on Google and Google's been chasing AWS's monetization strategy and away from their Google App Engine approach, which was fundamentally different. Azure may be more contemporary to Google's Cloud, but Microsoft has a tried-and-true large-scale corporate customer support engine that Google's only now being forced to build out to support its Cloud offering (the customer support story for ads was fundamentally different, and Apps for Domains didn't have the scale Cloud now does).

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#402

Post by patchtopic » Mon, Aug 22, 2022, 2:23 PM UTC

luka-birsa wrote: post #386

A suggestion for engineers that lack business experience. As soon as your project grows beyond the non-hobby scale and starts making real money, invest some time to reach out to company providing the services that keep it alive and find an account manager that will take care of you. We're a Google customer and of course we had our fair share of issues (btw: it's the same with Azure, based on our experience), and we a…

after Azure sunk my employers project with no recourse or discussion allowed, I always advise everyone I meet to run far away from Azure if they value a reliable service.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#403

Post by patchtopic » Mon, Aug 22, 2022, 2:26 PM UTC

intellectronica wrote: post #115

Generally, all cloud providers have two ways of engaging with and billing customers. One is using a credit card, which is very easy for getting started. The other one is having an established relationship with a sales / account team and setting up billing by invoice. Smaller companies sometimes don't understand that and stick with credit card billing for too long. Once you move to production and have your own custome…

the small azure project that got canned by Azure I was working on, credit card was the only way to pay, and Azure suspended my employers Azure account with no explanation why (though it appears in hindsight to be credit card related), but they refused to explain why. this victim blaming gaslighting is ridiculous.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#404

Post by the__alchemist » Mon, Aug 22, 2022, 2:30 PM UTC

helsinkiandrew wrote: post #158

baobabKoodaa wrote: post #133

Earlier quoted context omitted.

Ah yes, it was the customer's fault. Stupid customer using method recommended by service provider.

Hang on - what would you do if you were running a business with hundreds of thousands of users and someone doesn't pay? Was it an accident? has the customer gone bust, or died, or are they trying to defraud you and get a month or two of free service. Credit cards are exceptionally finicky, very susceptible to fraud, and payment can often fail. They're great for one off purchases and services that don't matter if they…

I strongly disagree. The host must reach out to the customer, person-to-person, before shutting off a service.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#405

Post by dafelst » Mon, Aug 22, 2022, 2:31 PM UTC

itsoktocry wrote: post #393

bastawhiz wrote: post #255

Earlier quoted context omitted.

How are you supposed to trust a company who told you they wouldn't screw you because of their own error, then screwed you anyway? Having a relationship with them doesn't matter if they still can't be trusted to do business. The type of payment method you use is not a factor that plays into whether you can trust your vendor.

> How are you supposed to trust a company who told you they wouldn't screw you because of their own error, then screwed you anyway? I mean, it's an error. Sure, it's infuriating, and in worst-case could financially harm (or cripple) your business. But Google didn't intend for it to happen. That makes no sense from a business perspective. So, as others here suggest, you have to plan around the fact that mistakes like…

I think the difficulty is (ESPECIALLY with Google and suspected fraud) that when they make a mistake, getting it corrected is basically impossible unless you know a person on the inside like an established account manager or a highly ranked exec.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#406

Post by judge2020 » Mon, Aug 22, 2022, 2:33 PM UTC

euph0ria wrote: post #46

The company I work for has forbidden the use of Google and their services (if possible) such as GCP etc. due to how they treat their customers. In particular, there never seems to be any human being that you can talk to and find out what you need to do to fix the issue. We do not want the same to occur to our production systems if there is an overflow from a Play Store ban to GCP etc. The business risk is too high wh…

> if there is an overflow from a Play Store ban to GCP etc

This only happens if you're using personal google accounts / @gmail accounts to manage gcp resources. Even if you don't use Workspace as your main directory, you can set it to be a SAML consumer so you still get full Workspace data governance.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#407

Post by whoevercares » Mon, Aug 22, 2022, 2:35 PM UTC

luka-birsa wrote: post #386

A suggestion for engineers that lack business experience. As soon as your project grows beyond the non-hobby scale and starts making real money, invest some time to reach out to company providing the services that keep it alive and find an account manager that will take care of you. We're a Google customer and of course we had our fair share of issues (btw: it's the same with Azure, based on our experience), and we a…

That’s a good suggestion - it worth noting it comes with a cost (eg X% of your cloud bill) but you get much better interaction with the support and service team.

AWS sometimes even connects small players with service team PMs and managers. It was really great but probably only for greenfield services

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#408

Post by intellectronica » Mon, Aug 22, 2022, 2:38 PM UTC

acka wrote: post #384

intellectronica wrote: post #245

Earlier quoted context omitted.

Most of it will actually be documented by your provider, and is often pretty obvious. For example, if you're paying with card, have at least one extra as backup payment method (which most providers allow and recommend). For extra safety: make one of these cards a pre-paid debit card loaded with enough money to pay for at least one month. The one thing you don't want to happen is to miss a payment, which could happen…

intellectronica: > For example, if you're paying with card, have at least one extra as backup payment method (which most providers allow and recommend). Please don't generalize. There are many countries where credit cards aren't common, and applying for just one (and getting it approved) can already involve a lot of work and take lots of time. Such as: the bank and the national credit authority needing to do a thorou…

That's true - it's a rather privileged view to assume that everyone all over the world enjoys the same facilities, I know that. If you are somewhere where you can't have services people from rich countries consider basic, like credit cards, then you are left out. That sucks. Hopefully the more decentralized internet being built these days will be a solution.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#409

Post by nwsm » Mon, Aug 22, 2022, 2:38 PM UTC

pontifier wrote: post #103

Murfie died because of Google cloud costs. It's a bit of a long story, but from what I was able to piece together, a Google cloud service provider recommended some changes to the way the storage was done to save money. Mostly, because of those changes they got hit by a surprise $30,000 bill for data egress. It was basically the last straw for a company that wasn't profitable due mostly to cloud costs. It killed the c…

Does the customer data have a proper home yet or is it still sitting in the broken-into shipping containers?

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#410

Post by 9dev » Mon, Aug 22, 2022, 2:39 PM UTC

mcny wrote: post #85

rglullis wrote: post #72

Earlier quoted context omitted.

These "issues" only exist because of the scale and reach of Corporations. You can not fix "too big to fail" issues unless you stop letting these institutions to grow too big in the first place. I'm yet to hear a single convincing argument of why we need to have organizations of such a large scale nowadays. Could you make a case of what we would be missing if we just "threw the whole thing out"?

I propose another option: unless you can show clear written instructions, the ceo and the board are personally criminally responsible for the acts of their agents, contractors, and employees at the course of their duty. If your branch manager incorrectly adds your teller to an industry wide list of do not hire, the board and CEO are personally criminally responsible for it. No hiding behind the corporate veil. I don’…

Wouldn’t that result in the CEO and board being literal scapegoats, employed by the people actually in charge, to be beheaded if something goes wrong? They would even sell it as „giving control to the employees“ or something.

I don’t believe in measures to radically change the world. That’s both a direct example of the Chesterton fence, and applied game theory - CEOs, board members, or investors, they are all gamers. Give them any set of rules, they will analyse and subsequently game them.