Tell HN: Google Cloud suspended our production projects at 1am on Saturday

It's simple - you give them notice during local working hours (i.e. not the weekend), that way the customer has time to rectify, and if they don't you can take action and have proof of warning them of the consequences of ignoring the notification. If they are having payment issues, it affords both parties time to discuss the issues and see what can reasonably done. It's called customer service - something that Google clearly struggles with.

It has been covered here on HN several times, but to summarize:

Budget cloud hosting on a credit card allows scammers to use stolen CC numbers to spin up VMs to mine crypto.

Essentially the hosting company is offering a cash equivalent for unreliable credit.

This kind of abuse occurs at enormous scale, because it can be worth it for an attacker to use botnets to throw tens of thousands of credit cards at hosting companies. If you never actually expect to pay any money (especially your own!), then even thirty minutes of free compute is worthwhile to chase. There have been similar attacks against GitHub Actions, because they're also free and general-purpose.

This is also why the free-tier, dev-only, education, or similar MSDN or VS Subscriber type cloud subscriptions never permit the use of GPU instances. Too tempting a target!

The large providers like Azure and AWS are basically printing money, so they don't care about the small-fry scammers. They're too busy trying to hold on to the firehose of cash.

Smaller, budget providers like Digital Ocean are running too lean to tolerate scammers, but also can't afford to have humans in the loop from the sheer scale of the attacks.

So they use heuristics that are 99.99% accurate (or whatever), which means one in ten thousand legit customers gets axed along with the scammers. Oops.

Essentially, as a customer of these small providers you are taking on some of their risk in exchange for the discount over the big-name vendors. This is especially true if you pay them with a credit card, irrespective of past payment history. Like I said, they simply can't afford to keep a human in the loop.[1]

An example that came up here was a startup that had an account "ticking along" with a handful of dev stuff, went "live" with a big launch, and so almost all of their servers was loaded to nearly 100% capacity thanks to efficient containerisation and auto-scale.

Good job devs... except that looks identical to a hacked account that has suddenly started to mine crypto on every machine.

Axed.

[1] It's even more complex than you think. Employees can and have been bribed to let the scammers through! The employees themselves might be mining crypto. The scammers can trick them, lie, beg, or just figure out the system from repeat conversations. The only recourse is to take the human out of the loop entirely, nothing else works for them. If you get to be the 0.01%, you generally have no recourse.

> billing works extremely well for me

I suppose billing works extremely well for most GCP customers as well.

As for me, I hosted a side project on Hetzner a few years back for some European presence. IIRC I used PayPal for payment and they didn't support PayPal recurring payment, so I had to settle an invoice each month. Used them for about two years without problem, then missed payment for two months during a chaotic period when I didn't notice their monthly invoice emails. I got exactly one email reminder about missed payment titled "Reminder" (yes, that's all, I got all their correspondence in front of me right now) before the final "Cancellation of Contract" email and termination of my account. At that point, politely pleading with them to reinstate the account was useless, I only got a threatening letter back demanding a wire transfer of the ~€60 I owed within three business days or risk being sent to collections.

Was I in the wrong? Sure. Were they friendly or patient? No.

> we won't allow you to sell any cloud-related services anymore until you don't have proper customer service

No, they attract and select people who see the world only from an engineering perspective. Combined with Google's cultural roots that come from MIT, where projects and services are things that are funded by the government, military or megacorporations in which you can pull the plug whenever and the other side would not flinch, this makes them totally blind to the existence of humans on the other side of their services. Resulting in an false reality in which you, me and the millions of other people who depend on their services do not exist.

So you see things like this happen. Things like 'deprecating' products in people's faces. Things like 'combining and merging and unmerging products because it makes more engineering sense' and creating messes like "Google Gmail Hangouts Chat Rooms Meet" or whatever they are calling the frankenstein they made by combining all those.

> Why should a business relationship with Google (or any other cloud service) be any different?

Because google choose to make this as difficult as possible. A couple of years ago I worked on a third party IAAS database cloud product (think Atlas or Elastic Cloud, but for a different database), and while Azure had the worst tech overall, Google were ultimately the hardest to deal with by a VERY long way because they don’t acknowledge the existence of people as anything other than ad targets.