Live data from Hacker News

Tell HN: Google Cloud suspended our production projects at 1am on Saturday

news.ycombinator.com

121–130 of 531 posts

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#121

Earlier quoted context omitted.

Right. Heavily-regulated industries like banking, auto and healthcare are known for being so much better at treating its customers like human beings...

In these industries the regulation is the only thing preventing them being unfathomably worse.

It's not the only thing.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#122

Also: If you're on AWS/GCP and only backup to S3/GCS, you're doing it wrong. All your backups to setup a new system (even if it takes time) need to be with a different provider.

> If you're on AWS/GCP and only backup to S3/GCS, you're doing it wrong.

If that’s how you feel then you should be applying that logic to any provider. Even if you rent rack space in a data center. Because unless you own everything through and through. This could happen anywhere.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#123
post #116

Earlier quoted context omitted.

I'll buy that when the German auto makers get to pay properly for Dieselgate. Until then, EU regulation will be just a demonstration that money and power still talks just like in the US, the only difference is that Europeans are better at fooling themselves about their moral superiority.

I fully agree that automakers should be held responsible. However, I am feeling a bit of negative sentiment here. What does 'properly' mean? There has been court cases that have been decided against them. Sure there is always more. But this sounds a bit like whatabotism. Also there is a high probability of defeat devices in Citroen, Dacia, Fiat, Ford and Honda cars as well. It is however true that the EU create a hug…

It's not whataboutism. Read the comment it was replying to.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#124
A few years ago I was working at a startup and we had just started moving some stuff over from AWS to GCP. Things were going pretty good with GCP until one day they pulled the plug on everything with no explanation. It turns out that our company credit card details had been fraudulently used without our knowledge. The criminal had decided to use the card to buy Google Ad Words or something like that - this is the same card we use on our GCP billing account. Anyway Google just took prod down with no notice on grounds of fraud. Could happen to anyone.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#125
post #16

Earlier quoted context omitted.

Regulation, full stop. The reason Big tech makes so much profits is because they have no accountability and at their scale, social shaming doesn’t work.

no accountability Can't they sue google? Surely this is a case here.

Sue google? Are you mad? What do you think you'll accomplish with that ?

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#126

Seeing all these horror stories, I can't help but wonder when people are going to switch back to on prem, or at least some hybrid. It's pretty apparent we can't trust cloud providers...

I think it's a vocal minority. The vast majority of GCS customers won't have this problem. These kind of posts typically do very well on HN.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#127
I keep hearing about this "automation" and sure, I'm probably too dumb to understand FAANG-level automation but aren't there switches that the automated workflows are expected to check against, atleast before running high-impact actions are like account suspension? Switches that can be pushed by the customer service groups with appropriate entitlement? I have written components with such overrides (with proper auditing to ensure review) and that worked pretty well as a safety-valve. Surely a bunch of PHDs with research papers can't be all wrong so what am I missing here?

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#128
post #116

Earlier quoted context omitted.

I'll buy that when the German auto makers get to pay properly for Dieselgate. Until then, EU regulation will be just a demonstration that money and power still talks just like in the US, the only difference is that Europeans are better at fooling themselves about their moral superiority.

I fully agree that automakers should be held responsible. However, I am feeling a bit of negative sentiment here. What does 'properly' mean? There has been court cases that have been decided against them. Sure there is always more. But this sounds a bit like whatabotism. Also there is a high probability of defeat devices in Citroen, Dacia, Fiat, Ford and Honda cars as well. It is however true that the EU create a hug…

"Properly" means that the automakers should pay a fine per vehicle sold that was producing more emissions than allowed, and that fine should be so high as to effectively get them into bankruptcy. It should also put in jail anyone that was aware of the violation and did not report to the authorities. Everyone, from the CEO to the test engineer who knew about the cheating devices.

Anything less than that and all you are getting is an incentive for the make a risk calculation between profit and losses of being caught.

> Also there is a high probability of defeat devices in Citroen, Dacia, Fiat, Ford and Honda cars as well.

Which would be a sign that the regulations are not really working and consumers are just getting fooled into believing that their beloved EU is oh-so-awesome.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#129

The company I work for has forbidden the use of Google and their services (if possible) such as GCP etc. due to how they treat their customers. In particular, there never seems to be any human being that you can talk to and find out what you need to do to fix the issue. We do not want the same to occur to our production systems if there is an overflow from a Play Store ban to GCP etc. The business risk is too high wh…

> In particular, there never seems to be any human being that you can talk to and find out what you need to do to fix the issue

This isn't really 100% true. It's just that Google uses third party contractors for the grunt work and they are working off a script. You can contact them but the comms with them is weird you won't get a direct response only canned responses as they are not allowed to talk to you directly. I'm pretty sure that these contractors are working out of India due to the times they would "respond" to the ticket.

I went through this recently with one of my apps because you can use the Google picker to load files. Very strange and stressful process I was this close to just deleting that feature due to the insane process.

Example was I had a spelling / grammar mistake on my landing page (minor) and they knocked back the application by sending me a premade email with all the rules that need followed which was full of spelling mistakes. Then once I corrected this they said I had a broken link (it was a link to instagram which didn't work and I took it off) again the email they sent to me was full of broken links. Very weird.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#130
This is why my old hosting company changed its auto disconnection code to work at 11am on a Monday morning, never at a weekend.

That way it's the bill-payers that notice and get in touch, not the poor tech on the out-of-hours rota.

Also, it only firewalled the customer's public IPs from the Internet - it didn't stop servers. That way when the account is settled there was often nothing for the techs to do.

Also it was only ever done after the bank reconciliation was marked as complete, just in case...

We relied on goodwill and recommendation of engineers for our growth, so were keen not to make their lives harder for someone else's bureaucratic (or cynical) mistake.

Post reply on HN