Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tell HN: Google Cloud suspended our production projects at 1am on Saturday

news.ycombinator.com

Post reply on HN
Print view View original

201–210 of 531 posts

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#201

Post by rglullis »

iakov wrote: post #196
rglullis wrote: post #20

Earlier quoted context omitted.

Right. Heavily-regulated industries like banking, auto and healthcare are known for being so much better at treating its customers like human beings...

I haven't heard stories about bank's AI deciding to close a person's account, take all their life savings, and refuse any explanation, contact or cooperation. Maybe regulation has to do something with it?

1) There have been stories of N26 customers who are locked out of their accounts, or had their devices stolen and lost their funds to thieves and the bank refused to reverse the transaction.

2) No, regulation (or the lack of it) has nothing to do with it.

3) No one is arguing for the end of regulation.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#202

Post by paskozdilar »

intellectronica wrote: post #185
baobabKoodaa wrote: post #141

Earlier quoted context omitted.

For the record, I was being sarcastic and do not actually believe it was the customer's fault. I believe this is an abhorrent way of running business by Google.

Right, but I think your sarcasm was misplaced. It is actually possible for customers to make a mistake. These mistakes are relatively easy to prevent. It could be that in this particular case there wasn't anything the OP could have done and they are simply unlucky. But that's very unlikely.

> These mistakes are relatively easy to prevent.

Assuming one knows about them. How would one learn about them, without making the same mistakes themselves?

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#203

Post by unity1001 »

KingOfCoders wrote: post #29

Also: If you're on AWS/GCP and only backup to S3/GCS, you're doing it wrong. All your backups to setup a new system (even if it takes time) need to be with a different provider.

You can get some cheap storage from Hetzner and also back up there. With their !sane! egress transfer costs compared to the lunacy that plagues Google, AWS etc, accessing the data wont ever be problem either.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#204

Post by cosmodisk »

FBISurveillance wrote: post #155
dvfjsdhgfv wrote: post #135

Earlier quoted context omitted.

I'm very happy Hetzner has all these options built in from the very start. There's no need to make any "extra effort". Actually, most of my servers are on yearly billing to make things even easier for everybody.

Except finance. At certain scale (in my case, 5 figures/mo spend with a vendor and up), paying upfront is not something finance usually likes and NET-60/NET-90 is the norm.

Of course they don't like it, but that's why CTOs and the likes should be talking about these things in management meetings and not just fairy dust future plans.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#205

Post by unity1001 »

silisili wrote: post #36

Seeing all these horror stories, I can't help but wonder when people are going to switch back to on prem, or at least some hybrid. It's pretty apparent we can't trust cloud providers...

People won't switch to on-prem because if Google and some other providers screw up, there still are many other cloud providers. For K8 they could have gone with anyone from Digitalocean to Linode. Actually, they could have already replicated their cluster to more than one provider to use as failover in such a case. (assuming that they were using K8)

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#206

Post by jquery »

doix wrote: post #163
baobabKoodaa wrote: post #133

Earlier quoted context omitted.

Ah yes, it was the customer's fault. Stupid customer using method recommended by service provider.

It's like looking both ways when crossing the road despite the light telling you that you can go. Sure you have right of way and if a car hits you, it's their fault. But at the end of the day, you are the one getting hit by the car. Google is at fault for how they handled it, but the parents comment is still good advice for anyone reading this thread.

It sounds more like they looked both ways, they saw a car, they made eye contact and the driver gestured for them to cross, then when they began to cross the driver ran them over. And now they’re getting advice to never cross streets as a pedestrian. Sound advice can also be victim blaming.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#207

Post by unity1001 »

euph0ria wrote: post #46

The company I work for has forbidden the use of Google and their services (if possible) such as GCP etc. due to how they treat their customers. In particular, there never seems to be any human being that you can talk to and find out what you need to do to fix the issue. We do not want the same to occur to our production systems if there is an overflow from a Play Store ban to GCP etc. The business risk is too high wh…

Eu's digital gatekeepers law will kick some of these bad behaviors in the curb. But letting large tech companies go unaccountable despite their near total domination in certain aspects of life has been something incredible that we have done.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#208

Post by magicalhippo »

jamespwilliams wrote: post #19

The same is true of DigitalOcean, fwiw. Shutting down machines used in production with no notice, outside working hours. I don’t know why these companies don’t care - they’re massively shooting themselves in their feet. Incidents like this turn customers off for life.

> outside working hours

Working hours are relative. Our support routinely gets awoken at like 3am because someone in Singapore, where one of our larger customers has outsourced their IT to, just started their day and decided to follow up on a case.

Though I'll agree that weekends should be considered off-limits regardless.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#209

Post by unity1001 »

tastemykungfu wrote: post #148

Really sad Google just continues to do this BS. At the company I work for, we are seriously evaluating moving off of AWS (about ~$10k/mo bill). This is not going to fly.

Try digitalocean. Their offerings are competitive, with a fraction of the price. With that kind of low cost, you can have failovers elsewhere. Like Linode, or even Hetzner.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#210

Post by lelanthran »

rglullis wrote: post #128
riedel wrote: post #116

Earlier quoted context omitted.

I fully agree that automakers should be held responsible. However, I am feeling a bit of negative sentiment here. What does 'properly' mean? There has been court cases that have been decided against them. Sure there is always more. But this sounds a bit like whatabotism. Also there is a high probability of defeat devices in Citroen, Dacia, Fiat, Ford and Honda cars as well. It is however true that the EU create a hug…

"Properly" means that the automakers should pay a fine per vehicle sold that was producing more emissions than allowed, and that fine should be so high as to effectively get them into bankruptcy. It should also put in jail anyone that was aware of the violation and did not report to the authorities. Everyone , from the CEO to the test engineer who knew about the cheating devices. Anything less than that and all you a…

> Anything less than that and all you are getting is an incentive for the make a risk calculation between profit and losses of being caught.

But that logic applies to anything that any company does. You want to effectively kill any company for any infraction?

That sounds a little too severe.

Post reply on HN