Live data from Hacker News

Tell HN: Google Cloud suspended our production projects at 1am on Saturday

news.ycombinator.com

331–340 of 531 posts

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#332
post #16
post #6

I wonder what it would take to bring a culture change to Google so they can finally start having some empathy for customers and stop the casual cruelty. A new CEO?

Regulation, full stop. The reason Big tech makes so much profits is because they have no accountability and at their scale, social shaming doesn’t work.

I'm not opposed to regulation, but in this case, another solution already exists. That is, competitive services that pride themselves on great customer service and satisfaction. Apple kind of does this in the iPhone vs Google phone wars. Azure and AWS do this in the cloud wars. Fastmail and Proton mail do this in the email wars, etc. Regulation didn't force Fastmail to be awesome; it's just the core of their business model and differentiation.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#333

Earlier quoted context omitted.

> Working hours are relative Sorry but for me this doesn't cut it when you have staff and availability zones in every continent on the planet. It isn't some mom and pop shop in one town. It shouldn't be much to ask a company of Google's size and breadth to consider your local timezone for urgent communications.

It was more a comment on how it's difficult to pin down working hours in this international day and age. I mean we're based in Norway, but what if we outsourced our IT to Chile? How would Google or similar know that they need to contact our IT guys during Chilean working hours, and not Norwegian working hours? If it's an invoice that's not paid, well that's done here in Norway though, so definitely Norwegian working…

Worst case they can ask, somewhere in the account settings. Defaulting to the billing region should handle 99% of cases though.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#334

Earlier quoted context omitted.

Use a “synthetic” project owner with an email address that forwards to the group.

That could backfire, since nobody really feels in charge. https://en.wikipedia.org/wiki/Bystander_effect

Have one person clearly in charge, but as the emails get more frequent with increasingly dramatic titles the other people can check in and ask if everything is under control

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#335

Earlier quoted context omitted.

> Smaller companies sometimes don't understand that [...] you should really make the extra effort Let me get this straight. This company does everything they were supposed to, paid on time, and Google's own error cuts them off, and you are blaming them?!?

You don't know that. Maybe that's the case (I'm sure it happens sometimes), but most likely it isn't.

If you assume, without evidence, that an HN user is lying if they share a story outside your experience -- there is little reason to use HN.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#336

Earlier quoted context omitted.

They probably meant their B2 storage service, which is like S3 and bills for storage, egress, some operations, etc. That’s one service I wouldn’t expect to have some cancellation for overload in place. You’re using a token anyway, why not just rate-limit you instead?

Blackblaze B2 is supposed to be AWS S3 competitor. S3 is simply unbreakable. If Blackblaze intends to compete with S3, they can't complain about load stress on their B2.And they can't rate limit either.

> S3 is simply unbreakable

Minor nitpick, but this is categorically not true. As a customer, one who proactively worked with AWS AMs, SAs, and S3 engineers directly before making the change, I still took down S3 by just changing how new files were stored in our S3 bucket. Yes, this S3 bucket was at the time the largest bucket (based on number of files), but still not “simply unbreakable”.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#338
post #189

Earlier quoted context omitted.

Need another Leetcode section on heartlessly screwing your SaaS customers.

Wasn't there an algorithm for that?

It sounds like they need a whiteboard session on something like “policy engine to decouple billing status from service status”.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#339

This is why my old hosting company changed its auto disconnection code to work at 11am on a Monday morning, never at a weekend. That way it's the bill-payers that notice and get in touch, not the poor tech on the out-of-hours rota. Also, it only firewalled the customer's public IPs from the Internet - it didn't stop servers. That way when the account is settled there was often nothing for the techs to do. Also it was…

The method you describe seems like the proper way to achieve this thorny issue.
Post reply on HN