Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tell HN: Google Cloud suspended our production projects at 1am on Saturday

news.ycombinator.com

Post reply on HN
Print view View original

251–260 of 531 posts

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#251

Post by thinkingemote »

bilalq wrote: post #6

I wonder what it would take to bring a culture change to Google so they can finally start having some empathy for customers and stop the casual cruelty. A new CEO?

Ironically, empathy is the reason why they are doing this. Empathy for the support worker. They literally call it Human Ops. The idea is that support work is horrible for their intelligent engineers and so they invest in AI to replace human the bad jobs that impact humans.

Of course human ops is originally about reducing actual traumatic jobs like reviewing actual evil images, but its basically now all brain work which techies hate.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#252

Post by Nebasuke »

jquery wrote: post #206
doix wrote: post #163

Earlier quoted context omitted.

It's like looking both ways when crossing the road despite the light telling you that you can go. Sure you have right of way and if a car hits you, it's their fault. But at the end of the day, you are the one getting hit by the car. Google is at fault for how they handled it, but the parents comment is still good advice for anyone reading this thread.

It sounds more like they looked both ways, they saw a car, they made eye contact and the driver gestured for them to cross, then when they began to cross the driver ran them over. And now they’re getting advice to never cross streets as a pedestrian. Sound advice can also be victim blaming.

I got hit by a car in a situation similar to this. They made eye contact (I thought), were slowing down, and still hit me anyway by accelerating, despite me having the right of way on a bike. Turns out it was the first time they used the car, 1 minute into the drive and their focus was somewhere else.

I would say the advice here is, wait until the car is basically stopped before you cross the street rather than trusting it will because you think all the signs say they will. I do think this is solid advice for the production systems as well. All the signs can point to yes, but making extra assurances to avoid the bad path is valid for something critical like your production systems (or your life).

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#253

Post by magicalhippo »

qabqabaca wrote: post #242
magicalhippo wrote: post #208

Earlier quoted context omitted.

> outside working hours Working hours are relative. Our support routinely gets awoken at like 3am because someone in Singapore, where one of our larger customers has outsourced their IT to, just started their day and decided to follow up on a case. Though I'll agree that weekends should be considered off-limits regardless.

> Working hours are relative Sorry but for me this doesn't cut it when you have staff and availability zones in every continent on the planet. It isn't some mom and pop shop in one town. It shouldn't be much to ask a company of Google's size and breadth to consider your local timezone for urgent communications.

It was more a comment on how it's difficult to pin down working hours in this international day and age.

I mean we're based in Norway, but what if we outsourced our IT to Chile? How would Google or similar know that they need to contact our IT guys during Chilean working hours, and not Norwegian working hours? If it's an invoice that's not paid, well that's done here in Norway though, so definitely Norwegian working hours there...

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#254

Post by intellectronica »

TomSwirly wrote: post #197
intellectronica wrote: post #115

Generally, all cloud providers have two ways of engaging with and billing customers. One is using a credit card, which is very easy for getting started. The other one is having an established relationship with a sales / account team and setting up billing by invoice. Smaller companies sometimes don't understand that and stick with credit card billing for too long. Once you move to production and have your own custome…

> Smaller companies sometimes don't understand that [...] you should really make the extra effort Let me get this straight. This company does everything they were supposed to, paid on time, and Google's own error cuts them off, and you are blaming them?!?

You don't know that. Maybe that's the case (I'm sure it happens sometimes), but most likely it isn't.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#255

Post by bastawhiz »

doix wrote: post #163
baobabKoodaa wrote: post #133

Earlier quoted context omitted.

Ah yes, it was the customer's fault. Stupid customer using method recommended by service provider.

It's like looking both ways when crossing the road despite the light telling you that you can go. Sure you have right of way and if a car hits you, it's their fault. But at the end of the day, you are the one getting hit by the car. Google is at fault for how they handled it, but the parents comment is still good advice for anyone reading this thread.

How are you supposed to trust a company who told you they wouldn't screw you because of their own error, then screwed you anyway? Having a relationship with them doesn't matter if they still can't be trusted to do business. The type of payment method you use is not a factor that plays into whether you can trust your vendor.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#256

Post by rightbyte »

whatshisface wrote: post #35
harry8 wrote: post #31

Earlier quoted context omitted.

Yup. Too big to fail sucks just as hard in banking. Google needs to be broken up as badly as banks do. It's a battle that keeps on coming up and here it is again.

What would someone break Google up into, ad and vertisements? If you look at their quarterly reports you'll see that they are almost entirely based around one product.

You could cut out ads from the rest. And make Google buy ad space from Foogle Search.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#257

Post by theshrike79 »

hazbo wrote: post #124

A few years ago I was working at a startup and we had just started moving some stuff over from AWS to GCP. Things were going pretty good with GCP until one day they pulled the plug on everything with no explanation. It turns out that our company credit card details had been fraudulently used without our knowledge. The criminal had decided to use the card to buy Google Ad Words or something like that - this is the sam…

Never mess with Ad Words, that makes the Alphabet Beast angry every time.

I've heard the exact same Ad Words story a few times now, someone used their CC or had at one point in the distant past used with Ad Words and the system flagged it. Everything gone with no warning, no explanation and no way to complain. Even getting to the front page of HN didn't work =)

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#258

Post by bastawhiz »

helsinkiandrew wrote: post #158
baobabKoodaa wrote: post #133

Earlier quoted context omitted.

Ah yes, it was the customer's fault. Stupid customer using method recommended by service provider.

Hang on - what would you do if you were running a business with hundreds of thousands of users and someone doesn't pay? Was it an accident? has the customer gone bust, or died, or are they trying to defraud you and get a month or two of free service. Credit cards are exceptionally finicky, very susceptible to fraud, and payment can often fail. They're great for one off purchases and services that don't matter if they…

I run a business with thousands of users. When payment fails, I try again eight times over two weeks. You get notices each time. There's no surprises. There's no excuse for shutting someone off abruptly for credit card issues.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#259

Post by theshrike79 »

silisili wrote: post #36

Seeing all these horror stories, I can't help but wonder when people are going to switch back to on prem, or at least some hybrid. It's pretty apparent we can't trust cloud providers...

On-prem requires a good half-dozen people just for managing the servers and doing on-call rotation.

For a 10-20 person company that's a huge expense with not that much ROI.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#260

Post by helsinkiandrew »

nosianu wrote: post #167
helsinkiandrew wrote: post #158

Earlier quoted context omitted.

Hang on - what would you do if you were running a business with hundreds of thousands of users and someone doesn't pay? Was it an accident? has the customer gone bust, or died, or are they trying to defraud you and get a month or two of free service. Credit cards are exceptionally finicky, very susceptible to fraud, and payment can often fail. They're great for one off purchases and services that don't matter if they…

Your rhetorical questions don't match OPs story. According to OP they did pay, manually, not even by credit card. > however we went ahead and made a manual payment covering all the outstanding amount + extra (the entire last paragraph of OPs text has those and more details) I suggest not using and then arguing with and against fantasized stuff when we are talking about a concrete problem for which we have a concrete…

> Your rhetorical questions don't match OPs story. According to OP they did pay, manually, not even by credit card.

That's fair enough - somehow I completely misread/understood that part of the post.

But my main point was that you should never be using credit cards for services that you rely on - particularly when billing/payment is automated and hidden from you. Credit card payments and checks fail regularly (10%+ for recurring payment charges) for legitimate (you don't have funds) and less legitimate reasons (a banks AI system thinking the charge looks fraudulent due to something outside of your account etc). Vendors are likely to stop the service as soon as they can get away with so they don't loose money.

Post reply on HN