Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tell HN: Google Cloud suspended our production projects at 1am on Saturday

news.ycombinator.com

Post reply on HN
Print view View original

291–300 of 531 posts

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#291

Post by PeterisP »

philliphaydon wrote: post #122
KingOfCoders wrote: post #29

Also: If you're on AWS/GCP and only backup to S3/GCS, you're doing it wrong. All your backups to setup a new system (even if it takes time) need to be with a different provider.

> If you're on AWS/GCP and only backup to S3/GCS, you're doing it wrong. If that’s how you feel then you should be applying that logic to any provider. Even if you rent rack space in a data center. Because unless you own everything through and through. This could happen anywhere.

Of course you should be applying that logic if you rent rack space in a data center - your backups should definitely be outside of that data center somewhere else.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#292

Post by shyn3 »

black_puppydog wrote: post #266
Nebasuke wrote: post #252

Earlier quoted context omitted.

I got hit by a car in a situation similar to this. They made eye contact (I thought), were slowing down, and still hit me anyway by accelerating, despite me having the right of way on a bike. Turns out it was the first time they used the car, 1 minute into the drive and their focus was somewhere else. I would say the advice here is, wait until the car is basically stopped before you cross the street rather than trust…

> wait until the car is basically stopped you mean like an email literally saying they acknowledge the situation and won't suspend your account?

In my experience, when dealing with vendors having an account manager can fix these kind of problems and prevent them.

Most places outsource their support to global teams don't have access. If you have an AM , in many places, they receive the same notice you receive, also they will personally send it and address it to you with a reminder. They also have a lot more sway as the sales side of every company has the power so they can get around all the process issues.

As the other commenter posted above, if you can have an account manager where you have critical services, it will provide you safety at the cost of a higher spend or minimum spending amounts.

Edit: Had about $1mm in GCP spend and couldn't open a ticket to get windows running properly. It took a week to find the AM and a day after he had it fixed.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#293

Post by Cederfjard »

intellectronica wrote: post #254
TomSwirly wrote: post #197

Earlier quoted context omitted.

> Smaller companies sometimes don't understand that [...] you should really make the extra effort Let me get this straight. This company does everything they were supposed to, paid on time, and Google's own error cuts them off, and you are blaming them?!?

You don't know that. Maybe that's the case (I'm sure it happens sometimes), but most likely it isn't.

I think it’s fair not to take any given story at face value, without ”receipts” to back it up. You however seem to me to go in the other direction and barely want to give OP the benefit of the doubt. What details would you like to see in order to be convinced?

From your profile I clicked through to your LinkedIn via your website, and I think it would’ve been becoming of you to disclose that you were a senior engineering manager at Google Cloud up until earlier this year, given the context of the discussion and your stance on it.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#295

Post by 7952 »

emsy wrote: post #16
bilalq wrote: post #6

I wonder what it would take to bring a culture change to Google so they can finally start having some empathy for customers and stop the casual cruelty. A new CEO?

Regulation, full stop. The reason Big tech makes so much profits is because they have no accountability and at their scale, social shaming doesn’t work.

I wonder if you could require that some data be made accessible to the user. Like transaction logs and actions relating to those transactions. So that you have direct incontrovertible evidennce of the internal state. That could then be used by third party support, or even courts.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#296

Post by lupire »

belorn wrote: post #227
rglullis wrote: post #100

Earlier quoted context omitted.

I did not argue that regulation is unnecessary or bad. The argument is that regulation of Big Industries does not bring you better support nor makes the companies more focused in treating customers as individuals.

It all depend on the reason and target of the regulation. Patients might have historically liked doctors more when they prescribed cocaine, alcohol and morphine for most ailments, but then the target of modern regulation is not customer short-term happiness. It is health. The issue of regulating big industries and big tech in particular is what outcomes we want to avoid and how a healthy market can be created where o…

You are arguing against yourself. Now, only drug addicts who ignore laws can get painkillers.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#297

Post by manishsharan »

smileybarry wrote: post #244
jeroenhd wrote: post #55

Earlier quoted context omitted.

Backblaze is very cheap and has several "don't overload our system" clauses in their terms and conditions. They're cheap and reliable, that generally comes at the risk of early termination. I don't think you can have a cheap, reliable, unlimited service; pick two at most.

They probably meant their B2 storage service, which is like S3 and bills for storage, egress, some operations, etc. That’s one service I wouldn’t expect to have some cancellation for overload in place. You’re using a token anyway, why not just rate-limit you instead?

Blackblaze B2 is supposed to be AWS S3 competitor. S3 is simply unbreakable. If Blackblaze intends to compete with S3, they can't complain about load stress on their B2.And they can't rate limit either.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#298

Post by paulgb »

theshrike79 wrote: post #257
hazbo wrote: post #124

A few years ago I was working at a startup and we had just started moving some stuff over from AWS to GCP. Things were going pretty good with GCP until one day they pulled the plug on everything with no explanation. It turns out that our company credit card details had been fraudulently used without our knowledge. The criminal had decided to use the card to buy Google Ad Words or something like that - this is the sam…

Never mess with Ad Words, that makes the Alphabet Beast angry every time. I've heard the exact same Ad Words story a few times now, someone used their CC or had at one point in the distant past used with Ad Words and the system flagged it. Everything gone with no warning, no explanation and no way to complain. Even getting to the front page of HN didn't work =)

We were lucky enough that getting to the front page of HN did work for us[1], but it's worth noting that it doesn't even have to be your card! Simply having a card from an issuer (in our case a legit, YC-backed, US-based startup bank) that got flagged was enough to have our account suspended and appeals denied.

[1] https://news.ycombinator.com/item?id=32237445

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#299

Post by intellectronica »

Cederfjard wrote: post #293
intellectronica wrote: post #254

Earlier quoted context omitted.

You don't know that. Maybe that's the case (I'm sure it happens sometimes), but most likely it isn't.

I think it’s fair not to take any given story at face value, without ”receipts” to back it up. You however seem to me to go in the other direction and barely want to give OP the benefit of the doubt. What details would you like to see in order to be convinced? From your profile I clicked through to your LinkedIn via your website, and I think it would’ve been becoming of you to disclose that you were a senior engineer…

And now I work for the competition :) In any case, I know enough about how cloud services work to be able to comment intelligently, and don't have any interest in Google (except for some stock I hold indirectly via ETFs). This is all just common sense plus what I know from having worked for several cloud services providers.
Post reply on HN