Live data from Hacker News

Tell HN: Google Cloud suspended our production projects at 1am on Saturday

news.ycombinator.com

391–400 of 531 posts

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#391

Earlier quoted context omitted.

Do we see much of this for AWS? Asking for a friend...

AWS will email you for months and months about billing issues before they terminate anything

Indeed. My company forgot to warn the the credit card had been updated, emails were being sent to the work email of an employee that left... Took 4 months before they started turning off stuff, and even that was quite gradual, just enough to draw attention

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#392
post #373

Earlier quoted context omitted.

Google has some good tech and then most part of the cloud is just years behind. Want an example: you can't configure the gcp docker registry [EDIT: artifact registry] to cleanup images automatically (after a number of days or number of images or anything). This is total standard in AWS and the ticket for this on GCP has been open for many years now.

Isn’t GCR backed by a GCS bucket in your own project? Can’t you configure object lifecycle management on said GCS bucket?

GCR is deprecated and I'm using Artifact Registry. (sorry for the confusion)

That is not backed by GCS buckets - if it were, I'm sure that I would have found this solution while searching the web.

But even if: I would like to e.g. keep the latest 100 images always. I doubt this would be possible with a simply GCS polcy without writing custom code or something like a cron job.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#393
post #163

Earlier quoted context omitted.

It's like looking both ways when crossing the road despite the light telling you that you can go. Sure you have right of way and if a car hits you, it's their fault. But at the end of the day, you are the one getting hit by the car. Google is at fault for how they handled it, but the parents comment is still good advice for anyone reading this thread.

How are you supposed to trust a company who told you they wouldn't screw you because of their own error, then screwed you anyway? Having a relationship with them doesn't matter if they still can't be trusted to do business. The type of payment method you use is not a factor that plays into whether you can trust your vendor.

>How are you supposed to trust a company who told you they wouldn't screw you because of their own error, then screwed you anyway?

I mean, it's an error.

Sure, it's infuriating, and in worst-case could financially harm (or cripple) your business. But Google didn't intend for it to happen. That makes no sense from a business perspective.

So, as others here suggest, you have to plan around the fact that mistakes like this can happen, especially when you're small.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#394
post #350

Earlier quoted context omitted.

This is a fundamentally unserious point unless you want to add more qualifiers. I’m pointing to this as evidence https://cloud.google.com/customers I feel like a lot of the problems that people run into with GCP are avoided with two main components. Number one you should look seriously at the “enterprise” designation for different products and services if you are looking for long term stability and guarantees about t…

or just cluster over several other providers and make sure to test failover the main problem with azure/aws/gcp is that their parent companies will track everything you and others do, and then close every related thing because they have enough market share to not give a fuck about you. There are more than enough blog posts of enterprise customers running into random issues with the big thing, buying "big three, promi…

> I'd suspect most of those customers have rapid migration plans written up.

I can assure you from experience that most large customers do not. For either major cloud, most large customers will have adopted proprietary services like AWS Redshift or GCP's Cloud Bigtable or Bigquery. None of these have anything like the possibility of a "rapid migration".

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#395
post #124

A few years ago I was working at a startup and we had just started moving some stuff over from AWS to GCP. Things were going pretty good with GCP until one day they pulled the plug on everything with no explanation. It turns out that our company credit card details had been fraudulently used without our knowledge. The criminal had decided to use the card to buy Google Ad Words or something like that - this is the sam…

I hear way too many stories like this about Google products. Whether it's Youtube, AdWords, GCP, or anything else - you get blackholed by them and there's zero recourse. I just can't trust Google for something like running production services.

Several years back I wanted to try my hand at making an Android app and putting it in the Play Store. I knew ahead of time from reading all the horror stories that it's best to make a second dedicated developer Google account because once you turn an account into a developer account you're basically painting a target on your back to be randomly deleted by the powers that be. Additionally, not sure if this is still the case but at the time once you converted to a developer account you could never change your account's country again.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#396

A suggestion for engineers that lack business experience. As soon as your project grows beyond the non-hobby scale and starts making real money, invest some time to reach out to company providing the services that keep it alive and find an account manager that will take care of you. We're a Google customer and of course we had our fair share of issues (btw: it's the same with Azure, based on our experience), and we a…

I found the Azure reps to be completely useless for any of the actual problems.

I worked for couple of really big enterprises, one of which had such a high commitment that Azure sent two engineers to sit with the teams working on their cloud. Highest level Enterprise support .. and yet for actual non-obvious problems it took them a month to reply with: can't help you with this, you must be doing it wrong.

This was techsupport tho, I'm sure the billing for multi-million commitments was just fine and dandy including wining and dining.

One other company was a big AWS user (and some Office365 so there were Azure pitches too) and GCP tried to get in with the business as well. When we went to their offices the level of patronizing smug that came from their reps was astonishing, so off-putting that I refused all further interaction with them as I wanted to barf.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#397
post #71

Cloud computing platforms are a bit of a double edged sword. I find it kind of ironic that big companies are moving their stuff to those services but they often have to rely on some consultancy firm that can deal with these problems. Congratulations, you had a problem and now you have two.

That's a feature not a bug.

They went from: (Remember, companies are people!)

We have to make choices about big expensive things. - 0 people to blame besides them. Vendors / Consultants make choices about bigger and more expensive things for them - 1 person to blame besides them. Consultants + "The Cloud" to blame - 2 people to blame besides them.

It is a feature to have many people to blame, in these type of environments.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#398

Also: If you're on AWS/GCP and only backup to S3/GCS, you're doing it wrong. All your backups to setup a new system (even if it takes time) need to be with a different provider.

You don't need to use a separate provider to address the problem here, where using a separate provider may not address the problem at all:

> "We're fully redundant across AZs, regions, and even cloud providers!" crows the engineer with a single corporate credit card backing the entire house of cards.

https://twitter.com/QuinnyPig/status/1288275701389389825

You could backup to a different account, possibly in a different region, with the same provider, as long as the other account has a different payment method. Using one provider would in most cases reduce the amount of effort to get back up and running.

The key piece would be how DNS is structured and hosted/billed.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#399

A suggestion for engineers that lack business experience. As soon as your project grows beyond the non-hobby scale and starts making real money, invest some time to reach out to company providing the services that keep it alive and find an account manager that will take care of you. We're a Google customer and of course we had our fair share of issues (btw: it's the same with Azure, based on our experience), and we a…

This is sound advice in my experience. The last company I worked with who was utilizing Google Cloud had a great experience, an account manager and a reseller who offered training as well.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#400

Earlier quoted context omitted.

You have a person to reach out to if things go pear shaped, and N(weeks|months) to sort out payments issues instead of having everything get suspended by robots.

What about telling those robots not to suspend everything and wait for N(weeks|months) or human intervention?

1. Because they'd probably have to hire several hundred humans and give them benefits. It's much cheaper just to trust the robots.

2. Unlike a small hosting company, Google is so big it will suffer essentially no financial consequences if their robots make a wrong decision for credit card customers a few times per month.

So there is no cost to Google for leaving the system as is, and a high cost for your solution. The decision is a no-brainer from their point of view.

Post reply on HN