Earlier quoted context omitted.
Do we see much of this for AWS? Asking for a friend...
AWS will email you for months and months about billing issues before they terminate anything
Tell HN: Google Cloud suspended our production projects at 1am on Saturday
391–400 of 531 posts
Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday
#392Earlier quoted context omitted.
Google has some good tech and then most part of the cloud is just years behind. Want an example: you can't configure the gcp docker registry [EDIT: artifact registry] to cleanup images automatically (after a number of days or number of images or anything). This is total standard in AWS and the ticket for this on GCP has been open for many years now.
Isn’t GCR backed by a GCS bucket in your own project? Can’t you configure object lifecycle management on said GCS bucket?
That is not backed by GCS buckets - if it were, I'm sure that I would have found this solution while searching the web.
But even if: I would like to e.g. keep the latest 100 images always. I doubt this would be possible with a simply GCS polcy without writing custom code or something like a cron job.
Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday
#393Earlier quoted context omitted.
It's like looking both ways when crossing the road despite the light telling you that you can go. Sure you have right of way and if a car hits you, it's their fault. But at the end of the day, you are the one getting hit by the car. Google is at fault for how they handled it, but the parents comment is still good advice for anyone reading this thread.
How are you supposed to trust a company who told you they wouldn't screw you because of their own error, then screwed you anyway? Having a relationship with them doesn't matter if they still can't be trusted to do business. The type of payment method you use is not a factor that plays into whether you can trust your vendor.
I mean, it's an error.
Sure, it's infuriating, and in worst-case could financially harm (or cripple) your business. But Google didn't intend for it to happen. That makes no sense from a business perspective.
So, as others here suggest, you have to plan around the fact that mistakes like this can happen, especially when you're small.
Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday
#394Earlier quoted context omitted.
This is a fundamentally unserious point unless you want to add more qualifiers. I’m pointing to this as evidence https://cloud.google.com/customers I feel like a lot of the problems that people run into with GCP are avoided with two main components. Number one you should look seriously at the “enterprise” designation for different products and services if you are looking for long term stability and guarantees about t…
or just cluster over several other providers and make sure to test failover the main problem with azure/aws/gcp is that their parent companies will track everything you and others do, and then close every related thing because they have enough market share to not give a fuck about you. There are more than enough blog posts of enterprise customers running into random issues with the big thing, buying "big three, promi…
I can assure you from experience that most large customers do not. For either major cloud, most large customers will have adopted proprietary services like AWS Redshift or GCP's Cloud Bigtable or Bigquery. None of these have anything like the possibility of a "rapid migration".
Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday
#395A few years ago I was working at a startup and we had just started moving some stuff over from AWS to GCP. Things were going pretty good with GCP until one day they pulled the plug on everything with no explanation. It turns out that our company credit card details had been fraudulently used without our knowledge. The criminal had decided to use the card to buy Google Ad Words or something like that - this is the sam…
I hear way too many stories like this about Google products. Whether it's Youtube, AdWords, GCP, or anything else - you get blackholed by them and there's zero recourse. I just can't trust Google for something like running production services.
Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday
#396A suggestion for engineers that lack business experience. As soon as your project grows beyond the non-hobby scale and starts making real money, invest some time to reach out to company providing the services that keep it alive and find an account manager that will take care of you. We're a Google customer and of course we had our fair share of issues (btw: it's the same with Azure, based on our experience), and we a…
I worked for couple of really big enterprises, one of which had such a high commitment that Azure sent two engineers to sit with the teams working on their cloud. Highest level Enterprise support .. and yet for actual non-obvious problems it took them a month to reply with: can't help you with this, you must be doing it wrong.
This was techsupport tho, I'm sure the billing for multi-million commitments was just fine and dandy including wining and dining.
One other company was a big AWS user (and some Office365 so there were Azure pitches too) and GCP tried to get in with the business as well. When we went to their offices the level of patronizing smug that came from their reps was astonishing, so off-putting that I refused all further interaction with them as I wanted to barf.
Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday
#397Cloud computing platforms are a bit of a double edged sword. I find it kind of ironic that big companies are moving their stuff to those services but they often have to rely on some consultancy firm that can deal with these problems. Congratulations, you had a problem and now you have two.
They went from: (Remember, companies are people!)
We have to make choices about big expensive things. - 0 people to blame besides them. Vendors / Consultants make choices about bigger and more expensive things for them - 1 person to blame besides them. Consultants + "The Cloud" to blame - 2 people to blame besides them.
It is a feature to have many people to blame, in these type of environments.
Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday
#398Also: If you're on AWS/GCP and only backup to S3/GCS, you're doing it wrong. All your backups to setup a new system (even if it takes time) need to be with a different provider.
> "We're fully redundant across AZs, regions, and even cloud providers!" crows the engineer with a single corporate credit card backing the entire house of cards.
https://twitter.com/QuinnyPig/status/1288275701389389825
You could backup to a different account, possibly in a different region, with the same provider, as long as the other account has a different payment method. Using one provider would in most cases reduce the amount of effort to get back up and running.
The key piece would be how DNS is structured and hosted/billed.
Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday
#399A suggestion for engineers that lack business experience. As soon as your project grows beyond the non-hobby scale and starts making real money, invest some time to reach out to company providing the services that keep it alive and find an account manager that will take care of you. We're a Google customer and of course we had our fair share of issues (btw: it's the same with Azure, based on our experience), and we a…
Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday
#400Earlier quoted context omitted.
You have a person to reach out to if things go pear shaped, and N(weeks|months) to sort out payments issues instead of having everything get suspended by robots.
What about telling those robots not to suspend everything and wait for N(weeks|months) or human intervention?
2. Unlike a small hosting company, Google is so big it will suffer essentially no financial consequences if their robots make a wrong decision for credit card customers a few times per month.
So there is no cost to Google for leaving the system as is, and a high cost for your solution. The decision is a no-brainer from their point of view.