Live data from Hacker News

Tell HN: Google Cloud suspended our production projects at 1am on Saturday

news.ycombinator.com

421–430 of 531 posts

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#421
post #373

Earlier quoted context omitted.

Isn’t GCR backed by a GCS bucket in your own project? Can’t you configure object lifecycle management on said GCS bucket?

GCR is deprecated and I'm using Artifact Registry. (sorry for the confusion) That is not backed by GCS buckets - if it were, I'm sure that I would have found this solution while searching the web. But even if: I would like to e.g. keep the latest 100 images always. I doubt this would be possible with a simply GCS polcy without writing custom code or something like a cron job.

> But even if: I would like to e.g. keep the latest 100 images always. I doubt this would be possible with a simply GCS polcy without writing custom code or something like a cron job.

https://cloud.google.com/storage/docs/lifecycle#numberofnewe...

> GCR is deprecated and I'm using Artifact Registry. (sorry for the confusion)

You're right that this functionality seems to not be built into the Artifact Registry backend (and that's weird†), but it does still exist: see https://github.com/GoogleCloudPlatform/gcr-cleaner (found linked from https://cloud.google.com/artifact-registry/docs/docker/manag...), and specifically the `keep` flag for it.

Note also how the project is hosted under the GoogleCloudPlatform GH org. To me, when I see GCP projects that are set up like this (in the GH org but disclaimed as "not official" in the GCP docs), this suggests that Google engineers built it knowing it's a pain point; and those engineers will support it to the best of their ability in the capacity of being maintainers of this open-source project; but Google as a company don't want to officially support it (yet), and so your GCP support contract won't get you any business-level support for it.

It's sort of like how, in Postgres, there is code which is maintained by the Postgres maintainers, but which lives under contrib/ as an extension. It's essentially a lability-waiver for that component.

---

† I do have a guess as to why Google do things this way. At least where dev tools are concerned, Google seems to eschew the usual distributed-systems architecture for long-running jobs (of having a thin API client binary that submits jobs to a cloud-side control-plane daemon, which then drives the job forward, and which can then be polled/subscribed for job status by said client.) Rather, Google seemingly have a philosophy of designing local fat clients that reach into the cloud to drive backend processes as the "control node" for those processes. The Cloud Dataflow (⇒ Apache Beam) architecture is designed this way, for example. I believe it's the reason that the Google Cloud SDK ships with so many binaries — there are a lot of fat clients in there that actually drive logic, rather than just sending messages to daemons that drive the logic.

And, presuming developers are issued good workstations, I can see the advantages of this architecture. A local control node synchronously knows its own status, rather than having to poll for it; a local control node can use local resources (like how Cloud Dataflow can consume and produce local files on the ends of the pipeline with the same streaming efficiency as a regular CLI text-processing shell command); and an operation started by mistake, with local control, can be cancelled by just ctrl+c-ing the control process.

Depending on how you design the client, it can also "mandate manual usage" — i.e. ensure that the developer is interactively running the process for the process to proceed, and therefore that said dev is available in case anything goes wrong. (I've personally dropped [async daemon-driven] Continuous Deployment, in favor of this sort of "synchronous dev-workstation-driven deployment.")

I wish someone at Google would write up a paper on this philosophy; it's pretty clearly implicit in a lot of their work, but I've never seen it mentioned explicitly anywhere. (Maybe it's just one dev-tools lead who has influenced a lot of these projects, doing what they think is "obvious"?)

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#422

Murfie died because of Google cloud costs. It's a bit of a long story, but from what I was able to piece together, a Google cloud service provider recommended some changes to the way the storage was done to save money. Mostly, because of those changes they got hit by a surprise $30,000 bill for data egress. It was basically the last straw for a company that wasn't profitable due mostly to cloud costs. It killed the c…

More details that I found about what Murfie is, and the rescue operation: https://www.theverge.com/2020/2/5/21121594/crossies-murfie-m...

What a fascinating concept.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#423
post #157

Earlier quoted context omitted.

Google is just horrible, when it comes to any form of support, paid or not. They make paypal look helpful by comparison. I really can't understand why anyone would use any of their services. Support counts. It's one of the first things I look at, when evaluating any external partner. What does it cost, how long does it take for it to happen, etc? Whatever anyone else says in this thread, that's what you need to deter…

> Google is just horrible, when it comes to any form of support, paid or not. They make paypal look helpful by comparison. We have been running production on Google Cloud for six years. We are not a large customer: our spend is in the low five figures. We pay for a middle-tier support plan and support has been excellent during this entire period. We very rarely need to take advantage of it, since the underlying syste…

Good for you, but if you used the service only "very rarely," I don't think you've experienced the problems that are being discussed here. I had signed up for paid support for a project a while back and remember having so much trouble even paying for it (some trouble with invoices), let alone get any value out of it, that I ended up canceling it.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#424

Earlier quoted context omitted.

> wait until the car is basically stopped you mean like an email literally saying they acknowledge the situation and won't suspend your account?

I think an email from support is more like "confirmation that they checked whether the car has seen me". I'm not sure how successfully you've dealt with support with companies, but I have almost never been able to rely on the promises of a support agent, as they are simply not the ones actually making the decisions (in this case on whether your account gets suspended or not).

sounds more like you want a written and signed letter from the driver that they don't intend to restart the car, co-signed by their spouse

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#425

This is why my old hosting company changed its auto disconnection code to work at 11am on a Monday morning, never at a weekend. That way it's the bill-payers that notice and get in touch, not the poor tech on the out-of-hours rota. Also, it only firewalled the customer's public IPs from the Internet - it didn't stop servers. That way when the account is settled there was often nothing for the techs to do. Also it was…

That's how hetzner handles late payments and only does this after 2 warning emails.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#426
post #6

I wonder what it would take to bring a culture change to Google so they can finally start having some empathy for customers and stop the casual cruelty. A new CEO?

They should be freaking broken:

1. Ad business should be one thing 2. Other products should be different companies: Gmail, GCS, Domains, etc etc. They should not be "cushioned" by Ad business revenue.

That way they will play an even playing field with similar companies. And I think Amazon/AWS should have the same split.

If that happens, these companies will have to differentiate to their customers with non-crappy customer support.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#427

A suggestion for engineers that lack business experience. As soon as your project grows beyond the non-hobby scale and starts making real money, invest some time to reach out to company providing the services that keep it alive and find an account manager that will take care of you. We're a Google customer and of course we had our fair share of issues (btw: it's the same with Azure, based on our experience), and we a…

I have dealt with Google and AWS under 4 different companies that spent between low double digit to high triple digit a month. General view: * AWS - Great customer service. Steak house level customer service when moved from cc billing to contract billing. * GCP - terrible service regardless of spend and regardless of dynamic or contract billing. AM do not know products. AM's are equivalent of BOA branch employees tha…

Same experiences here. AWS was awesome. They did crazy things I wouldn’t expect like spend days in person and join us at equinix colo to get the direct connect working. We didn’t even spend that much relatively, under $100k/mo.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#428

Earlier quoted context omitted.

Never mess with Ad Words, that makes the Alphabet Beast angry every time. I've heard the exact same Ad Words story a few times now, someone used their CC or had at one point in the distant past used with Ad Words and the system flagged it. Everything gone with no warning, no explanation and no way to complain. Even getting to the front page of HN didn't work =)

But the person you're replying to didn't use ad words! Someone stole his CC number and used adwords.

Alphabet doesn't care, as far as they are concerned, someone messed with their Holy Money Machine and doesn't deserve to exist.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#430
post #237

Earlier quoted context omitted.

The pendulum will swing over cost, not over outages like these, because the cost challenges are real, while these outages are rare. It won't go all back to on prem because cloud providers will slash their margins once they struggle to maintain growth, and doing so they'll diminish the advantages of on-prem/hybrid setups for a lot of people, and so we'll find some equilibrium or other. I'd expect more emphasis on hybr…

Full on prem is a pretty bad for a large company because your company now depends on a single monopolistic service provider that is run as a cost center (ie: the department managing your on prem stuff). Eventually that leads to a very bad engineer experience which leads to a massive opportunity cost for any tech company. It works for a while and then over time becomes more and more dysfunctional. Hybrid at least incl…

Most places de facto depend on a single provider that becomes a cost center when using cloud providers too. While you can shop around, in effect it tends to get entrenched and get more and more dysfunctional as people try to exploit characteristics of your approvals process for using resources from your chosen provider. For a big company, you need a team to manage these resources - I've lost count of the number of recruiters who has contacted me just this year for roles involving management or architecture of cloud strategy etc. because they're building whole supporting organisations around a specific paradigm for hosting rather than building an organisation responsible for understanding and providing the best possible substrate for their applications.

I'm all for hybrid solutions, but mostly because it means you usually end up being able to cut the cost of on prem, colocation or managed servers even more relative to cloud setups because you can increase the utilisation rate (e.g. run your base load on cheap Hetzner servers but being able to spin up EC2 instances on short notice to take spices or handle failures up to and including all of Hetzner falling off the face of the earth). Most larger managed providers today also offers cloud services and many also offers colo services, so you can often easily mix and match as long as you're conscious off egress pricing which is often the biggest barrier to such setups today (especially if the big cloud providers are in the mix, though it's better than it was).

That said, I prefer not putting my eggs in one basket, and instead going the direction of putting in place layers to treat a multi-provider setup as one. In the past I've e.g. done zero downtime migrations between AWS->GCP->Hetzner that way, and also had systems split between actual on prem (as in a data room at our office), multiple colo's, dedicated servers at Hetzner, and a couple of VM providers where everything was transparent to our engineers (they didn't need to know on what continent a service ran unless doing performance optimisation or reliability engineering). We prepped to tie in AWS resources too, but it never become cost effective for that company to do so vs. the prices of the other providers used - it would have been if we had sudden extreme load spikes, but it was a business where traffic was linked closely to physical capacity at restaurants and nightclubs, and so traffic was very predictable.

Post reply on HN