Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

cloud.google.com

Post reply on HN
Print view View original

101–110 of 191 posts

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#101

Post by ElevenLathe »

dragandj wrote: post #67
anonymousab wrote: post #59

Earlier quoted context omitted.

I remember going out to dinner, years ago, with a fairly senior AWS billing engineer. An acquaintance of a coworker. He looked completely surprised when I asked about runaway billing and why there wasn't any simple options to cap a given resource to prevent those cases. His response was that they didn't build that because none of their customers wanted anything like that, as far as he was aware.

Yeah, right. Capping a resource, such a wild idea. Of course they won't implement it for the same reason bar owners don't put a cap on drinks.

Aren't bars actually required to cap drinks? It's usually phrased as having to refuse serving if you're visibly drunk, but still effectively a cap. That said, a big cloud bill doesn't make you intoxicated. The more I examine this analogy, the less it makes sense.

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#102

Post by rkoots »

dragandj wrote: post #67
anonymousab wrote: post #59

Earlier quoted context omitted.

I remember going out to dinner, years ago, with a fairly senior AWS billing engineer. An acquaintance of a coworker. He looked completely surprised when I asked about runaway billing and why there wasn't any simple options to cap a given resource to prevent those cases. His response was that they didn't build that because none of their customers wanted anything like that, as far as he was aware.

Yeah, right. Capping a resource, such a wild idea. Of course they won't implement it for the same reason bar owners don't put a cap on drinks.

[dead]

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#103

Post by Workaccount2 »

ninetyninenine wrote: post #25

Im tired of using AI in cloud services. I want user friendly locally owned AI hardware. Right now nothing is consumer friendly. I can’t get a packaged deal of some locally running ChatGPT quality UI or voice command system in an all in one package. Like what Macs did for PCs I want the same for AI.

From the most unexpected place (but maybe expected if you believed they were paying attention)

Maxsun is releasing a a 48GB dual Intel Arc Pro B60 GPU. It's expected to cost ~$1000.

So for around $4k you should be able to build an 8 core 192GB local AI system, which would allow you to locally run some decent models.

This also assumes the community builds an intel workflow, but given how greedy Nvidia is with vram, it seems poised to be a hit.

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#104

Post by gabe_monroy »

ashishb wrote: post #54

I love Google Cloud Run and highly recommend it as the best option[1]. The Cloud Run GPU, however is not something I can recommend. It is not cost effective (instance based billing is expensive as opposed to request based billing), GPU choices are limited, and the general loading/unloading of model (gigabytes) from GPU memory makes it slow to be used as server less. Once you compare the numbers it is better to use a…

google vp here: we appreciate the feedback! i generally agree that if you have a strong understanding of your static capacity needs, pre-provisioning VMs is likely to be more cost efficient with today's pricing. cloud run GPUs are ideal for more bursty workloads -- maybe a new AI app that doesn't yet have PMF, where you really need that scale-to-zero + fast start for more sparse traffic patterns.

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#105

Post by shiftyck »

AChampaign wrote: post #83
icedchai wrote: post #81

Earlier quoted context omitted.

Cloud Run is a great service. I find it much easier to work with than AWS's equivalent (ECS/Fargate.)

I think Lambda is more or less the AWS equivalent.

Eh idk Cloud Run is much better suited to long running instances than Lambda. You would use Cloud Functions for those types of workloads in GCP.

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#106

Post by gabe_monroy »

icedchai wrote: post #81
ashishb wrote: post #54

I love Google Cloud Run and highly recommend it as the best option[1]. The Cloud Run GPU, however is not something I can recommend. It is not cost effective (instance based billing is expensive as opposed to request based billing), GPU choices are limited, and the general loading/unloading of model (gigabytes) from GPU memory makes it slow to be used as server less. Once you compare the numbers it is better to use a…

Cloud Run is a great service. I find it much easier to work with than AWS's equivalent (ECS/Fargate.)

i am biased, but i agree :)

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#107

Post by gabe_monroy »

montebicyclelo wrote: post #28
isoprophlex wrote: post #6

All the cruft of a big cloud provider, AND the joy of uncapped yolo billing that has the potential to drain your creditcard overnight. No thanks, I'll personally stick with Modal and vast.ai

Not providing a cap on spending is a major flaw of GCP for individuals / small projects. With Cloud Run, AFAIK, spending can effectively be capped by: limiting concurrency, plus limiting the max number of instances it can scale to. (But this is not as good as GCP having a proper cap.)

Heard on this feedback. While not quite a hard cap, I'd also point to https://cloud.google.com/billing/docs/how-to/budgets which many customers are having success with for this use case.

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#108

Post by weberer »

shiftyck wrote: post #105
AChampaign wrote: post #83

Earlier quoted context omitted.

I think Lambda is more or less the AWS equivalent.

Eh idk Cloud Run is much better suited to long running instances than Lambda. You would use Cloud Functions for those types of workloads in GCP.

For those who don't know, AWS Lambda functions have a hard limit of 15 minutes.

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#110

Post by yarri »

montebicyclelo wrote: post #28
isoprophlex wrote: post #6

All the cruft of a big cloud provider, AND the joy of uncapped yolo billing that has the potential to drain your creditcard overnight. No thanks, I'll personally stick with Modal and vast.ai

Not providing a cap on spending is a major flaw of GCP for individuals / small projects. With Cloud Run, AFAIK, spending can effectively be capped by: limiting concurrency, plus limiting the max number of instances it can scale to. (But this is not as good as GCP having a proper cap.)

[edit - Gabe responded]. See this Cloud Run spending cap recommendation [0] to disable billing, which potentially irreversibly deletes resources but does cap spend!

[0] https://cloud.google.com/billing/docs/how-to/disable-billing...

Post reply on HN