Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

cloud.google.com

Post reply on HN
Print view View original

81–90 of 191 posts

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#81

Post by icedchai »

ashishb wrote: post #54

I love Google Cloud Run and highly recommend it as the best option[1]. The Cloud Run GPU, however is not something I can recommend. It is not cost effective (instance based billing is expensive as opposed to request based billing), GPU choices are limited, and the general loading/unloading of model (gigabytes) from GPU memory makes it slow to be used as server less. Once you compare the numbers it is better to use a…

Cloud Run is a great service. I find it much easier to work with than AWS's equivalent (ECS/Fargate.)

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#82

Post by dig1 »

ashishb wrote: post #54

I love Google Cloud Run and highly recommend it as the best option[1]. The Cloud Run GPU, however is not something I can recommend. It is not cost effective (instance based billing is expensive as opposed to request based billing), GPU choices are limited, and the general loading/unloading of model (gigabytes) from GPU memory makes it slow to be used as server less. Once you compare the numbers it is better to use a…

> I love Google Cloud Run and highly recommend it as the best option

I'd love to see the numbers for Cloud Run. It's nice for toy projects, but it's a money sink for anything serious, at least from my experience. On one project, we had a long-standing issue with G regarding autoscaling - scaling to zero sounds nice on paper, but they will not mention you the warmup phases where CR can spin up multiple containers for a single request and keep them for a while. And good luck hunting for unexplainedly running containers when there are no apparent cpu or network uses (G will happily charge you for this).

Additionally, startup is often abysmal with Java and Python projects (although it might perform better with Go/C++/Rust projects, but I don't have experience running those on CR).

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#83

Post by AChampaign »

icedchai wrote: post #81
ashishb wrote: post #54

I love Google Cloud Run and highly recommend it as the best option[1]. The Cloud Run GPU, however is not something I can recommend. It is not cost effective (instance based billing is expensive as opposed to request based billing), GPU choices are limited, and the general loading/unloading of model (gigabytes) from GPU memory makes it slow to be used as server less. Once you compare the numbers it is better to use a…

Cloud Run is a great service. I find it much easier to work with than AWS's equivalent (ECS/Fargate.)

I think Lambda is more or less the AWS equivalent.

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#84

Post by Gracana »

petesergeant wrote: post #41
ninetyninenine wrote: post #25

Im tired of using AI in cloud services. I want user friendly locally owned AI hardware. Right now nothing is consumer friendly. I can’t get a packaged deal of some locally running ChatGPT quality UI or voice command system in an all in one package. Like what Macs did for PCs I want the same for AI.

Hoping the DGX Spark will deliver on this

It will not. 273GB/s memory bandwidth is not enough.

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#85

Post by ZeroCool2u »

AChampaign wrote: post #83
icedchai wrote: post #81

Earlier quoted context omitted.

Cloud Run is a great service. I find it much easier to work with than AWS's equivalent (ECS/Fargate.)

I think Lambda is more or less the AWS equivalent.

I think Cloud Run Functions would be the direct equivalent to Lambda.

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#86

Post by coredog64 »

mwest217 wrote: post #69
anonymousab wrote: post #59

Earlier quoted context omitted.

I remember going out to dinner, years ago, with a fairly senior AWS billing engineer. An acquaintance of a coworker. He looked completely surprised when I asked about runaway billing and why there wasn't any simple options to cap a given resource to prevent those cases. His response was that they didn't build that because none of their customers wanted anything like that, as far as he was aware.

Disclaimer: I work at Google but not on cloud. Opinions my own. I think the reason this doesn’t get prioritized is that large customers don’t actually want a “stop serving if I pass this limit” amount. If there’s a spike in traffic, they probably would rather pay the money to serve it. The customers that would want this feature are small-dollar customers, and from an economic perspective it makes less sense to priori…

As noted above, there is enough value here such that AWS implemented this several years ago. Said implementation is appropriate for both personal AWS accounts and large scale multi-account organizations.

Having implemented this on behalf of others several times, I'll share the common pain points: * There's a long lead time. You need to enable Cost Explorer (24-48 hours). If you're trying for fine distinctions, activating tags as cost allocation tags is another 24 hours * AWS cost data is a lagging indicator, so you need to be able to absorb a day of charges * Automation support is poor, especially for organizations * Organization budgets configured at the account level are misleading if you don't understand how they're configured

What's really wanted here is that AWS needs to commit to more timely cost data delivery such that you can create an hourly budget with an associated action.

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#88

Post by omneity »

infecto wrote: post #60
omneity wrote: post #37

> Time-to-First-Token of approximately 19 seconds for a gemma3:4b model (this includes startup time, model loading time, and running the inference) This is my biggest pet-peeve with serverless GPU. 19 seconds is a horrible latency from the user’s perspective and that’s a best case scenario. If this is the best one of the most experienced teams in the world can do, with a small 4B model, then it feels like serverless…

If you were running a real business with these would the aim not be to overprovision and to setup auto scaling in such a way that you always have excess capacity?

That seems to be the gist of it. You cannot rely on serverless alone and you need one or many pre-warmed instances at all times. This distinction is rarely mentioned in serverless GPU spaces yet has been my experience in general.

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#90

Post by hn_throwaway_99 »

ZeroCool2u wrote: post #85
AChampaign wrote: post #83

Earlier quoted context omitted.

I think Lambda is more or less the AWS equivalent.

I think Cloud Run Functions would be the direct equivalent to Lambda.

I agree, but in the GCP world, a lot of these things are merging. My understanding is that Cloud Run, Cloud Run Functions (previously known as Cloud Functions Gen2) and even App Engine Flexible all run in the same underlying cloud run infrastructure, so it's essentially just some interface differences that to me now seem more like historical legacy/backwards compatibility reasons than meaningful functionality differences (e.g. Functions can now handle multiple concurrent requests).
Post reply on HN