Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

cloud.google.com

Post reply on HN
Print view View original

161–170 of 191 posts

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#161

Post by [deleted] »

wut42 wrote: post #55
omneity wrote: post #37

> Time-to-First-Token of approximately 19 seconds for a gemma3:4b model (this includes startup time, model loading time, and running the inference) This is my biggest pet-peeve with serverless GPU. 19 seconds is a horrible latency from the user’s perspective and that’s a best case scenario. If this is the best one of the most experienced teams in the world can do, with a small 4B model, then it feels like serverless…

Definitely -- and yet it's kinda a feat compared to other solutions: when i tried Runpod Serverless i could wait up to five minutes for a cold start to a even more smaller model than a 4B.

[deleted]

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#162

Post by progbits »

mythz wrote: post #20

The pricing doesn't look that compelling, here are the hourly rate comparisons vs runpod.io vs vast.ai: 1x L4 24GB: google: $0.71; runpod.io: $0.43, spot: $0.22 4x L4 24GB: google: $4.00; runpod.io: $1.72, spot: $0.88 1x A100 80GB: google: $5.07; runpod.io: $1.64, spot: $0.82; vast.ai $0.880, spot: $0.501 1x H100 80GB: google: $11.06; runpod.io: $2.79, spot: $1.65; vast.ai $1.535, spot: $0.473 8x H200 141GB: google:…

You can just go to "create compute instance" to see the spot pricing.

Eg GCP price for spot 1xH100 is $2.55/hr, lower with sustained use discounts. But only hobbyists pay these prices, any company is going to ask for a discount and will get it.

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#163

Post by JoshTriplett »

rpei wrote: post #159
JoshTriplett wrote: post #99

Earlier quoted context omitted.

Does Cloud Run still use a fake Linux kernel emulated by Go, rather than a real VM? Does Cloud Run give you root?

We (I work on Cloud Run) are working on root access. If you'd like to know more you can reach me rpei@google.com

Awesome! I'll reach out to you, thank you.

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#164

Post by steren »

happyopossum wrote: post #157
rvnx wrote: post #16

Earlier quoted context omitted.

According to the press release, "we achieved an impressive Time-to-First-Token of approximately 19 seconds for a gemma3:4b model" Imagine, you have a very small weak model, and you have to wait 20 seconds for your request.

> Imagine, you have a very small weak model, and you have to wait 20 seconds for your request. For your first request, after having scaled to 0 while it wasn’t in use. For a lot of use cases, that sounds great.

Also, a GPU instance needs 5s to start. The test depends on how large the model is. So a "very small weak model" can lead much faster than 20s

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#165

Post by m1 »

Love cloud run and this looks like a great addition. Only things I wish from cloud run is being able to run self hosted GitHub runners on it (last time I checked this wasn’t possible as it requires root), also the new worker pool feature seems great in practice but it looks like you have to write the scaler yourself rather than it being built in.

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#166

Post by mgraczyk »

krembo wrote: post #146
gabe_monroy wrote: post #104

Earlier quoted context omitted.

google vp here: we appreciate the feedback! i generally agree that if you have a strong understanding of your static capacity needs, pre-provisioning VMs is likely to be more cost efficient with today's pricing. cloud run GPUs are ideal for more bursty workloads -- maybe a new AI app that doesn't yet have PMF, where you really need that scale-to-zero + fast start for more sparse traffic patterns.

How does that compare to spinning up some ec2s with amazon trainium gpus?

Depending on your model, you may spend a lot of time trying to get it to work with Trainium

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#167

Post by advisedwang »

montebicyclelo wrote: post #28
isoprophlex wrote: post #6

All the cruft of a big cloud provider, AND the joy of uncapped yolo billing that has the potential to drain your creditcard overnight. No thanks, I'll personally stick with Modal and vast.ai

Not providing a cap on spending is a major flaw of GCP for individuals / small projects. With Cloud Run, AFAIK, spending can effectively be capped by: limiting concurrency, plus limiting the max number of instances it can scale to. (But this is not as good as GCP having a proper cap.)

It's rock and a hard place for the cloud providers.

Cap billing, and you have created an outage waiting to happen, one that will be triggered if they ever have sudden success growth.

Don't cap billing, and you have created a bankruptcy waiting to happen.

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#168

Post by aniruddhc »

m1 wrote: post #165

Love cloud run and this looks like a great addition. Only things I wish from cloud run is being able to run self hosted GitHub runners on it (last time I checked this wasn’t possible as it requires root), also the new worker pool feature seems great in practice but it looks like you have to write the scaler yourself rather than it being built in.

Hi! I'm the Eng Manager responsible for Autoscaling for Serverless and Worker Pools.

We're actively defining our roadmap, and understanding your use case would be incredibly valuable. If you're open to it, please email me at @google.com. I'd love to learn more about how you'd use worker pools and what kind of workloads you need to scale.

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#169

Post by Sn0wCoder »

gabe_monroy wrote: post #104
ashishb wrote: post #54

I love Google Cloud Run and highly recommend it as the best option[1]. The Cloud Run GPU, however is not something I can recommend. It is not cost effective (instance based billing is expensive as opposed to request based billing), GPU choices are limited, and the general loading/unloading of model (gigabytes) from GPU memory makes it slow to be used as server less. Once you compare the numbers it is better to use a…

google vp here: we appreciate the feedback! i generally agree that if you have a strong understanding of your static capacity needs, pre-provisioning VMs is likely to be more cost efficient with today's pricing. cloud run GPUs are ideal for more bursty workloads -- maybe a new AI app that doesn't yet have PMF, where you really need that scale-to-zero + fast start for more sparse traffic patterns.

Has this changed? When I looked pre-ga the requirements were you need to pay for the CPU 24x7 to attach a GPU so that is not really scaling to zero unless this requirement has changed...

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#170

Post by ashishb »

Sn0wCoder wrote: post #169
gabe_monroy wrote: post #104

Earlier quoted context omitted.

google vp here: we appreciate the feedback! i generally agree that if you have a strong understanding of your static capacity needs, pre-provisioning VMs is likely to be more cost efficient with today's pricing. cloud run GPUs are ideal for more bursty workloads -- maybe a new AI app that doesn't yet have PMF, where you really need that scale-to-zero + fast start for more sparse traffic patterns.

Has this changed? When I looked pre-ga the requirements were you need to pay for the CPU 24x7 to attach a GPU so that is not really scaling to zero unless this requirement has changed...

Speaking from my experience, it does scale to zero except you pay for 15 mins after the last request.

So if you get all your requests in a 2 hours window then that's great. It will scale to zero for rest of the 22 hours.

However, if you get at least one request every 15 mins then you will pay for 24 hours and it is ~3X more expensive then equivalent VM on Google Cloud.

Post reply on HN