Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

cloud.google.com

Post reply on HN
Print view View original

151–160 of 191 posts

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#151

Post by dr_kiszonka »

jbarrow wrote: post #8

I’m personally a huge fan of Modal, and have been using their serverless scale-to-zero GPUs for a while. We’ve seen some nice cost reductions from using them, while also being able to scale WAY UP when needed. All with minimal development effort. Interesting to see a big provider entering this space. Originally swapped to Modal because big providers weren’t offering this (e.g. AWS lambdas can’t run on GPU instances).…

Thanks for sharing! They even support running HIPAA-compliant workloads, which I didn't anticipate.

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#152

Post by oldandboring »

isoprophlex wrote: post #6

All the cruft of a big cloud provider, AND the joy of uncapped yolo billing that has the potential to drain your creditcard overnight. No thanks, I'll personally stick with Modal and vast.ai

> uncapped yolo billing

This made me laugh out loud, thank you for this!

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#153

Post by badrequest »

yarri wrote: post #110
montebicyclelo wrote: post #28

Earlier quoted context omitted.

Not providing a cap on spending is a major flaw of GCP for individuals / small projects. With Cloud Run, AFAIK, spending can effectively be capped by: limiting concurrency, plus limiting the max number of instances it can scale to. (But this is not as good as GCP having a proper cap.)

[edit - Gabe responded]. See this Cloud Run spending cap recommendation [0] to disable billing, which potentially irreversibly deletes resources but does cap spend! [0] https://cloud.google.com/billing/docs/how-to/disable-billing...

Sure, but why post a tutorial of how to spin this up in GCP instead of...productizing it in GCP?

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#154

Post by ashishb »

jakecodes wrote: post #137
gabe_monroy wrote: post #104

Earlier quoted context omitted.

google vp here: we appreciate the feedback! i generally agree that if you have a strong understanding of your static capacity needs, pre-provisioning VMs is likely to be more cost efficient with today's pricing. cloud run GPUs are ideal for more bursty workloads -- maybe a new AI app that doesn't yet have PMF, where you really need that scale-to-zero + fast start for more sparse traffic patterns.

Appreciate the thoughtful response! I’m actually right in the ICP you described — I’ve run my own VMs in the past and recently switched to Cloud Run to simplify ops and take advantage of scale-to-zero. In my case, I was running a few inference jobs and expected a ~$100 bill. But due to the instance-based behavior, it stayed up the whole time, and I ended up with a $1,000 charge for relatively little usage. I’m fairly…

> But due to the instance-based behavior, it stayed up the whole time, and I ended up with a $1,000 charge for relatively little usage.

Indeed. IIRC, if you get a single request every 15 mins (~100 requests a day), you will pay for Cloud Run GPU for the full day.

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#155

Post by ashishb »

gabe_monroy wrote: post #106
icedchai wrote: post #81

Earlier quoted context omitted.

Cloud Run is a great service. I find it much easier to work with than AWS's equivalent (ECS/Fargate.)

i am biased, but i agree :)

Yeah, anyone who uses GCP and AWS thoroughly will agree that GCP is a superior developer experience.

The problem is continuous product churn. This was discussed at length at https://news.ycombinator.com/item?id=41614795

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#156

Post by happyopossum »

coredog64 wrote: post #86
mwest217 wrote: post #69

Earlier quoted context omitted.

Disclaimer: I work at Google but not on cloud. Opinions my own. I think the reason this doesn’t get prioritized is that large customers don’t actually want a “stop serving if I pass this limit” amount. If there’s a spike in traffic, they probably would rather pay the money to serve it. The customers that would want this feature are small-dollar customers, and from an economic perspective it makes less sense to priori…

As noted above, there is enough value here such that AWS implemented this several years ago. Said implementation is appropriate for both personal AWS accounts and large scale multi-account organizations. Having implemented this on behalf of others several times, I'll share the common pain points: * There's a long lead time. You need to enable Cost Explorer (24-48 hours). If you're trying for fine distinctions, activa…

> Said implementation is appropriate for both personal AWS accounts and large scale multi-account organizations.

Followed by a list of caveats that make it wholly irrelevant for an individual who is afraid of a surprise charge covering less than several days.

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#157

Post by happyopossum »

rvnx wrote: post #16
lexandstuff wrote: post #14

Earlier quoted context omitted.

Not to mention, if it's an ML workload, you'll also have to factor in downloading the weights and loading them into memory, which can double that time or more.

According to the press release, "we achieved an impressive Time-to-First-Token of approximately 19 seconds for a gemma3:4b model" Imagine, you have a very small weak model, and you have to wait 20 seconds for your request.

> Imagine, you have a very small weak model, and you have to wait 20 seconds for your request.

For your first request, after having scaled to 0 while it wasn’t in use. For a lot of use cases, that sounds great.

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#158

Post by happyopossum »

ringeryless wrote: post #63

i wonder what all this hype-driven overcapacity will be used for by future generations. once this bubble pops we are going to have some serious albeit high-latency hardware

> overcapacity

I’m Not sure that word means what you think it means. There is a pretty severe shortage of GPU capacity in the industry right now.

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#159

Post by rpei »

JoshTriplett wrote: post #99
ashishb wrote: post #54

I love Google Cloud Run and highly recommend it as the best option[1]. The Cloud Run GPU, however is not something I can recommend. It is not cost effective (instance based billing is expensive as opposed to request based billing), GPU choices are limited, and the general loading/unloading of model (gigabytes) from GPU memory makes it slow to be used as server less. Once you compare the numbers it is better to use a…

Does Cloud Run still use a fake Linux kernel emulated by Go, rather than a real VM? Does Cloud Run give you root?

We (I work on Cloud Run) are working on root access. If you'd like to know more you can reach me rpei@google.com

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#160

Post by happyopossum »

omneity wrote: post #37

> Time-to-First-Token of approximately 19 seconds for a gemma3:4b model (this includes startup time, model loading time, and running the inference) This is my biggest pet-peeve with serverless GPU. 19 seconds is a horrible latency from the user’s perspective and that’s a best case scenario. If this is the best one of the most experienced teams in the world can do, with a small 4B model, then it feels like serverless…

Sure, but how often is an enterprise deployed LLM application really cold-starting? While you could run this for one-off and personal use, this is probably more geared towards bursty ‘here’s an agent for my company sales reps’ kind of workloads, so you can have an instance warmed, then autoscale up at 8:03am when everyone gets online (or in the office or whatever).

At that point, 19 seconds looks great, as lower latency startup times allow for much more efficient autoscaling.

Post reply on HN