Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

Post by hn_throwaway_99 » Wed, Jun 04, 2025, 5:20 PM UTC

yegle wrote: post #92

hn_throwaway_99 wrote: post #90

Earlier quoted context omitted.

I agree, but in the GCP world, a lot of these things are merging. My understanding is that Cloud Run, Cloud Run Functions (previously known as Cloud Functions Gen2) and even App Engine Flexible all run in the same underlying cloud run infrastructure, so it's essentially just some interface differences that to me now seem more like historical legacy/backwards compatibility reasons than meaningful functionality differe…

FWIW, App Engine Flexible is a different product that runs on GCE VM. Other products (App Engine standard, Cloud Functions gen1, Cloud Run, Cloud Run Functions) share many underlying infrastructures.

Post by agcat » Wed, Jun 04, 2025, 5:38 PM UTC

diggan wrote: post #51

omneity wrote: post #37

> Time-to-First-Token of approximately 19 seconds for a gemma3:4b model (this includes startup time, model loading time, and running the inference) This is my biggest pet-peeve with serverless GPU. 19 seconds is a horrible latency from the user’s perspective and that’s a best case scenario. If this is the best one of the most experienced teams in the world can do, with a small 4B model, then it feels like serverless…

That has to be cold-start, and next N requests would surely be using the already started thing? It sounds bananas they'd even mention using something like that with 19 seconds latency for all requests in any context.

Post by steren » Wed, Jun 04, 2025, 5:47 PM UTC

mythz wrote: post #20

The pricing doesn't look that compelling, here are the hourly rate comparisons vs runpod.io vs vast.ai: 1x L4 24GB: google: $0.71; runpod.io: $0.43, spot: $0.22 4x L4 24GB: google: $4.00; runpod.io: $1.72, spot: $0.88 1x A100 80GB: google: $5.07; runpod.io: $1.64, spot: $0.82; vast.ai $0.880, spot: $0.501 1x H100 80GB: google: $11.06; runpod.io: $2.79, spot: $1.65; vast.ai $1.535, spot: $0.473 8x H200 141GB: google:…

Post by [deleted] » Wed, Jun 04, 2025, 5:57 PM UTC

vrosas wrote: post #127

randlet wrote: post #66

Earlier quoted context omitted.

Setting max instances effectively caps your spend right?

Yes. CR has had this feature since day 1, people just don't bother to read the docs and would rather write long blog posts blaming their cloud provider for manufacturing the gun they shot themself in the foot with.

Post by [deleted] » Wed, Jun 04, 2025, 5:59 PM UTC

vrosas wrote: post #127

randlet wrote: post #66

Earlier quoted context omitted.

Setting max instances effectively caps your spend right?

Yes. CR has had this feature since day 1, people just don't bother to read the docs and would rather write long blog posts blaming their cloud provider for manufacturing the gun they shot themself in the foot with.