That’s 67ct / hour for a gpu enabled instance. That’s pretty good, but I have no idea how T4 GPU’s compare against others.
Cloud Run GPUs, now GA, makes running AI workloads easier for everyone
31–40 of 191 posts
Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone
#32Im tired of using AI in cloud services. I want user friendly locally owned AI hardware. Right now nothing is consumer friendly. I can’t get a packaged deal of some locally running ChatGPT quality UI or voice command system in an all in one package. Like what Macs did for PCs I want the same for AI.
Your local computer is not powerful enough, and that's why you must welcome those brand new mainframes... I mean, "cloud services."
Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone
#33The pricing doesn't look that compelling, here are the hourly rate comparisons vs runpod.io vs vast.ai: 1x L4 24GB: google: $0.71; runpod.io: $0.43, spot: $0.22 4x L4 24GB: google: $4.00; runpod.io: $1.72, spot: $0.88 1x A100 80GB: google: $5.07; runpod.io: $1.64, spot: $0.82; vast.ai $0.880, spot: $0.501 1x H100 80GB: google: $11.06; runpod.io: $2.79, spot: $1.65; vast.ai $1.535, spot: $0.473 8x H200 141GB: google:…
I think the Google prices are billed per-second so under 20min you are better on Google?
runpod is billed by the minute
Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone
#34Reason Cloud Run is so nice compared to other providers is that it has autoscaling, with scaling to 0. Meaning it can cost basically 0 if it's not being used. Also can set a cap on the scaling, e.g. 5 instances max, which caps the max cost of the service too. - Note, I only have experience with the CPU version of Cloud Run, (which is very reliable / easy).
Even regular Cloud Run can take a lot of time to boot (~3 to 30 seconds), so this can be a problem when scaling to 0
That's not my experience, using Go. Never measured, but it goes to 0 all the time, so I would definitely noticed more than a couple of seconds.
Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone
#35Earlier quoted context omitted.
Those, on gcp, are just alerts, not hard limits, no?
Yeah. I think you can hack together a function which pulls the plug automatically if a billing alert fires, but IIRC the alerts can take a few hours to respond, so extreme runaway usage could still result in a bad time.
I would rather not realize i had left a bug in that hack after the fact.
Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone
#36All the cruft of a big cloud provider, AND the joy of uncapped yolo billing that has the potential to drain your creditcard overnight. No thanks, I'll personally stick with Modal and vast.ai
Cloud Run is great but no billing limits is too scary. No idea why they don't address this. They must know if they support individuals we'll eventually leave our saases there.
Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone
#37> Time-to-First-Token of approximately 19 seconds for a gemma3:4b model (this includes startup time, model loading time, and running the inference)
This is my biggest pet-peeve with serverless GPU. 19 seconds is a horrible latency from the user’s perspective and that’s a best case scenario.
If this is the best one of the most experienced teams in the world can do, with a small 4B model, then it feels like serverless is really restricted to non-interactive use cases.
Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone
#38All the cruft of a big cloud provider, AND the joy of uncapped yolo billing that has the potential to drain your creditcard overnight. No thanks, I'll personally stick with Modal and vast.ai
You can set max instances in Cloud Run, which is an effective limit on how much you'll spend.
Also, hard dollar caps are rarely if ever the right choice. App Engine used to have these, and the practical effect was that your website would completely stop working exactly when you least want it to (posted on HN etc).
It's better to set billing alerts and make the call yourself if they go off.
Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone
#39All the cruft of a big cloud provider, AND the joy of uncapped yolo billing that has the potential to drain your creditcard overnight. No thanks, I'll personally stick with Modal and vast.ai
I've abandoned DataDog in production for just this reason. Is the amount of money they make on dinging people who screw up really worth the ill-will and people who decide they're just not going to start projects on these platforms?
Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone
#40The pricing doesn't look that compelling, here are the hourly rate comparisons vs runpod.io vs vast.ai: 1x L4 24GB: google: $0.71; runpod.io: $0.43, spot: $0.22 4x L4 24GB: google: $4.00; runpod.io: $1.72, spot: $0.88 1x A100 80GB: google: $5.07; runpod.io: $1.64, spot: $0.82; vast.ai $0.880, spot: $0.501 1x H100 80GB: google: $11.06; runpod.io: $2.79, spot: $1.65; vast.ai $1.535, spot: $0.473 8x H200 141GB: google:…
I think the Google prices are billed per-second so under 20min you are better on Google?
RunPod also charges per second [1], also this is Google's expected avg cost per hour after running it 24/7 for an entire month, I couldn't find an hourly cost for each GPU.
When you need under [1] https://docs.runpod.io/serverless/pricing