Cloud Run GPUs, now GA, makes running AI workloads easier for everyone
1–10 of 191 posts
Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone
#2If I understand this correctly, I should be able to stand up an API running arbitrary models (e.g. from Hugging Face), and it’s not quite charged by the token but should be very cheap if my usage is sporadic. Is that correct? Seems pretty huge if so, most of the providers I looked at required a monthly fee to run a custom model.
Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone
#3If I understand this correctly, I should be able to stand up an API running arbitrary models (e.g. from Hugging Face), and it’s not quite charged by the token but should be very cheap if my usage is sporadic. Is that correct? Seems pretty huge if so, most of the providers I looked at required a monthly fee to run a custom model.
Runpod, vast, coreweave, replicate... just a bunch of alternatives that let you run serverless GPU inference.
Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone
#4If I understand this correctly, I should be able to stand up an API running arbitrary models (e.g. from Hugging Face), and it’s not quite charged by the token but should be very cheap if my usage is sporadic. Is that correct? Seems pretty huge if so, most of the providers I looked at required a monthly fee to run a custom model.
Runpod, vast, coreweave, replicate... just a bunch of alternatives that let you run serverless GPU inference.
you can't just sign up for coreweave, can you?
Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone
#5Reason Cloud Run is so nice compared to other providers is that it has autoscaling, with scaling to 0. Meaning it can cost basically 0 if it's not being used. Also can set a cap on the scaling, e.g. 5 instances max, which caps the max cost of the service too. - Note, I only have experience with the CPU version of Cloud Run, (which is very reliable / easy).
Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone
#6All the cruft of a big cloud provider, AND the joy of uncapped yolo billing that has the potential to drain your creditcard overnight. No thanks, I'll personally stick with Modal and vast.ai
Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone
#7The Nvidia L4 has 24GB of VRAM and consumes 72 watts, which is relatively low compared to other datacenter cards. It's not a monster GPU, but it should work OK for inference.
Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone
#8I’m personally a huge fan of Modal, and have been using their serverless scale-to-zero GPUs for a while. We’ve seen some nice cost reductions from using them, while also being able to scale WAY UP when needed. All with minimal development effort.
Interesting to see a big provider entering this space. Originally swapped to Modal because big providers weren’t offering this (e.g. AWS lambdas can’t run on GPU instances). Assuming all providers are going to start moving towards offering this?
Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone
#9Reason Cloud Run is so nice compared to other providers is that it has autoscaling, with scaling to 0. Meaning it can cost basically 0 if it's not being used. Also can set a cap on the scaling, e.g. 5 instances max, which caps the max cost of the service too. - Note, I only have experience with the CPU version of Cloud Run, (which is very reliable / easy).
Even regular Cloud Run can take a lot of time to boot (~3 to 30 seconds), so this can be a problem when scaling to 0
Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone
#10The value in this really is running small custom models or the absolute latest open weight models.
Why bother when you can get payg API access to popular open weights models like Llama on Vertex AI model garden or at the edge on Cloudflare?