Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

cloud.google.com

Post reply on HN
Print view View original

91–100 of 191 posts

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#91

Post by otherjason »

mythz wrote: post #20

The pricing doesn't look that compelling, here are the hourly rate comparisons vs runpod.io vs vast.ai: 1x L4 24GB: google: $0.71; runpod.io: $0.43, spot: $0.22 4x L4 24GB: google: $4.00; runpod.io: $1.72, spot: $0.88 1x A100 80GB: google: $5.07; runpod.io: $1.64, spot: $0.82; vast.ai $0.880, spot: $0.501 1x H100 80GB: google: $11.06; runpod.io: $2.79, spot: $1.65; vast.ai $1.535, spot: $0.473 8x H200 141GB: google:…

Where did you get the pricing for vast.ai here? Looking at their pricing page, I don't see any 8xH200 options for less than $21.65 an hour (and most are more than that).

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#92

Post by yegle »

hn_throwaway_99 wrote: post #90
ZeroCool2u wrote: post #85

Earlier quoted context omitted.

I think Cloud Run Functions would be the direct equivalent to Lambda.

I agree, but in the GCP world, a lot of these things are merging. My understanding is that Cloud Run, Cloud Run Functions (previously known as Cloud Functions Gen2) and even App Engine Flexible all run in the same underlying cloud run infrastructure, so it's essentially just some interface differences that to me now seem more like historical legacy/backwards compatibility reasons than meaningful functionality differe…

FWIW, App Engine Flexible is a different product that runs on GCE VM.

Other products (App Engine standard, Cloud Functions gen1, Cloud Run, Cloud Run Functions) share many underlying infrastructures.

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#93

Post by bodantogat »

ashishb wrote: post #54

I love Google Cloud Run and highly recommend it as the best option[1]. The Cloud Run GPU, however is not something I can recommend. It is not cost effective (instance based billing is expensive as opposed to request based billing), GPU choices are limited, and the general loading/unloading of model (gigabytes) from GPU memory makes it slow to be used as server less. Once you compare the numbers it is better to use a…

I had the opposite experience with cloud run. Mysterious scale outs/restarts - I had to buy a paid subscription to cloud support to get answers and found none. Moved to self managed VMs. Maybe things have changed now.

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#94

Post by tylertreat »

dig1 wrote: post #82
ashishb wrote: post #54

I love Google Cloud Run and highly recommend it as the best option[1]. The Cloud Run GPU, however is not something I can recommend. It is not cost effective (instance based billing is expensive as opposed to request based billing), GPU choices are limited, and the general loading/unloading of model (gigabytes) from GPU memory makes it slow to be used as server less. Once you compare the numbers it is better to use a…

> I love Google Cloud Run and highly recommend it as the best option I'd love to see the numbers for Cloud Run. It's nice for toy projects, but it's a money sink for anything serious, at least from my experience. On one project, we had a long-standing issue with G regarding autoscaling - scaling to zero sounds nice on paper, but they will not mention you the warmup phases where CR can spin up multiple containers for…

> It's nice for toy projects, but it's a money sink for anything serious, at least from my experience.

This is really not my experience with Cloud Run at all. We've found it to actually be quite cost effective for a lot of different types of systems. For example, we ended up helping a customer migrate a ~$5B/year ecommerce platform onto it (mostly Java/Spring and Typescript services). We originally told them they should target GKE but they were adamant about serverless and it ended up being a perfect fit. They were paying like $5k/mo which is absurdly cheap for a platform generating that kind of revenue.

I guess it depends on the nature of each workload, but for businesses that tend to "follow the sun" I've found it to be a great solution, especially when you consider how little operations overhead there is with it.

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#95

Post by esafak »

ringeryless wrote: post #63

i wonder what all this hype-driven overcapacity will be used for by future generations. once this bubble pops we are going to have some serious albeit high-latency hardware

What overcapacity? People are struggling to find affordable GPUs.

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#96

Post by ninetyninenine »

Disposal8433 wrote: post #32
ninetyninenine wrote: post #25

Im tired of using AI in cloud services. I want user friendly locally owned AI hardware. Right now nothing is consumer friendly. I can’t get a packaged deal of some locally running ChatGPT quality UI or voice command system in an all in one package. Like what Macs did for PCs I want the same for AI.

Your local computer is not powerful enough, and that's why you must welcome those brand new mainframes... I mean, "cloud services."

My local computer is not powerful enough to run training but it can certainly run an LLM. How do I know? Many other people and I have already done it. Deepseek for example can be run locally but it’s not a user friendly setup.

I want an Amazon echo agent running my home with a locally running LLM.

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#97

Post by PaulMest »

bodantogat wrote: post #93
ashishb wrote: post #54

I love Google Cloud Run and highly recommend it as the best option[1]. The Cloud Run GPU, however is not something I can recommend. It is not cost effective (instance based billing is expensive as opposed to request based billing), GPU choices are limited, and the general loading/unloading of model (gigabytes) from GPU memory makes it slow to be used as server less. Once you compare the numbers it is better to use a…

I had the opposite experience with cloud run. Mysterious scale outs/restarts - I had to buy a paid subscription to cloud support to get answers and found none. Moved to self managed VMs. Maybe things have changed now.

Sadly this is still the case. Cloud Run helped us get off the ground. But we've had two outages where Google Enhanced Support could give us no suggestion other than "increase the maximum instances" (not minimum instances). We were doing something like 13 requests/min on this instance at the time. The resource utilization looked just fine. But somehow we had a blip in any containers being available. It even dropped below our min containers. The fix was to manually redeploy the latest revision.

We're now investigating moving to Kubernetes where we will have more control over our destiny. Thankfully a couple people on the team have experience with this.

Something like this never happened with Fargate in the years my previous team had used that.

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#98

Post by sharifhsn »

weinzierl wrote: post #74
isoprophlex wrote: post #6

All the cruft of a big cloud provider, AND the joy of uncapped yolo billing that has the potential to drain your creditcard overnight. No thanks, I'll personally stick with Modal and vast.ai

I never used modal or vast.ai and from their pages it was not obvious how they solve the yolo billing issue? Are they pre-paid or do they support caps?

I know vast.ai uses a prepaid credits system.

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#99

Post by JoshTriplett »

ashishb wrote: post #54

I love Google Cloud Run and highly recommend it as the best option[1]. The Cloud Run GPU, however is not something I can recommend. It is not cost effective (instance based billing is expensive as opposed to request based billing), GPU choices are limited, and the general loading/unloading of model (gigabytes) from GPU memory makes it slow to be used as server less. Once you compare the numbers it is better to use a…

Does Cloud Run still use a fake Linux kernel emulated by Go, rather than a real VM?

Does Cloud Run give you root?

Post reply on HN