Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

cloud.google.com

Post reply on HN
Print view View original

111–120 of 191 posts

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#111

Post by seabrookmx »

JoshTriplett wrote: post #99
ashishb wrote: post #54

I love Google Cloud Run and highly recommend it as the best option[1]. The Cloud Run GPU, however is not something I can recommend. It is not cost effective (instance based billing is expensive as opposed to request based billing), GPU choices are limited, and the general loading/unloading of model (gigabytes) from GPU memory makes it slow to be used as server less. Once you compare the numbers it is better to use a…

Does Cloud Run still use a fake Linux kernel emulated by Go, rather than a real VM? Does Cloud Run give you root?

You're thinking of gvisor. But no, the "gen2" runtime is a microvm ala firecracker and performs a lot better as a result.

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#112

Post by icedchai »

AChampaign wrote: post #83
icedchai wrote: post #81

Earlier quoted context omitted.

Cloud Run is a great service. I find it much easier to work with than AWS's equivalent (ECS/Fargate.)

I think Lambda is more or less the AWS equivalent.

It's not. Cloud Run can be longer running: you can have batch and services. Lambda is closer to Cloud Functions.

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#113

Post by psanford »

icedchai wrote: post #81
ashishb wrote: post #54

I love Google Cloud Run and highly recommend it as the best option[1]. The Cloud Run GPU, however is not something I can recommend. It is not cost effective (instance based billing is expensive as opposed to request based billing), GPU choices are limited, and the general loading/unloading of model (gigabytes) from GPU memory makes it slow to be used as server less. Once you compare the numbers it is better to use a…

Cloud Run is a great service. I find it much easier to work with than AWS's equivalent (ECS/Fargate.)

AWS AppRunner is the closest equivalent to Cloud Run. Its really not close though, AppRunner is an unloved service at AWS and is missing a lot of the features that make Cloud Run nice.

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#114

Post by icedchai »

gabe_monroy wrote: post #106
icedchai wrote: post #81

Earlier quoted context omitted.

Cloud Run is a great service. I find it much easier to work with than AWS's equivalent (ECS/Fargate.)

i am biased, but i agree :)

hah. I looked at your comments and saw you were a google VP! I've migrated some small systems from AWS to GCP for various POCs and prototypes, mostly Lambda and ECS to Cloud Run, and find GCP provides a better developer experience overall.

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#116

Post by ata_aman »

ninetyninenine wrote: post #25

Im tired of using AI in cloud services. I want user friendly locally owned AI hardware. Right now nothing is consumer friendly. I can’t get a packaged deal of some locally running ChatGPT quality UI or voice command system in an all in one package. Like what Macs did for PCs I want the same for AI.

I made something[0] last year to have something very consumer friendly. Unbox->connect->run. First iteration is purely to test out the concept and is pretty low power, currently working on a GPU version for bigger models and launching Q4 this year.

[0] https://persys.ai

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#117

Post by nullpointerexp »

omneity wrote: post #88
infecto wrote: post #60

Earlier quoted context omitted.

If you were running a real business with these would the aim not be to overprovision and to setup auto scaling in such a way that you always have excess capacity?

That seems to be the gist of it. You cannot rely on serverless alone and you need one or many pre-warmed instances at all times. This distinction is rarely mentioned in serverless GPU spaces yet has been my experience in general.

When scaling from 0 to 1 instances, yes, you have to wait 19 seconds.

For scaling N --> N+1 - If you configure the correct concurrency value (the number of parallel requests one instance can handle), Cloud Run will scale up to additional instances when getting to X% (I think it's 70%). That will be before the instance is fully exhausted. So your users should not experience the 19 seconds cold start.

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#118

Post by albeebe1 »

Oh this is great news. After a $1000 bill running a model on vertex.ai continuously for a little test i forgot to shut down, this will be my go to now. I've been using Cloud Run for years running production microservices, and little hobby projects and i've found it simple and cost effective.

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#119

Post by AndresSRG »

jbarrow wrote: post #8

I’m personally a huge fan of Modal, and have been using their serverless scale-to-zero GPUs for a while. We’ve seen some nice cost reductions from using them, while also being able to scale WAY UP when needed. All with minimal development effort. Interesting to see a big provider entering this space. Originally swapped to Modal because big providers weren’t offering this (e.g. AWS lambdas can’t run on GPU instances).…

I’m also a big fan.

Modal has the fastest cold-start I’ve seen for 10GB+ models.

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#120

Post by gabe_monroy »

icedchai wrote: post #114
gabe_monroy wrote: post #106

Earlier quoted context omitted.

i am biased, but i agree :)

hah. I looked at your comments and saw you were a google VP! I've migrated some small systems from AWS to GCP for various POCs and prototypes, mostly Lambda and ECS to Cloud Run, and find GCP provides a better developer experience overall.

love that you're enjoying the devex. we put a lot of sweat into it, especially in services like cloud run.
Post reply on HN