Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

cloud.google.com

Post reply on HN
Print view View original

51–60 of 191 posts

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#51

Post by diggan »

omneity wrote: post #37

> Time-to-First-Token of approximately 19 seconds for a gemma3:4b model (this includes startup time, model loading time, and running the inference) This is my biggest pet-peeve with serverless GPU. 19 seconds is a horrible latency from the user’s perspective and that’s a best case scenario. If this is the best one of the most experienced teams in the world can do, with a small 4B model, then it feels like serverless…

That has to be cold-start, and next N requests would surely be using the already started thing? It sounds bananas they'd even mention using something like that with 19 seconds latency for all requests in any context.

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#52

Post by triknomeister »

diggan wrote: post #49
huksley wrote: post #46

A small and independent EU GPU cloud provider, DataCrunch (I am not affiliated), offers VMs with Nvidia GPUs even cheaper than Run Pod, etc 1x A100 80Gb 1.37€/hour 1x H100 80Gb 2.19€/hour

Or go P2P with Vast.ai, cheapest A100 right now is a setup with 2x A100 for $0.8/hour (so $0.4 per A100). Not affiliated with them, but mostly happy user. Be vary of network speeds though, some hosts are clearly on shared bandwidth and reported numbers don't always line up with reality, which kind of sucks when you're trying to shuffle around 100GB of data.

You really need NVL for some performance.

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#53

Post by sigmoid10 »

huksley wrote: post #46

A small and independent EU GPU cloud provider, DataCrunch (I am not affiliated), offers VMs with Nvidia GPUs even cheaper than Run Pod, etc 1x A100 80Gb 1.37€/hour 1x H100 80Gb 2.19€/hour

That's funny. You can get a 1x H100 80Gb VM at lambda.ai for $2.49/hour. At the current exchange rate, that's exactly 2.19€. Coincidence or is this actually some kind of ceiling?

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#54

Post by ashishb »

I love Google Cloud Run and highly recommend it as the best option[1]. The Cloud Run GPU, however is not something I can recommend. It is not cost effective (instance based billing is expensive as opposed to request based billing), GPU choices are limited, and the general loading/unloading of model (gigabytes) from GPU memory makes it slow to be used as server less.

Once you compare the numbers it is better to use a VM + GPU if the utilization of your service is even only for 30% of the day.

1 - https://ashishb.net/programming/free-deployment-of-side-proj...

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#55

Post by wut42 »

omneity wrote: post #37

> Time-to-First-Token of approximately 19 seconds for a gemma3:4b model (this includes startup time, model loading time, and running the inference) This is my biggest pet-peeve with serverless GPU. 19 seconds is a horrible latency from the user’s perspective and that’s a best case scenario. If this is the best one of the most experienced teams in the world can do, with a small 4B model, then it feels like serverless…

Definitely -- and yet it's kinda a feat compared to other solutions: when i tried Runpod Serverless i could wait up to five minutes for a cold start to a even more smaller model than a 4B.

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#57

Post by pjmlp »

Disposal8433 wrote: post #32
ninetyninenine wrote: post #25

Im tired of using AI in cloud services. I want user friendly locally owned AI hardware. Right now nothing is consumer friendly. I can’t get a packaged deal of some locally running ChatGPT quality UI or voice command system in an all in one package. Like what Macs did for PCs I want the same for AI.

Your local computer is not powerful enough, and that's why you must welcome those brand new mainframes... I mean, "cloud services."

It is funny how using a Web IDE, and a cloud shell, is such a déjà vu from when I used to do development on a common UNIX server shared by the whole team.

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#58

Post by ipaddr »

decimalenough wrote: post #38
isoprophlex wrote: post #6

All the cruft of a big cloud provider, AND the joy of uncapped yolo billing that has the potential to drain your creditcard overnight. No thanks, I'll personally stick with Modal and vast.ai

You can set max instances in Cloud Run, which is an effective limit on how much you'll spend. Also, hard dollar caps are rarely if ever the right choice. App Engine used to have these, and the practical effect was that your website would completely stop working exactly when you least want it to (posted on HN etc). It's better to set billing alerts and make the call yourself if they go off.

One bad actor / misconfiguration / attack can put you out of business. It not the safest strategy to allow unlimited liability in business or for personal projects.

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#59

Post by anonymousab »

brutus1213 wrote: post #47
montebicyclelo wrote: post #28

Earlier quoted context omitted.

Not providing a cap on spending is a major flaw of GCP for individuals / small projects. With Cloud Run, AFAIK, spending can effectively be capped by: limiting concurrency, plus limiting the max number of instances it can scale to. (But this is not as good as GCP having a proper cap.)

Amazon is the same I think? I live in constant fear we will have a runaway job one day. I get daily emails to myself (as a manager) and to my finance person. We had one instance where a team member forgot to turn off a machine for a few months :( I get why it is a business strategy to not have limits .. but I wonder if providers would get more usage if people had more trusts on costs/predictability.

I remember going out to dinner, years ago, with a fairly senior AWS billing engineer. An acquaintance of a coworker.

He looked completely surprised when I asked about runaway billing and why there wasn't any simple options to cap a given resource to prevent those cases.

His response was that they didn't build that because none of their customers wanted anything like that, as far as he was aware.

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#60

Post by infecto »

omneity wrote: post #37

> Time-to-First-Token of approximately 19 seconds for a gemma3:4b model (this includes startup time, model loading time, and running the inference) This is my biggest pet-peeve with serverless GPU. 19 seconds is a horrible latency from the user’s perspective and that’s a best case scenario. If this is the best one of the most experienced teams in the world can do, with a small 4B model, then it feels like serverless…

If you were running a real business with these would the aim not be to overprovision and to setup auto scaling in such a way that you always have excess capacity?
Post reply on HN