Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

cloud.google.com

Post reply on HN
Print view View original

171–180 of 191 posts

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#171

Post by geodel »

petesergeant wrote: post #39
isoprophlex wrote: post #6

All the cruft of a big cloud provider, AND the joy of uncapped yolo billing that has the potential to drain your creditcard overnight. No thanks, I'll personally stick with Modal and vast.ai

I've abandoned DataDog in production for just this reason. Is the amount of money they make on dinging people who screw up really worth the ill-will and people who decide they're just not going to start projects on these platforms?

> Is the amount of money they make on dinging people who screw up really worth the ill-will

I think it is .

1) They make money for services they provided instead of looking into meaning of what customer actually wanted.

2) Small time customers move away so they concentrate energy on big enterprise sales.

Not justifying anything here but it just kind of make business sense for them.

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#172

Post by geodel »

sharifhsn wrote: post #98
weinzierl wrote: post #74

Earlier quoted context omitted.

I never used modal or vast.ai and from their pages it was not obvious how they solve the yolo billing issue? Are they pre-paid or do they support caps?

I know vast.ai uses a prepaid credits system.

Doesn't seem vast. Seems tight-budget.ai to me :-)

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#173

Post by petesergeant »

geodel wrote: post #171
petesergeant wrote: post #39

Earlier quoted context omitted.

I've abandoned DataDog in production for just this reason. Is the amount of money they make on dinging people who screw up really worth the ill-will and people who decide they're just not going to start projects on these platforms?

> Is the amount of money they make on dinging people who screw up really worth the ill-will I think it is . 1) They make money for services they provided instead of looking into meaning of what customer actually wanted. 2) Small time customers move away so they concentrate energy on big enterprise sales. Not justifying anything here but it just kind of make business sense for them.

Definitely possible. I wonder over what time period you miss out on small customers who become big customers and go on that journey with you; perhaps that would be minimal anyway.

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#174

Post by jiggawatts »

mwest217 wrote: post #69
anonymousab wrote: post #59

Earlier quoted context omitted.

I remember going out to dinner, years ago, with a fairly senior AWS billing engineer. An acquaintance of a coworker. He looked completely surprised when I asked about runaway billing and why there wasn't any simple options to cap a given resource to prevent those cases. His response was that they didn't build that because none of their customers wanted anything like that, as far as he was aware.

Disclaimer: I work at Google but not on cloud. Opinions my own. I think the reason this doesn’t get prioritized is that large customers don’t actually want a “stop serving if I pass this limit” amount. If there’s a spike in traffic, they probably would rather pay the money to serve it. The customers that would want this feature are small-dollar customers, and from an economic perspective it makes less sense to priori…

Every large enterprise has insurmountable difficult even imagining why customers would want something as bizarre as a "stop loss" on their spending...

... right up until it's their own bottom line that is at risk, and then like magic spending limits become a critical feature.

For example, Azure has no stop-loss feature for paid customers, but it does for the "free" Visual Studio subscriber credits. Because if some random dev with a VS subscription blows through $100K of GPU time due to a missing spending constraint, that's Microsoft's problem, not their own.

It's as simple as that.

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#175

Post by [deleted] »

m1 wrote: post #165

Love cloud run and this looks like a great addition. Only things I wish from cloud run is being able to run self hosted GitHub runners on it (last time I checked this wasn’t possible as it requires root), also the new worker pool feature seems great in practice but it looks like you have to write the scaler yourself rather than it being built in.

[deleted]

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#176

Post by holografix »

ajayvk wrote: post #135
bodantogat wrote: post #93

Earlier quoted context omitted.

I had the opposite experience with cloud run. Mysterious scale outs/restarts - I had to buy a paid subscription to cloud support to get answers and found none. Moved to self managed VMs. Maybe things have changed now.

https://github.com/claceio/clace is project I am building which gives a Cloud Run type deployment experience on your own VMs. For each app, it supports scale down to zero containers (scaling up beyond one is being built). The authorization and auditing features are designed for internal tools, any app can be deployed otherwise.

Have a look at Knative

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#177

Post by Bombthecat »

ashishb wrote: post #54

I love Google Cloud Run and highly recommend it as the best option[1]. The Cloud Run GPU, however is not something I can recommend. It is not cost effective (instance based billing is expensive as opposed to request based billing), GPU choices are limited, and the general loading/unloading of model (gigabytes) from GPU memory makes it slow to be used as server less. Once you compare the numbers it is better to use a…

You don't go to cloud services because they are cheaper.

You go there because you are already there or have contracts etc etc

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#178

Post by Bombthecat »

holografix wrote: post #176
ajayvk wrote: post #135

Earlier quoted context omitted.

https://github.com/claceio/clace is project I am building which gives a Cloud Run type deployment experience on your own VMs. For each app, it supports scale down to zero containers (scaling up beyond one is being built). The authorization and auditing features are designed for internal tools, any app can be deployed otherwise.

Have a look at Knative

Knative is amazing!

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#179

Post by felix_tech »

I've been using this for daily/weekly ETL tasks which saves quite a lot of money vs having an instance on all the time but it's been clunky.

The main issue is despite there being a 60 minute timeout available the API will just straight up not return a response code if your request takes > ~5 minutes in most cases so you gotta make sure you can poll where the datas being stored and let the client time out

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#180

Post by seabrookmx »

pryz wrote: post #130
seabrookmx wrote: post #111

Earlier quoted context omitted.

You're thinking of gvisor. But no, the "gen2" runtime is a microvm ala firecracker and performs a lot better as a result.

https://github.com/cloud-hypervisor/cloud-hypervisor or something else?

Possibly? I haven't found any public documentation that says specifically what hypervisor is used.

Google built crosvm which was the initial inspiration for firecracker, but Cloud Run runs on top of Borg (this fact is publicly documented). Borg is closed source, so it's possible the specific hypervisor they're using is as well.

Post reply on HN