Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

cloud.google.com

Post reply on HN
Print view View original

71–80 of 191 posts

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#71

Post by tmoertel »

brutus1213 wrote: post #47
montebicyclelo wrote: post #28

Earlier quoted context omitted.

Not providing a cap on spending is a major flaw of GCP for individuals / small projects. With Cloud Run, AFAIK, spending can effectively be capped by: limiting concurrency, plus limiting the max number of instances it can scale to. (But this is not as good as GCP having a proper cap.)

Amazon is the same I think? I live in constant fear we will have a runaway job one day. I get daily emails to myself (as a manager) and to my finance person. We had one instance where a team member forgot to turn off a machine for a few months :( I get why it is a business strategy to not have limits .. but I wonder if providers would get more usage if people had more trusts on costs/predictability.

> I get why it is a business strategy to not have limits...

What is the strategy? Is is purely market segmentation? (As in: "If you need to worry about spending too much, you're not the big-money kind of enterprise customer we want"?)

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#72

Post by mtrovo »

dragandj wrote: post #67
anonymousab wrote: post #59

Earlier quoted context omitted.

I remember going out to dinner, years ago, with a fairly senior AWS billing engineer. An acquaintance of a coworker. He looked completely surprised when I asked about runaway billing and why there wasn't any simple options to cap a given resource to prevent those cases. His response was that they didn't build that because none of their customers wanted anything like that, as far as he was aware.

Yeah, right. Capping a resource, such a wild idea. Of course they won't implement it for the same reason bar owners don't put a cap on drinks.

I don't know if the analogy works that well, the assumption is that you're making more money then you put in the more traffic you get. As a bar owner is the choice between closing your bar for the month when you run out of beer or running to the supplier to bring more kegs.

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#74

Post by weinzierl »

isoprophlex wrote: post #6

All the cruft of a big cloud provider, AND the joy of uncapped yolo billing that has the potential to drain your creditcard overnight. No thanks, I'll personally stick with Modal and vast.ai

I never used modal or vast.ai and from their pages it was not obvious how they solve the yolo billing issue? Are they pre-paid or do they support caps?

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#75

Post by coredog64 »

brutus1213 wrote: post #47
montebicyclelo wrote: post #28

Earlier quoted context omitted.

Not providing a cap on spending is a major flaw of GCP for individuals / small projects. With Cloud Run, AFAIK, spending can effectively be capped by: limiting concurrency, plus limiting the max number of instances it can scale to. (But this is not as good as GCP having a proper cap.)

Amazon is the same I think? I live in constant fear we will have a runaway job one day. I get daily emails to myself (as a manager) and to my finance person. We had one instance where a team member forgot to turn off a machine for a few months :( I get why it is a business strategy to not have limits .. but I wonder if providers would get more usage if people had more trusts on costs/predictability.

There's a coarse option: Set up a budget and then a budget action. While ECS doesn't have GPU capabilities, the equivalent here would be "IAM action of budget sets deny on expensive service IAM action" (SCP is also available, but that requires an AWS Org, at which point you've probably got a team that already knows this)

It's coarse because it's daily and not hourly. However, you could also self-service do some of this with CloudWatch metrics to map to a cost and then have an alarm action.

https://aws.amazon.com/blogs/mt/manage-cost-overruns-part-1/

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#76

Post by 152132124 »

anonymousab wrote: post #59
brutus1213 wrote: post #47

Earlier quoted context omitted.

Amazon is the same I think? I live in constant fear we will have a runaway job one day. I get daily emails to myself (as a manager) and to my finance person. We had one instance where a team member forgot to turn off a machine for a few months :( I get why it is a business strategy to not have limits .. but I wonder if providers would get more usage if people had more trusts on costs/predictability.

I remember going out to dinner, years ago, with a fairly senior AWS billing engineer. An acquaintance of a coworker. He looked completely surprised when I asked about runaway billing and why there wasn't any simple options to cap a given resource to prevent those cases. His response was that they didn't build that because none of their customers wanted anything like that, as far as he was aware.

None of their Big Customers they meant, the small ones who worry about this doesn't matter.

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#77

Post by 827a »

gizzlon wrote: post #34
rvnx wrote: post #9

Earlier quoted context omitted.

Even regular Cloud Run can take a lot of time to boot (~3 to 30 seconds), so this can be a problem when scaling to 0

That's not my experience, using Go. Never measured, but it goes to 0 all the time, so I would definitely noticed more than a couple of seconds.

It depends on whether you're on gen1 or gen2 Cloud Run; the default execution environment is `default` which means "you have no idea because GCP selects for you" (not joking).

Counterintuitively (again, not joking): gen2 suffers from really bad startup speeds, because its more like a full-on linux VM/container than whatever weird shim environment gen1 runs. My Gen2 containers basically never start up faster than 3 seconds. Gen1 is much faster.

Note that gen1 and gen2 Cloud Run execution environments are an entirely different concept than first generation and second generation Cloud Functions. First gen Cloud Functions are their own thing. Second generation Cloud Functions can be either first generation or second generation Cloud Run workloads, because they default to the default execution environment. Believe it or not, humans made this.

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#78

Post by montebicyclelo »

mwest217 wrote: post #69
anonymousab wrote: post #59

Earlier quoted context omitted.

I remember going out to dinner, years ago, with a fairly senior AWS billing engineer. An acquaintance of a coworker. He looked completely surprised when I asked about runaway billing and why there wasn't any simple options to cap a given resource to prevent those cases. His response was that they didn't build that because none of their customers wanted anything like that, as far as he was aware.

Disclaimer: I work at Google but not on cloud. Opinions my own. I think the reason this doesn’t get prioritized is that large customers don’t actually want a “stop serving if I pass this limit” amount. If there’s a spike in traffic, they probably would rather pay the money to serve it. The customers that would want this feature are small-dollar customers, and from an economic perspective it makes less sense to priori…

I guess where it could potentially bring value is by:

Removing a major concern that prevents individuals / small customers from using GCP in the first place; so more of them do use it

That could then lead to value in two ways:

- They make small projects that go on to be large projects later, (e.g. a small app that grows / becomes successful, becomes a moneymaker)

- Or, they might then be more inclined to get their big corp to use GCP later on, if they've already been using it as an individual

But that's long term, and hard to measure / put a number on

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#79

Post by nprateem »

tmoertel wrote: post #71
brutus1213 wrote: post #47

Earlier quoted context omitted.

Amazon is the same I think? I live in constant fear we will have a runaway job one day. I get daily emails to myself (as a manager) and to my finance person. We had one instance where a team member forgot to turn off a machine for a few months :( I get why it is a business strategy to not have limits .. but I wonder if providers would get more usage if people had more trusts on costs/predictability.

> I get why it is a business strategy to not have limits... What is the strategy? Is is purely market segmentation? (As in: "If you need to worry about spending too much, you're not the big-money kind of enterprise customer we want"?)

It's not a strategy. It's technically difficult, opens them to liability if runaway happens so fast their system can't stop it, and is only wanted by bottom of the barrel customers.

Re: Cloud Run GPUs, now GA, makes running AI workloads easier for everyone

#80

Post by ivape »

ashishb wrote: post #54

I love Google Cloud Run and highly recommend it as the best option[1]. The Cloud Run GPU, however is not something I can recommend. It is not cost effective (instance based billing is expensive as opposed to request based billing), GPU choices are limited, and the general loading/unloading of model (gigabytes) from GPU memory makes it slow to be used as server less. Once you compare the numbers it is better to use a…

Maybe I just don't know, but I really don't think most people here can even point to a cloud GPU with 1000 concurrent users and not end up with a million dollar bill.
Post reply on HN