Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Alternative clouds are booming as companies seek cheaper access to GPUs

techcrunch.com

Post reply on HN
Print view View original

131–140 of 336 posts

Re: Alternative clouds are booming as companies seek cheaper access to GPUs

#131

Post by claytonjy »

What are folk's experiences with alternative cloud GPUs for inference?

If you're doing a lot of model training, buying GPUs or long-term reservations of GPUs is a no-brainer. But when it comes to inference, latency matters and it gets trickier talking between e.g. your AWS infra and your GPUs somewhere else.

It seems lots of providers can give you enough to get by doing inference in a company's earliest stages. But what if I need hundreds or thousands of A100s during peak usage? Is anyone doing this successfully with a non-hyperscaler?

Re: Alternative clouds are booming as companies seek cheaper access to GPUs

#132

Post by jerrygenser »

rootedbox wrote: post #79

@ Core-weave an A100 40GB NVLink is $2.06 ... the only way they are doing this is by burning investor money. At that price it running it 24/7 it will take way over a year to recoup hardware + electricity cost. So my suggestion.. dump all your work @ Core-weave.. It's cheaper than buying the hardware yourself let alone the cost of managing it.

Does this assume retail price? Maybe they are getting a discount for buying in bulk

Re: Alternative clouds are booming as companies seek cheaper access to GPUs

#133

Post by cyberax »

zb3 wrote: post #120
ikiris wrote: post #89

Earlier quoted context omitted.

Was it "I didn't understand it and/or put in the effort to understand it first" ? Nah it must be "theres a conspiracy to make it confusing so we pay more" AWS has one of the nicer to comprehend billing systems.

Is there a pre-paid option?

Yes. AWS Advance Pay.

Re: Alternative clouds are booming as companies seek cheaper access to GPUs

#134

Post by Palmik »

jedberg wrote: post #19
freeqaz wrote: post #2

How are these companies even getting these GPUs? I would imagine NVIDIA would give them all to Microsoft and Google if they are in short supply.

I head up Product at Lambda. We are an NVIDIA preferred partner (and keep winning the preferred partner award). I don't know what the allocation numbers are for other companies, but we get a lot. It's in NVIDIAs best interest to spread the love around and make sure all the GPUs don't go to the hyperscalers.

Are there any plans to support your on demand cloud in the future? In the past few months there are basically no GPUs available.

My understanding is that your reserved cloud starts at 64 GPUs (8 pods) and with quite a long reservation time frame.

Re: Alternative clouds are booming as companies seek cheaper access to GPUs

#135

Post by TeMPOraL »

ryandrake wrote: post #41
apitman wrote: post #22

Over the weekend a coworker was helping someone in our lab prepare data for an important conference talk. We accidentally ran up a rather large bill because while the EFS storage pricing was simple enough, the usage pricing bit us. It seems like AWS' entire business model is making the pricing so confusing that you don't know what it will cost until after you've used it. It feels weirdly similar to the US healthcare/…

> It seems like AWS' entire business model is making the pricing so confusing that you don't know what it will cost until after you've used it. It feels weirdly similar to the US healthcare/insurance situation. Not just the healthcare situation, but everything. There's nothing much more stereotypically American than "not knowing what you're going to pay for something until you're billed." Dozens of little fees on you…

Maybe that's why credit cards are so extremely popular in the US? I suppose if you never know how much you're going to pay for anything, it helps to have a buffer between the vendor and your bank account, allowing you to review the charges after the fact, and should you dispute some, be on a somewhat even playing field with the vendor.

Re: Alternative clouds are booming as companies seek cheaper access to GPUs

#136

Post by Palmik »

nothercastle wrote: post #59
autoexecbat wrote: post #20

Earlier quoted context omitted.

At some point, nvidia does have to worry about amd getting their act together once they've annoyed all the current customers

Amd will never figure out drivers and software stuff. That’s always been their weakness

MI250 and MI300 are competitive for inference and training. The problem is that there's no cloud with simple registration that offers these for reasonable on demand price. This means that basically no one in the open source AI community can use them, so most development and tooling is being tested and built for NVIDIA.

Re: Alternative clouds are booming as companies seek cheaper access to GPUs

#137

Post by silverquiet »

ryandrake wrote: post #41
apitman wrote: post #22

Over the weekend a coworker was helping someone in our lab prepare data for an important conference talk. We accidentally ran up a rather large bill because while the EFS storage pricing was simple enough, the usage pricing bit us. It seems like AWS' entire business model is making the pricing so confusing that you don't know what it will cost until after you've used it. It feels weirdly similar to the US healthcare/…

> It seems like AWS' entire business model is making the pricing so confusing that you don't know what it will cost until after you've used it. It feels weirdly similar to the US healthcare/insurance situation. Not just the healthcare situation, but everything. There's nothing much more stereotypically American than "not knowing what you're going to pay for something until you're billed." Dozens of little fees on you…

I have to say that my response to "not knowing what you're going to pay for something until you're billed" is basically, "well I guess we'll see if I can pay it then". You sort of start asking yourself, "what if I just don't pay?". I have a (deadbeat) buddy who's entire medical plan is essentially to give a fake name to the ER, and if you're poor in America, what else can you do?

Re: Alternative clouds are booming as companies seek cheaper access to GPUs

#138

Post by thundergolfer »

maxchehab wrote: post #110

Both Core Weave & Lambda Labs have fairly predatory pricing making it impossible to rent GPUs without a yearly contract. This doesn't make sense for training models, where a training run is on the scale of days & weeks. I wished that the techcrunch article mentioned other companies, like sfcompute, which offer hourly compute instead of yearly contracts.

Lambda Labs has on-demand GPUs. Just put in a credit card and you're able to launch. I launched an 1x H100 server just 10 minutes ago on Lambda Labs.

The price is also $2.49/hr which does not seem predatory at all.

Re: Alternative clouds are booming as companies seek cheaper access to GPUs

#140

Post by margorczynski »

dinvlad wrote: post #50
zeroxfe wrote: post #44

Earlier quoted context omitted.

There's no way you're getting a machine for $40/month. :-) Did you mean a dedicated VM or VPS? (I have a bunch of actual dedicated machines with different providers, and this would save me a lot of money.) (Edit: holy moly those prices are fantastic!)

Yeah, Hetzner has 20 vCPU 64GB RAM i5-13500 servers for $40/mo: https://www.hetzner.com/dedicated-rootserver/matrix-ex/ This is ~10x cheaper than the closest AWS option, and without the extra fees. Also of note that Hetzner is profitable, which means AWS has been operating at an insane markup.

And this is just for the "dumb" EC2 instance, the markup on their "smarter" stuff is probably much higher. In general I understand why one would want to start off in the cloud but staying there for 10+ years is quite absurd given the costs.
Post reply on HN