Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Alternative clouds are booming as companies seek cheaper access to GPUs

techcrunch.com

Post reply on HN
Print view View original

241–250 of 336 posts

Re: Alternative clouds are booming as companies seek cheaper access to GPUs

#241

Post by amluto »

hot_gril wrote: post #228
amluto wrote: post #186

Earlier quoted context omitted.

There’s a huge middle ground between on-prem and GCP/AWS. You can rent space and connectivity from in very competent datacenter without any of these big fixed costs.

Can rent the space, but you still have to buy the hardware. Maybe there's money to be made running some low-availability cloud service offering newer hardware.

Have you checked the price for a system capable of using two redundant 10Gbps links lately? It’s cheap. You could put gear like this in your closet at home and not feel particularly silly about it, especially if you are willing to buy still-current used enterprise gear.

For that matter, have you checked the price, in qty 1, of a server that will absolutely destroy anything reasonable from a major cloud vendor in terms of IOPS to stick behind that switch or router? Even if you believe the numbers on the website of a major server vendor and forget to ask for a discount, it’s still quite reasonable in comparison to a major cloud.

Re: Alternative clouds are booming as companies seek cheaper access to GPUs

#242

Post by s1artibartfast »

krisoft wrote: post #184
doctorpangloss wrote: post #52

Earlier quoted context omitted.

People complain about this relentlessly, but never change. Additionally, every single business owner I know complains that whatever it is they're selling, (1) it isn't worth the brain damage to sell to customers looking for the lowest price, (2) as long as people comparison shop in a harebrained way, hook pricing (aka up front pricing that looks low and turns out high) is only rational. It's not like they're providin…

> People complain about this relentlessly, but never change What do you mean by this? How would “people change” to get out of resort fees or confusing pricing systems?

people could stop buying from sellers with confusing pricing systems.

People are capable of change.

Re: Alternative clouds are booming as companies seek cheaper access to GPUs

#243

Post by jonfromsf »

silverquiet wrote: post #239
rmah wrote: post #238

Earlier quoted context omitted.

Most Americans don't realize just how huge and extensive medicare/medicaid is. They cover 65mil and 88mil people respectively, including 40mil children. That's 153mil people out of 330mil population -- that's 46% of the USA's population. Total spending is $1.9T for both. A cost of $12,400/person covered. $1.9T is 7.5% of the total US GDP of $25.44T. Medicare/medicaid is a massive program. Bigger than social security…

In that case, why not just go all the way to Medicare for All?

Because with the current medical cost, it would be absolutely ruinous. The root problem is American health care is far too expensive. The entire system is massively wasteful and complex.

Re: Alternative clouds are booming as companies seek cheaper access to GPUs

#244

Post by silverquiet »

jonfromsf wrote: post #243
silverquiet wrote: post #239

Earlier quoted context omitted.

In that case, why not just go all the way to Medicare for All?

Because with the current medical cost, it would be absolutely ruinous. The root problem is American health care is far too expensive. The entire system is massively wasteful and complex.

Seems like unifying it under a single, already-extant federal system would greatly reduce administrative overhead though, don’t you think?

What’s funny is I’m probably one of the people whose paycheck is dependent on me not understanding that to paraphrase Upton Sinclair. When I started at my current employer, one of the devs straight up said the government should be doing what we do and we shouldn’t have a business.

Re: Alternative clouds are booming as companies seek cheaper access to GPUs

#245

Post by jiggawatts »

015a wrote: post #16

We're absolutely seeing the same thing even in non-GPU traditional servers. Here is an estimated monthly cost breakdown for a new GCP instance type, the n4-standard-2, in us-central1: 2 vCPU + 8 GB memory: $69.18 10 GB Hyperdisk Balanced: $0.80 3060 provisioned IOPS: $15.30 155 MB/s provisioned throughput: $6.20 Total: $91.48 Like, you can tear apart that $69/mo for 2vCPU + 8gb of memory, no problem. That's utterly i…

> start struggling over the next

They’re already struggling!

In the past, the three big clouds would deploy cutting-edge CPUs at scale ahead of general availability for ordinary rackmount servers.

Now?

The AMD EPYC 9004 series processors were announced over a year ago in March 2023, but are still trickling out as “preview” in selected regions in Azure. Similarly, Intel Xeon fourth-gen CPUs haven’t even been announced by Azure, but Intel is already shipping fifth-generation CPUs!

I suspect that up until a couple of years ago, the usage of public cloud was increasing at such a pace that the providers were buying a truckload of CPUs every six months, so they were keeping up with the latest tech.

They must have had new signups dry up as soon as interest rates went up, and they’re now milking their existing kit instead of expanding with new generation servers.

Re: Alternative clouds are booming as companies seek cheaper access to GPUs

#246

Post by ikiris »

TacticalCoder wrote: post #154
ikiris wrote: post #92

Earlier quoted context omitted.

If you think offering a cloud service is at all the same as offering someone a box in a rack, I invite you to compete in the space.

It's more than a box in a rack. These providers do actively monitor and fix these boxen. They can all be rebooted remotely as if you physically hit the button, you've got interfaces to access the machine as if you were logging in physically with a DB-9/RS232/ethernet/whatever console, etc. It's not just "space in a rack and you deal with the servers yourself and you come to fix them if they break". These companies kn…

You're missing the point. There's a massive difference between getting a box service, and getting a highly available regionally distributed service with a semblance of a SLA of bandwidth to anywhere on the planet. To quote a former manager, its not even apples and oranges, is apples and pumpkins. They simply aren't in any way the same scope.

Re: Alternative clouds are booming as companies seek cheaper access to GPUs

#247

Post by hinkley »

jedberg wrote: post #19
freeqaz wrote: post #2

How are these companies even getting these GPUs? I would imagine NVIDIA would give them all to Microsoft and Google if they are in short supply.

I head up Product at Lambda. We are an NVIDIA preferred partner (and keep winning the preferred partner award). I don't know what the allocation numbers are for other companies, but we get a lot. It's in NVIDIAs best interest to spread the love around and make sure all the GPUs don't go to the hyperscalers.

I was at a Christmas party a couple jobs ago and they did a presentation that was supposed to be all good news and instead basically ruined the party for me.

The bit that freaked me out was 2 customers were 60% of revenue.

Those two customers can change your policies in ways you don't want to, because you can't afford to piss them off. And if something happens to one of them you're fucked.

Re: Alternative clouds are booming as companies seek cheaper access to GPUs

#248

Post by hot_gril »

amluto wrote: post #241
hot_gril wrote: post #228

Earlier quoted context omitted.

Can rent the space, but you still have to buy the hardware. Maybe there's money to be made running some low-availability cloud service offering newer hardware.

Have you checked the price for a system capable of using two redundant 10Gbps links lately? It’s cheap . You could put gear like this in your closet at home and not feel particularly silly about it, especially if you are willing to buy still-current used enterprise gear. For that matter, have you checked the price, in qty 1, of a server that will absolutely destroy anything reasonable from a major cloud vendor in ter…

Yeah, tends to be. But it's more efficient for multiple customers who don't need the hardware full-time to share it. Someone could set that up without all the expensive HA guarantees and other stuff a regular cloud provides. Maybe was too niche in the past, but now with the AI boom...

Re: Alternative clouds are booming as companies seek cheaper access to GPUs

#249

Post by indigodaddy »

dangrairo wrote: post #211
Palmik wrote: post #130

Earlier quoted context omitted.

It's horrible horrible pricing! Their on demand price for A100 is what gets you H100 sxm in other places.

I am curious what "other places" are you comparing it to.

Also we’d have to do apples to apples right? One cannot complain about fly.io gpu pricing and then point to say vast.ai or lambda labs as “evidence” of that. They aren’t at all the same type of service..

Re: Alternative clouds are booming as companies seek cheaper access to GPUs

#250

Post by indigodaddy »

benwaffle wrote: post #199
indigodaddy wrote: post #95

Earlier quoted context omitted.

Wow that is quite decent pricing actually. This makes me want to try to deploy something like llamafile/ollama or similar to fly.io+gpu for my personal on-demand llama/llm whims. (@simonw I’m looking at you— I feel like if you haven’t already done this on fly.io, that you’re probably thinking about it :) ) Seems private enough.. could throw some basic auth on top of it for me and trusted friends/family so I don’t get…

We've got a one-liner for spinning up your own ollama UI. See https://github.com/fly-apps/ollama-open-webui

Oh snap thank you!
Post reply on HN