Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Alternative clouds are booming as companies seek cheaper access to GPUs

techcrunch.com

Post reply on HN
Print view View original

261–270 of 336 posts

Re: Alternative clouds are booming as companies seek cheaper access to GPUs

#261

Post by [deleted] »

ryandrake wrote: post #41
apitman wrote: post #22

Over the weekend a coworker was helping someone in our lab prepare data for an important conference talk. We accidentally ran up a rather large bill because while the EFS storage pricing was simple enough, the usage pricing bit us. It seems like AWS' entire business model is making the pricing so confusing that you don't know what it will cost until after you've used it. It feels weirdly similar to the US healthcare/…

> It seems like AWS' entire business model is making the pricing so confusing that you don't know what it will cost until after you've used it. It feels weirdly similar to the US healthcare/insurance situation. Not just the healthcare situation, but everything. There's nothing much more stereotypically American than "not knowing what you're going to pay for something until you're billed." Dozens of little fees on you…

[deleted]

Re: Alternative clouds are booming as companies seek cheaper access to GPUs

#262

Post by jeffbee »

freeqaz wrote: post #2

How are these companies even getting these GPUs? I would imagine NVIDIA would give them all to Microsoft and Google if they are in short supply.

CoreWeave does not exactly have an arms-length relationship with nvidia.

Re: Alternative clouds are booming as companies seek cheaper access to GPUs

#263

Post by [deleted] »

jonfromsf wrote: post #243
silverquiet wrote: post #239

Earlier quoted context omitted.

In that case, why not just go all the way to Medicare for All?

Because with the current medical cost, it would be absolutely ruinous. The root problem is American health care is far too expensive. The entire system is massively wasteful and complex.

[deleted]

Re: Alternative clouds are booming as companies seek cheaper access to GPUs

#264

Post by ryandrake »

cogman10 wrote: post #230
AnthonyMouse wrote: post #227

Earlier quoted context omitted.

> There is no reason you can't show final price including tax on the label in a physical shop. Sure there is. When the price label is affixed by the factory/warehouse then you would have to track where everything is going and be unable to share inventory. Also, if the sales tax rate changes then all the labels become wrong. These would ultimately increase costs for consumers. Adding sales tax is also not at all misle…

> When the price label is affixed by the factory/warehouse then you would have to track where everything is going and be unable to share inventory. Price tags are pretty much universally handled at the stores. There are some goods where the price tag is attached, but that's more the exception and not the rule. This is a solvable problem. So much so that if you've traveled in most nations you'll see that all prices in…

Yep, this is one of those "This problem can't be solved - Says the only nation that has this problem" situations.

Re: Alternative clouds are booming as companies seek cheaper access to GPUs

#265

Post by zackangelo »

nextworddev wrote: post #88
zackangelo wrote: post #46

Would like to take a moment to recommend fly.io for GPU workloads. I've been building a prototype using them for the last couple of weeks and it's been great to use. I didn't have to jump through any hoops or apply for any quota adjustments to get started. And I especially appreciate how easy they make it to automatically scale your GPU instances to zero based on traffic.

Taking a step back, was it necessary to roll your own LLM (or whatever FM) API as opposed to using an off the shelf API

Yes we’re running our own inference server because we do nonstandard things in the decoder loop (think logit masking, other types of sampling, etc).

Re: Alternative clouds are booming as companies seek cheaper access to GPUs

#266

Post by ikiris »

exceptione wrote: post #256
ikiris wrote: post #246

Earlier quoted context omitted.

You're missing the point. There's a massive difference between getting a box service, and getting a highly available regionally distributed service with a semblance of a SLA of bandwidth to anywhere on the planet. To quote a former manager, its not even apples and oranges, is apples and pumpkins. They simply aren't in any way the same scope.

What do you mean? The big bare metal providers have multiple datacenters with fat pipelines and peering. You can put those geo-separated servers in the same subnet. Europe and US is covered, it looks like Asia a black spot for Hetzner, I will give you that. I doubt that so many business have such size and global reach, that world wide latency is a priority. If it would be, the average site would not connect to 20 dom…

Sure, go build your own less reliable cloud. But don't pretend that single box pricing is comparable to single cloud service as a result.

Re: Alternative clouds are booming as companies seek cheaper access to GPUs

#267

Post by thaumaturgy »

s1artibartfast wrote: post #240
thaumaturgy wrote: post #108

Earlier quoted context omitted.

I have begun to think of this as a "large population error", and I'm noticing it in a lot of different markets. At small scales, annoying your customers is bad business. You only need to lose a few before it begins to hurt. Customer complaints are more likely to be a consideration in business decisions. At larger scales, a business can begin to preferentially adopt practices intended to drive away some customers. Per…

The intuitive answer is competition, where customers move from user-hostile companies to new ones. The problem in my mind is that the largest companies have too much efficiencies of scale to compete with on price. When competition can't undercut on price, it is hard to argue that customers aren't being served by monopoly mega corps.

Yeah, this is a pretty standard answer to the same market failures that so many people have been talking about recently. It assumes though that low prices are the best possible benefit consumers might get from competition, and it also assumes that megacorps necessarily lead to the lowest possible prices for consumers.

Chokepoint Capitalism is a pretty okay book that kicks the legs out from under both assumptions.

Re: Alternative clouds are booming as companies seek cheaper access to GPUs

#268

Post by inkyoto »

apitman wrote: post #22

Over the weekend a coworker was helping someone in our lab prepare data for an important conference talk. We accidentally ran up a rather large bill because while the EFS storage pricing was simple enough, the usage pricing bit us. It seems like AWS' entire business model is making the pricing so confusing that you don't know what it will cost until after you've used it. It feels weirdly similar to the US healthcare/…

> It seems like AWS' entire business model is making the pricing so confusing that you don't know what it will cost until after you've used it.

This is patently not true. AWS is very transparent with the pricing and offers a cost calculator. Specifically for EFS, you could have used the EFS service cost calculator before setting off with your endeavours:

https://calculator.aws/#/createCalculator/EFS

Other services are available at https://calculator.aws/

Any work on any cloud platform has an imperative step: the cost analysis. Once constituents of the solution are finalised (step 1), it is compulsory to proceed to step 2: estimate the charges it is going to cost and decide whether it is affordable or not. Skipping the step 2 and blaming the cloud platform provider business model is not fair and is akin to overspending on a credit card and blaming the credit card issuer for allowing you to do it.

Cloud is not dissimilar from UNIX/Linux – both give one a large variety of elaborate footguns to shoot oneself with, and if they worked as root on a software development project on a UNIX/Linux box and accidentally wiped the root file system out, who is to blame? Since cloud charges are usage based («pay as you go»), the cost estimate is a requisite that can't be avoided unless the budget is unlimited.

Re: Alternative clouds are booming as companies seek cheaper access to GPUs

#269

Post by gjs278 »

inkyoto wrote: post #268
apitman wrote: post #22

Over the weekend a coworker was helping someone in our lab prepare data for an important conference talk. We accidentally ran up a rather large bill because while the EFS storage pricing was simple enough, the usage pricing bit us. It seems like AWS' entire business model is making the pricing so confusing that you don't know what it will cost until after you've used it. It feels weirdly similar to the US healthcare/…

> It seems like AWS' entire business model is making the pricing so confusing that you don't know what it will cost until after you've used it. This is patently not true. AWS is very transparent with the pricing and offers a cost calculator. Specifically for EFS, you could have used the EFS service cost calculator before setting off with your endeavours: https://calculator.aws/#/createCalculator/EFS Other services ar…

[dead]

Re: Alternative clouds are booming as companies seek cheaper access to GPUs

#270

Post by metadat »

phh wrote: post #34
015a wrote: post #16

We're absolutely seeing the same thing even in non-GPU traditional servers. Here is an estimated monthly cost breakdown for a new GCP instance type, the n4-standard-2, in us-central1: 2 vCPU + 8 GB memory: $69.18 10 GB Hyperdisk Balanced: $0.80 3060 provisioned IOPS: $15.30 155 MB/s provisioned throughput: $6.20 Total: $91.48 Like, you can tear apart that $69/mo for 2vCPU + 8gb of memory, no problem. That's utterly i…

You feel like it's new, I don't. I've evaluated migrating from full blown servers to cloud at various scales for various usages: OTA server for 1M active devices, stat server for 1M active devices, build server for Android (both pure Android that parallelizes and checks out super nearly and vendor Android where they broke all of that). In all those cases the cost of cloud was like an order of magnitude costlier. You'…

> In all those cases the cost of cloud was like an order of magnitude costlier.

Don't discount the value-add of your skills. Giving any of my family members a stack of bare metal servers would serve no end-purpose or web requests. Approximately zero of said servers would even end up plugged in at all, much less operationalized. My family all works in tech or similarly demanding fields. That is, the cost of "the ops team / guy" can be significant for small and medium enterprises.

Despite this, the AWS pricing margins have grown to the point of excess, and are no longer competitive, even against other major cloud players. This is strange to me because the tooling is all interoperable (e.g., Terraform/Vaggi-form), with no fatal friction or lock-in. It is a rare use-case that the latest CPU is worth paying a penny extra for compared to a 3 or 4 generation old chip. The only winner is the cloud provider because it improves their COGS ratio.

TL;DR: the top tier clouds are priced on the order of luxury goods, on par with a Bugatti or G6 jet. That is, unless you ruthlessly track and prune each expense, which costs you time and attention which could have been spent growing the business or at least non-overhead tasks. The eventuality of managing your own DIY fleet of machines is a total headache, as any given computer may work flawlessly for the next ten years or only the next 10 minutes. When it goes south, you're back to being a monkey plugging in cables and scratching your head. This sad activity is only a few people's cup of tea.

Obligatory reference to Warren G's Regulators:

"You've gotta be handy with the steel, if you know what I mean."

https://youtube.com/watch?v=hms5vmekId4

Edit: Sorry for the rambling comment, I suppose it is a more complex topic than I realized prior to crafting the words above.

Post reply on HN