Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apple Silicon costs more than OpenRouter

williamangel.net

Post reply on HN
Print view View original

231–240 of 322 posts

Re: Apple Silicon costs more than OpenRouter

#231

Post by PunchyHamster »

ericpauley wrote: post #152
odo1242 wrote: post #150

Earlier quoted context omitted.

Yea this; it’s the same reason why mortgaging is cheaper than renting

This is far from a universal truth: https://www.nytimes.com/interactive/2024/upshot/buy-rent-cal... Real estate is only a clearly good investment if you ignore opportunity cost.

It is very close to universal truth, aside some small areas with very warped market.

Even if you move out after 5 years, you still own the place and can rent it out and then it pays for itself, to skip the cost of selling it back to market

Re: Apple Silicon costs more than OpenRouter

#232

Post by macwhisperer »

I run the latest 20b-30b models on a MacBook Air... running inference with an MoE (25 tps) for like 2 hours is like 10% battery.. (look me up on huggingface to download my models)

also you gotta realize frontier models have massive "system prompts" that clog up the context window with garbage.

being able to write your own system prompts gives you a MASSIVE edge..

Re: Apple Silicon costs more than OpenRouter

#233

Post by bastawhiz »

PunchyHamster wrote: post #229
bastawhiz wrote: post #14

This isn't a good analysis, and it's because it keeps rounding everything up. He rounds up the cost of electricity by 10%. He has a range of power use, takes the high end (which is 2x the low end) and multiplies it by the inflated electricity cost. But then they talk about using a newly purchased Mac to do the inference, running at full capacity, 24/7. Why would you do that? Apple silicon is fast but the author point…

> Yeah, data centers don't pay residential electricity rates. There are 2 caveats here: Some places have higher prices for industrial than residential power as residential one might be subsidied by govt. And DC also pay for cooling, which residential will only effectively pay if they have AC and is hot outside. So power rates are some multiply of industrial pricing.

Generally you don't build a data center in a place that doesn't sell you electricity for cheap

Re: Apple Silicon costs more than OpenRouter

#234

Post by PunchyHamster »

orangecat wrote: post #215
sgt wrote: post #210

Earlier quoted context omitted.

Surely though, the HOA and all that would likely be baked into a renter's price. And you didn't need to go live in a HOA. I don't, and it's much cheaper.

Surely though, the HOA and all that would likely be baked into a renter's price. Sure, the same way that the benefits of a fixed mortgage payment are baked into sale prices. The efficient market hypothesis would say that neither renting nor buying should be obviously superior in the long term, because if either was then people would bid up rents/prices until it wasn't. And you didn't need to go live in a HOA I pretty…

> The efficient market hypothesis would say that neither renting nor buying should be obviously superior in the long term, because if either was then people would bid up rents/prices until it wasn't.

Buying have much higher entry point, need a bunch of cash at start then a ton of paperwork.

It is absolutely possible that local buying market is inflated precisely because the area is so desirable buying to rent is (or was) good investment, but that's rarely is true for a bigger market

Re: Apple Silicon costs more than OpenRouter

#235

Post by bastawhiz »

dist-epoch wrote: post #161
bastawhiz wrote: post #55

Earlier quoted context omitted.

That's the point: why would you buy a device that's specifically not optimized to be used for 24/7 inference? It's expensive hardware that's not designed to be used in that situation! The power use for inference isn't especially good and you're not getting even a fraction of the benefit from the hardware that you're paying for.

> why would you buy a device that's specifically not optimized to be used for 24/7 inference because it costs $1k-$2k instead of $10k-30k+ for optimized devices

Nobody is suggesting you buy a pair of A100s, which is what 15k gets you these days. Get a used 5090. And the author specifically priced the hardware at over 4k, which is double the 1-2k you're noting

Re: Apple Silicon costs more than OpenRouter

#236

Post by bastawhiz »

groundzeros2015 wrote: post #39
dist-epoch wrote: post #28

Earlier quoted context omitted.

using it 24/7 brings the average cost down, not up. the less you use local LLM, the less sense it makes since you paid a lot for hardware you don't use

The hardware has multiple uses for the same cost. The pay-per-use server does not.

The author isn't pricing in the multiple uses. You either compare it apples to apples or you don't. If you're using the machine for general purpose computing on top of inference then the amortized hardware costs are pointless to measure. This is exactly what I said.

Re: Apple Silicon costs more than OpenRouter

#237

Post by shaewest »

ericpauley wrote: post #152
odo1242 wrote: post #150

Earlier quoted context omitted.

Yea this; it’s the same reason why mortgaging is cheaper than renting

This is far from a universal truth: https://www.nytimes.com/interactive/2024/upshot/buy-rent-cal... Real estate is only a clearly good investment if you ignore opportunity cost.

Real estate is generally a "good" investment as it's considered a relatively safe way to get significant leverage. 5x leverage in the case of a 20% deposit, or even up to 20x leverage with countries that allow for 5% deposits (New Zealand).

In addition, the interest payments almost always end up being near the rent the owner would have paid, so mortgage payments are higher, but that increase is generally (and quickly becomes) principal while being able to counteract inflation of rent.

Re: Apple Silicon costs more than OpenRouter

#238

Post by bastawhiz »

cyanydeez wrote: post #151
bastawhiz wrote: post #56

Earlier quoted context omitted.

Whether you think building data centers or not is a good idea it's inarguable that the per-token efficiency (power, hardware, etc) is FAR higher in a data center. That's literally what it's designed for.

im talking per value. look at the efgiency of chinese open source models; then look at SOTA sucking gigawatts, then the proposals. America is basically proposing AI using the equivalent bloatware of Windows 11.

I run two 49B parameter models on a pair of used A100s full time and it sucks down 250 watts at peak utilization. That's not gigawatts, and it's completely within the realm of comparison to what the author is describing.

Re: Apple Silicon costs more than OpenRouter

#239

Post by bastawhiz »

trollbridge wrote: post #86
cyanydeez wrote: post #38

Earlier quoted context omitted.

nothing about the current data center craze looks efficient.

Probably because lots of data centres are being built (or half-built) which are sitting idle.

If you have racks of idle H100s, you are doing a terrible job of running a business.
Post reply on HN