Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A 10 year old Xeon is all you need

point.free

Post reply on HN
Print view View original

91–100 of 301 posts

Re: A 10 year old Xeon is all you need

#91

Post by IdiotSavage »

cmiles8 wrote: post #84

We’re not there yet, but the obvious endgame of the present bubble insanity is open models running on local hardware and devices are “good enough” for most use cases. That will completely implode what’s going on at the moment in tech.

I find that hard to believe. The AI companies will want to control what's possible and find new things to do that "need" their services. Otherwise it would be like Intel and Microsoft had decided in the year 2000 that computers are "good enough" now and we would have explored what's possible with that hardware ever since.

Re: A 10 year old Xeon is all you need

#92

Post by wazoox »

fooker wrote: post #44
wazoox wrote: post #24

Earlier quoted context omitted.

I've been running various models on a Mac Pro 2013 (8 cores, 32 GB RAM) at about 8 to 10 t/s for months. It's not fast, but it's more than enough for many actual tasks, in particular background tasks. An iMac pro will do just as well I suppose.

What are the tasks that do well with 8-10 t/s ?

The sort of task you don't expect to end immediately. If extracting data from a bunch of PDFs takes 1 hour or the whole night, that doesn't make much difference to me. It's not fast enough for auto completion and slightly too slow for chat (but bearable IMO).

Re: A 10 year old Xeon is all you need

#93

Post by Kinrany »

herval wrote: post #89
cmiles8 wrote: post #84

We’re not there yet, but the obvious endgame of the present bubble insanity is open models running on local hardware and devices are “good enough” for most use cases. That will completely implode what’s going on at the moment in tech.

this is sorta like saying that being able to run your blog on your laptop will completely implode the cloud business

More like implode proprietary blog hosting platforms and replace them with commodity VMs that can be used for blog hosting, among other things

Re: A 10 year old Xeon is all you need

#94

Post by naasking »

deng wrote: post #54

Nice post and technically impressive work. I agree we need to understand the build pipeline and be able to do things locally. However, depending on your electricity cost, it might not make sense financially. These old servers are not energy efficient at all (I'm guessing that old Xeon server will easily pull 200W on load), and that model is currently at 0.1$/0.3$ per 1M tokens (with 76 tps and 262k context) in Openro…

These servers are loud if you're trying to fit them into a 1U or 2U, which requires high speed fans to generate the necessary static pressure to push air through the case. I run a similar setup in a 4U case with slow 120mm fans and it's fine.

Re: A 10 year old Xeon is all you need

#95

Post by dvdkon »

lreeves wrote: post #85

Doesn't accepting 100% of the MTP draft tokens mean you should just be using the smaller model? Usually the acceptance rate in Qwen36 at least is around 60-70% and the "wrong" tokens are still filled in entirely by the base model, but when you just accept 100% of the draft tokens it seems kind of self defeating unless I'm wrong. Also I feel like everyone leaves off prompt processing/prefill speeds in these articles.…

As far as I know, speculative decoding still verifies that the proposed tokens are what the "big" model would generate, it just uses the guesses to make that process faster. Setting the probability threshold too low then shouldn't affect correctness, just speed (time will be wasted verifying bad guesses).

Re: A 10 year old Xeon is all you need

#96

Post by grumpymuppet »

herval wrote: post #89
cmiles8 wrote: post #84

We’re not there yet, but the obvious endgame of the present bubble insanity is open models running on local hardware and devices are “good enough” for most use cases. That will completely implode what’s going on at the moment in tech.

this is sorta like saying that being able to run your blog on your laptop will completely implode the cloud business

It's a little different because cloud and blogs didn't actively get in the way of your home compute. To wit, the various cost spikes for hardware.

People -- WANT -- this technology on their home devices and (apparently?) the providers of this tech don't seem to be running a profit so they probably don't want the maintenance tail on their side either.

I think it's a bit different. Inevitable that this becomes a household-run thing? Not likely.

Re: A 10 year old Xeon is all you need

#97

Post by montroser »

Result is ~12 tokens per second, as reported by OP down in these comments here.

An impressive effort, and better than I would have thought possible on this hardware -- but still pretty far short of what one needs for an satisfactory interactive session.

Re: A 10 year old Xeon is all you need

#98

Post by benterix »

cmiles8 wrote: post #84

We’re not there yet, but the obvious endgame of the present bubble insanity is open models running on local hardware and devices are “good enough” for most use cases. That will completely implode what’s going on at the moment in tech.

I wouldn't say "completely implode", too much money was poured int it, but it's clear we're heading in that direction. You get a model that is "good enough", plus privacy, plus savings in the long term.

Paradoxically, the better results we get from general harness of coding agents, the less moat Claude and co. get. It's unbelievably how fast some open models outpaced frontier models of just a few months ago.

Re: A 10 year old Xeon is all you need

#99

Post by naasking »

lreeves wrote: post #85

Doesn't accepting 100% of the MTP draft tokens mean you should just be using the smaller model? Usually the acceptance rate in Qwen36 at least is around 60-70% and the "wrong" tokens are still filled in entirely by the base model, but when you just accept 100% of the draft tokens it seems kind of self defeating unless I'm wrong. Also I feel like everyone leaves off prompt processing/prefill speeds in these articles.…

It depends on the type of MTP. If you're using two models, draft + full, then arguably yes, the larger model isn't providing much benefit if you really are seeing 100% acceptance rates. There are other forms of speculative decoding that work within the larger model by itself though, eg. Qwen has additional speculative decoding attention heads, so there is no secondary drafting model.

Re: A 10 year old Xeon is all you need

#100

Post by ciupicri »

jansommer wrote: post #55
deng wrote: post #54

Nice post and technically impressive work. I agree we need to understand the build pipeline and be able to do things locally. However, depending on your electricity cost, it might not make sense financially. These old servers are not energy efficient at all (I'm guessing that old Xeon server will easily pull 200W on load), and that model is currently at 0.1$/0.3$ per 1M tokens (with 76 tps and 262k context) in Openro…

It should be closer to 85W on load. And it's incredibly silent on even a low end cooler. I rarely get above 50° Celcius.

85W for the whole system?! The specifications for the CPU mention a TDP of 85W [1].

[1] https://www.intel.com/content/www/us/en/products/sku/92986/i...

Post reply on HN