Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A 10 year old Xeon is all you need

point.free

Post reply on HN
Print view View original

281–290 of 301 posts

Re: A 10 year old Xeon is all you need

#281

Post by trollbridge »

yard2010 wrote: post #248
observationist wrote: post #219

Earlier quoted context omitted.

That level of local AI is also more or less what you need for competent autonomous robots, too. If your household robots are orchestrated from your phone, the local security and cloud convenience converge on a single device. No extra servers, etc, reduced cost, all that - local AI is a massive market amplifier.

Let me speculate - we are going in the weird direction of no private property unless you're an overlord that rents his property to peasants. I like to call it the revenge of communism. See how the market behaves in the llm space - it's more viable to share infrastructure than to own it. Imagine the private car revolution in the US was a bus revolution.

We’ve been dreaming about this since the days of talking about wifi mesh networking, but it seems to never happen.

Re: A 10 year old Xeon is all you need

#282

Post by ranger_danger »

cmiles8 wrote: post #84

We’re not there yet, but the obvious endgame of the present bubble insanity is open models running on local hardware and devices are “good enough” for most use cases. That will completely implode what’s going on at the moment in tech.

Given the current performance requirements for "good enough for most people", I just don't see that happening any time soon.

Most users (potential or actual) are not on a desktop and don't have a beefy discrete GPU. There are "NPU" ASIC chips like what is being put in the new raspberry pi's but their performance and compatibility is not what you might think it is. To get GPU-like performance the ASIC would have to be closer to the size of a real GPU, and at that point why bother. And many devices just don't have the room.

Re: A 10 year old Xeon is all you need

#283

Post by asdfsa32 »

gowld wrote: post #228
asdfsa32 wrote: post #133

Earlier quoted context omitted.

The primary feature of a blog or any website is that it is available around the clock, that is the primary feature of cloud: around on the clock computer and network that scales on demand. The primary feature of "AI" is to process information and reason with a natural language interface at speed, the primary feature of AI bigboys is to provide the machinery that runs the "models". See the difference?

You severely underestimate how little the fraction of the performance and human labor of a frontier AI is in "the model". Hosting a blog 24x7 on a laptop is trivial, except for hyperscaling to the front page of HN and Reddit.

Yeah, exactly, hosting on a laptop is trivial except for when it is not. However, I am using an AI on a mac mini just fine, Qwen 3.6 27B at Q6. Works just as good as STOA models for most things.

Re: A 10 year old Xeon is all you need

#284

Post by benterix »

lazide wrote: post #234
benterix wrote: post #103

Earlier quoted context omitted.

> The AI companies will want to control what's possible and find new things to do that "need" their services. That's correct. The problem is they have smart people, tons of money, and several years to figure that out, and the best thing they can come up is a coding agent.

That isn’t the best thing they’ve come up with. It’s a marquee product that is fit for public consumption, however. The ‘best’ things are; - fuzzy pattern matching algorithms for traffic analysis, human and other image target recognition. - targeting algorithms that identify ‘suspicious’ individuals in large volumes of metadata. - fraud analysis - antagonistic image and video generation, both for fooling other fraud…

But you're mentioning several things that predate the current LLM craze and belong to the ML domain. These mostly benefit from GPUs but often have much lower hardware requirements. I'm talking specifically about the moat of LLM providers.

Re: A 10 year old Xeon is all you need

#285

Post by [deleted] »

overfeed wrote: post #278
m463 wrote: post #267

Earlier quoted context omitted.

I bought a renewed 2x E5-2690v4 server (28c/56t) 128gb on amazon for under $500 2 years ago (28c/56t) dell T7810 search amazon for "chia farming" ...and scroll past chia seeds :) now same machine is 2.5x the price https://www.amazon.com/dp/B095TRGCSX but way cheaper than current ddr5 machines

> now same machine is 2.5x the price 2.5x?! I have a bunch of older Haswell servers I got for free that are rotting away in my garage. I had initially thought of stripping out the ECC DDR4, but now I'm wondering if I'll get takers on Marketplace...

[deleted]

Re: A 10 year old Xeon is all you need

#286

Post by rbanffy »

The other day I was considering the adoption of a POWER7+ box. Sadly, Linux hasn't supported POWER7 in quite some time. The machine looked pretty nice, with 4 CPUs with 8 cores each, a total of 128 threads and 512 GB of RAM. I'm not sure it'd run AIX without a license though, which is unfortunate - it's a gorgeous box.

Re: A 10 year old Xeon is all you need

#287

Post by lazide »

benterix wrote: post #284
lazide wrote: post #234

Earlier quoted context omitted.

That isn’t the best thing they’ve come up with. It’s a marquee product that is fit for public consumption, however. The ‘best’ things are; - fuzzy pattern matching algorithms for traffic analysis, human and other image target recognition. - targeting algorithms that identify ‘suspicious’ individuals in large volumes of metadata. - fraud analysis - antagonistic image and video generation, both for fooling other fraud…

But you're mentioning several things that predate the current LLM craze and belong to the ML domain. These mostly benefit from GPUs but often have much lower hardware requirements. I'm talking specifically about the moat of LLM providers.

Sure, but all fall under the same marketing umbrella.

Ad/marketing manipulation are exceptionally well done with LLMs in particular.

If you asked someone if drone auto targeting/image recognition or data analysis was ‘AI’, 99% of the time they’ll say yes.

Re: A 10 year old Xeon is all you need

#288

Post by mantesso »

duffyjp wrote: post #225
throwaway2027 wrote: post #45

Glad to see other people realizing this. I've been running Gemma 26B-A4B Q4 on a 2012 Xeon with 16GB to 24GB of RAM in a container. It's getting around 8 to 12 tokens per second. Obviously it's not comparable to huge contexts and running it on a GPU and the image decoder in llama.cpp is super slow compared to a GPU but for some small automation tasks and general trivia questions it's decent. The speed is just enough…

I'm setting up a Frankenstein system at the moment. It's a Chinese DDR3 X99 motherboard with a 12 core Xeon v3, 32gb 1866MT/s ram, and a 1080 Ti. I'm shoehorning it back in the Optiplex that donated the ram, so it's not ready to go at the moment, but when I had it running on top of the motherboard box as a test I ran the (9B?) gemma4:e4b-it-q4_K_M since it can fit entirely in the 11gb vram. It flew , more than 50tk/s…

> I'd love to figure out a Wake-on-Use

if you have an openwrt router this is very easy to do. i have a script on my main working machine that will ssh openwrt and turn on the server and this work well

Re: A 10 year old Xeon is all you need

#289

Post by _zoltan_ »

lightedman wrote: post #269
_zoltan_ wrote: post #78

Earlier quoted context omitted.

right, and they talk about "v4" which is DDR4.

There were several V4 Xeon models that supported DDR3 AND DDR4 simultaneously. If you had a motherboard with an X79 chipset it would (sometimes) work properly.

I am not aware of any commercial vendor shipping v3/v4 boards with DDR3. I have a couple hundred Supermicro systems that are stuck on v2 CPUs with DDR3...

Re: A 10 year old Xeon is all you need

#290

Post by dark-star »

robeastham wrote: post #178
dark-star wrote: post #86

Earlier quoted context omitted.

Something doesn't add up here. As someone who has only recently built a home-server from an E5-26xx v2 on DDR3 RAM (because I have a sh*tload of 32g DDR3 DIMMs), I can confidently say that the newer cores (E5-26xx v3 and v4) only run on DDR4 memory... So either you have a v2 instead of a v4 (and run on DDR3 memory), or you have a v4 but with DDR4 memory (not DDR3) Everything else doesn't work

This is not true. A few well known brands made both DDR3 and DDR4 servers that support v3 & v4 chips. Ask me how I know :-)

crazy, I really did not know that. Do you happen to know if such boards also exist that take registered DDR3 RAM? None of them explicitly call out DDR3-R RAM so I assume they only take consumer RAM?
Post reply on HN