Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A 10 year old Xeon is all you need

point.free

Post reply on HN
Print view View original

211–220 of 301 posts

Re: A 10 year old Xeon is all you need

#211

Post by skdb476 »

phaser wrote: post #126
skdb476 wrote: post #41

Earlier quoted context omitted.

Its a convenience thing. You can run a whole lot of stuff locally from wikipedia to social media/email/video servers whatever. Most people with a full time job and 2 kids dont do it cause who has time and energy to patch and maintain the ever growing complexity of this stuff. These systems will keep growing complex. That also means more bugs. Age old tradeoff between freedom and convenience.

You can run mediawiki at home but you won't have wikipedia. You can run a video server but you won't have all the movies that Netfix has. A local model is actually the real thing.

you can have the whole wiki loaded with full search available locally. check out kiwix.

Re: A 10 year old Xeon is all you need

#212

Post by PLenz »

aorloff wrote: post #198
PLenz wrote: post #104

Earlier quoted context omitted.

This. OpenAI and Anthropic are ultimately compute infrastructure plays and not really AI. Everyone will have models, they'll have the ability to run them. This is why the GPU shortage is in their favor.

Maybe. But if we can all run our own model locally in 2 years on commodity hardware OpenAI and Anthropic will start to look like WeWork during the pandemic

I agree with you that they are headed in that direction! The GPU shortage is (I think) similar to the pandemic era hiring binge. It's less about the extra compute and more about denying the GPUs to potential competitors. They're racing against time to find something that gives them real moat (gen ai I guess?) and they are trading money for time.

This is also why the money being poured into datacenters isn't going to result in as much development as you think. It's about leveraging other people's money to lockdown more future hardware. This is going to end exactly like fiber build out in the 2000s. Eventually that fiber got used but the folks who originally paid for it got hosed.

Re: A 10 year old Xeon is all you need

#213

Post by jayd16 »

rmoriz wrote: post #201
aorloff wrote: post #198

Earlier quoted context omitted.

Maybe. But if we can all run our own model locally in 2 years on commodity hardware OpenAI and Anthropic will start to look like WeWork during the pandemic

And free model supply will stop…

I wonder if Google will put out a free model with the ads already baked in.

Re: A 10 year old Xeon is all you need

#214

Post by Aurornis »

llama.cpp includes a benchmarking tool called llama-bench https://github.com/ggml-org/llama.cpp/blob/master/tools/llam...

ik_llama includes llama-sweep-bench https://github.com/ikawrakow/ik_llama.cpp/blob/main/examples...

When comparing hardware, the output of these tools is very helpful to let others put it into context. The post says the output is "reading speed" but knowing the prefill and token generation speeds would be a lot more helpful.

Re: A 10 year old Xeon is all you need

#215

Post by PLenz »

rmoriz wrote: post #202
PLenz wrote: post #119

Earlier quoted context omitted.

Companies won't but I suspect this is a role that something else open source-y will fill that niche. Maybe orgs like wikimedia or internet archive, maybe some hackers just making things, maybe nation states that want to disrupt other players. Also model training will get better and better both on the algo and the hardware side. You can easily see a world where you might be able to train a good enough model on a home…

But you will need training data. Like a whole Internet search engine or massive data scraping. That‘s a thing that will not change with better algorithms, hardware or cheaper energy.

Data is the only moat but they'll be starting in the same place the current set of players statyed out just a few years ago. I suspect that the delta between what is publicly available (if not legally publicly available! see scihub) and what open ai and anthropic have is relatively small.

Re: A 10 year old Xeon is all you need

#216

Post by duffyjp »

gdjdhdheb wrote: post #36
cafkafk wrote: post #2

Hi HN. I wrote this post after getting frustrated by the lack of ways to run the new Gemma 4 Drafter models, and mainstream tools not prioritizing this, and hiding all the performance levers. I ended up getting a modern 26B MoE model (Gemma 4) running at reading speed on an old recycled server with a single Xeon E5-2620 v4 and 128GB of DDR3 RAM (and no GPU). It took a lot of work, but it actually worked out somehow.…

You sure you got DDR3 .. I have 2 e5 v4 rigs at home and both have ddr4 ... Unless I am wrong and 2011-3 supports ddr3 and ddr4

I won't speak for cafkafk, but I have two E5 (v3/v4) systems one on DDR4 and one on DDR3. This generation of CPU all support DDR4, but a few skus do support DDR3 also. ChatGPT told me they were niche products to meet specific customer needs.

I just picked up the DDR3 board, an Aliexpress "XD3" so I could reuse some DDR3 ram on a better CPU. Quad channel 1866MT/s is not bad!

Re: A 10 year old Xeon is all you need

#217

Post by tomega2134 »

I wish this were somehow tagged with AI, so I would know that it's not about say, general computing or cost-efficiency (e.g. using an old xeon machine from ebay instead of new, in these cost-conscious times.)

As it is, the title is click-bait for me, as 1) it says I need at least a Xeon somehow and 2) as it doesn't say what I actually need it for.

Re: A 10 year old Xeon is all you need

#218

Post by ThatMedicIsASpy »

dangus wrote: post #129
RetroTechie wrote: post #106

Earlier quoted context omitted.

How many kWh to fabricate a brand new machine better suited to the task? As long as performance is useable (apply your own metrics!), pulling it from existing hardware is likely the option with the lower eco footprint. Also: chances are it'll only be used for this purpose occasionally, and/or for a short while. In that scenario [fabricating new hardware] always has the bigger eco footprint.

I don’t know why you’d assume that an older system is lower footprint. If you’ve got something consuming 100 watts average over your 24 hour period, and your electricity costs 20 cents per kWh, you’re already spending almost as much as a Claude subscription. Just on electricity, this assumes your hardware never fails and you never incur any additional costs. There’s a big reason why newer more efficient hardware is i…

The reason more performance/watt is in demand because a datacenter can't suddenly draw twice as much power.

Re: A 10 year old Xeon is all you need

#219

Post by observationist »

trollbridge wrote: post #150
zozbot234 wrote: post #131

Earlier quoted context omitted.

Phones and laptops are terrible devices for local AI, way too constrained by bad thermals and small batteries. MiniPC's (many of them using mobile hardware) don't have that particular issue, and can easily run on a 24/7 basis.

Phones are also a terrible place to run a radio, but there's a huge amount of benefit in figuring out how to do so.

That level of local AI is also more or less what you need for competent autonomous robots, too. If your household robots are orchestrated from your phone, the local security and cloud convenience converge on a single device. No extra servers, etc, reduced cost, all that - local AI is a massive market amplifier.

Re: A 10 year old Xeon is all you need

#220

Post by smilespray »

bflesch wrote: post #168
s20n wrote: post #79

Earlier quoted context omitted.

I agree with the first part. I think this article by FSF about Intel's ME summarizes the issue https://static.fsf.org/nosvn/blogs/Intel_ME_Carikli_article_... As for the second part, I am not sure about how living in a five eyes state would mitigate it. What do you mean by that?

As five eyes citizen you have at least some rights on paper and you can appeal to your government, but if you are foreigner these guys can go gloves off without any fear of retribution. Try analyzing Epstein files and posting about it, they'll give you a proper penetration test of all your devices to see what you found out about their ex employee. Nowadays even EU citizens migrating away from US cloud providers are a…

Isn't the whole five eyes argument moot because member states spy on citizens from the other countries and trade intel with each other?
Post reply on HN