Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A 10 year old Xeon is all you need

point.free

Post reply on HN
Print view View original

261–270 of 301 posts

Re: A 10 year old Xeon is all you need

#261

Post by wazoox »

fooker wrote: post #190
wazoox wrote: post #92

Earlier quoted context omitted.

The sort of task you don't expect to end immediately. If extracting data from a bunch of PDFs takes 1 hour or the whole night, that doesn't make much difference to me. It's not fast enough for auto completion and slightly too slow for chat (but bearable IMO).

Running a local llm at 10 t/s overnight to extract data from a few PDFs will burn more in electricity than paying cents for the hosted kimi models. You can (sometimes) break even if you have a workstation GPU.

Sometimes data privacy is paramount.

Re: A 10 year old Xeon is all you need

#262

Post by SXX »

segmondy wrote: post #200
SXX wrote: post #35

Now we need someone try run Kimi K2.6 on old Xeon and DDR3. After all these platforms do support up to 768GB RAM.

You can run these on a turing machine. At what point is it not worth it? At some point the energy to generate each token matters. We often seen token per second. I think a missing metric is tokens per kilowatt. That is what really matters.

This is just like running Crysis via software rendering on CPU / llvmpipe. It dont have to be practical in order to be fun to try.

Re: A 10 year old Xeon is all you need

#263

Post by inetknght »

throwaway2037 wrote: post #65
jansommer wrote: post #52

The E5-2620 v4 is great. Have been using it for 10 years now. Wanted to upgrade until I saw current prices. I have 64 GB ddr4. Paired it with rx 9060 xt 16 GB and games run as fast as ever. Perhaps the cpu is a slight bottleneck in DOOM The Dark Ages, but i'm at 60 fps, so no problem. Light llm on the gpu is a nobrainer, and it's cool to see that things can be tuned to run ok on the cpu. I bought 2667 v4 a month ago…

> The E5-2620 v4 is great. Have been using it for 10 years now. 10 years? Damn, that is a long time. I always assumed that heat-induced damage will kill a CPU after a certain amount of time (5-7 years). Am I wrong here? I assume yes. Or are CPUs must stronger/tougher than the bad old days?

> 10 years? Damn, that is a long time. I always assumed that heat-induced damage will kill a CPU after a certain amount of time (5-7 years). Am I wrong here? I assume yes. Or are CPUs must stronger/tougher than the bad old days?

My i7 920 is still running fine. Or, it was when I decommissioned it in 2017. I don't imagine any reason it shouldn't, except perhaps bitrot of spinning rust (spinning rust rotting is no joke, especially after ~20 years) and maybe aging of thermal paste.

My i7 6950X is still running fine, in use since 2017 even today to write this message.

Re: A 10 year old Xeon is all you need

#264

Post by ducttapecrown »

ryandvm wrote: post #125
PLenz wrote: post #104

Earlier quoted context omitted.

This. OpenAI and Anthropic are ultimately compute infrastructure plays and not really AI. Everyone will have models, they'll have the ability to run them. This is why the GPU shortage is in their favor.

And like Google and Meta, these companies are going to morph into advertising giants. Advertising is an economic black hole and it eats everything that comes close.

I feel like LLM's will change advertising like internet search changed advertising.

Here "like" means similarly in magnitude, not direction. If I could predict the future etc.

Re: A 10 year old Xeon is all you need

#265

Post by kingnothing »

montroser wrote: post #97

Result is ~12 tokens per second, as reported by OP down in these comments here. An impressive effort, and better than I would have thought possible on this hardware -- but still pretty far short of what one needs for an satisfactory interactive session.

It's not terrible for interactive... https://mikeveerman.github.io/tokenspeed/?rate=12&mode=text

And it should be just fine for plenty of background use cases.

Re: A 10 year old Xeon is all you need

#266

Post by kingnothing »

throwaway2037 wrote: post #65
jansommer wrote: post #52

The E5-2620 v4 is great. Have been using it for 10 years now. Wanted to upgrade until I saw current prices. I have 64 GB ddr4. Paired it with rx 9060 xt 16 GB and games run as fast as ever. Perhaps the cpu is a slight bottleneck in DOOM The Dark Ages, but i'm at 60 fps, so no problem. Light llm on the gpu is a nobrainer, and it's cool to see that things can be tuned to run ok on the cpu. I bought 2667 v4 a month ago…

> The E5-2620 v4 is great. Have been using it for 10 years now. 10 years? Damn, that is a long time. I always assumed that heat-induced damage will kill a CPU after a certain amount of time (5-7 years). Am I wrong here? I assume yes. Or are CPUs must stronger/tougher than the bad old days?

I've never had a cpu die in the decades I've been using them. I've bought 10-20 year old computers that still work just fine. I kept my last MacBook for 9 years before I upgraded out of want for more RAM.

Most computer equipment fails quickly, otherwise you'll get a long life out of whatever it is.

Re: A 10 year old Xeon is all you need

#267

Post by m463 »

cafkafk wrote: post #2

Hi HN. I wrote this post after getting frustrated by the lack of ways to run the new Gemma 4 Drafter models, and mainstream tools not prioritizing this, and hiding all the performance levers. I ended up getting a modern 26B MoE model (Gemma 4) running at reading speed on an old recycled server with a single Xeon E5-2620 v4 and 128GB of DDR3 RAM (and no GPU). It took a lot of work, but it actually worked out somehow.…

I bought a renewed 2x E5-2690v4 server (28c/56t) 128gb on amazon for under $500 2 years ago (28c/56t) dell T7810

search amazon for "chia farming" ...and scroll past chia seeds :)

now same machine is 2.5x the price

https://www.amazon.com/dp/B095TRGCSX

but way cheaper than current ddr5 machines

Re: A 10 year old Xeon is all you need

#268

Post by thomashabets2 »

danbruc wrote: post #207
knorker wrote: post #204

Earlier quoted context omitted.

I'd expect it to require too much RAM bandwidth to be feasible. RAM is really slow at silicon speeds. Very little is reachable in one clock cycle, unless the clock cycle is abysmally slow.

No RAM. Instead of having a general purpose multiplier that multiplies an input with a weight stored in RAM, just have a multiplier that hardcodes the weight. In some sense replace each weight with a specialized multiplier and wire them together with accumulators and activation functions in between. And some registers for pipelining. If one goes for four bit quantization, one could have sixteen optimized multipliers,…

That sounds like wiring the RAM information into order of magnitude same number of transistors. A modern CPU has (quick googling) 184B transistors. If they were bits then that's 23GB. But presumably a model bit needs more than one transistor to represent how it acts as a neuron with its interactions.

Then there's the current speedup in inference from restricting which subset of the model is used, which is not a "swap in" that would work with hard wired neurons.

But I dunno. Maybe. I'm just guessing.

Re: A 10 year old Xeon is all you need

#269

Post by lightedman »

_zoltan_ wrote: post #78
lightedman wrote: post #49

Earlier quoted context omitted.

The first two generations supported DDR3 only. Haswell and Broadwell (v4) brought DDR4 support.

right, and they talk about "v4" which is DDR4.

There were several V4 Xeon models that supported DDR3 AND DDR4 simultaneously. If you had a motherboard with an X79 chipset it would (sometimes) work properly.

Re: A 10 year old Xeon is all you need

#270

Post by HerbManic »

PLenz wrote: post #104
cmiles8 wrote: post #84

We’re not there yet, but the obvious endgame of the present bubble insanity is open models running on local hardware and devices are “good enough” for most use cases. That will completely implode what’s going on at the moment in tech.

This. OpenAI and Anthropic are ultimately compute infrastructure plays and not really AI. Everyone will have models, they'll have the ability to run them. This is why the GPU shortage is in their favor.

It is for now but they cannot keep demand on their side high enough to suck up supply forever. Manufacturing isnt going to stop, not unless there is a Taiwan incident.

For those with tin foil hats, scheme away at possible futures!

Post reply on HN