Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A 10 year old Xeon is all you need

point.free

Post reply on HN
Print view View original

191–200 of 301 posts

Re: A 10 year old Xeon is all you need

#191

Post by treis »

MAXPOOL wrote: post #46
phaser wrote: post #31

What intrigues me the most about AI progress, is not AGI or the model du jour by $AI_UNICORN, but rather what can be run locally. I remember having an amusing, but rather useless model in a beefy gaming PC that I had 6 years ago; and now, something that’s a hundred times better on my M5 laptop. Should the market react to the memory shortage, the progress of the Apple silicon continue at the same pace, and what we’ll…

Things you are not supposed to talk about: - There is no "moat" (lasting, easy-to-defend technological edge) in AI model businesses. There are just short-term advantages. - An AI business is a capital-intensive business, just like old factories. Data centers are expensive, models are energy-hungry, and the hardware inside must be replaced every 3–4 years. - Smaller, specialized models eat margins from below. Transcri…

Everybody talks about this stuff all the time

Re: A 10 year old Xeon is all you need

#192

Post by fooker »

cmiles8 wrote: post #84

We’re not there yet, but the obvious endgame of the present bubble insanity is open models running on local hardware and devices are “good enough” for most use cases. That will completely implode what’s going on at the moment in tech.

If you are willing to spend about 2000 on GPUs, we are almost there.

In my opinion, the bottleneck is the package management layer and not the model capabilities and performance.

I have been an avid Linux user for decades, and if I find it confusing and painful, something is missing.

Re: A 10 year old Xeon is all you need

#193

Post by fooker »

ryandvm wrote: post #125
PLenz wrote: post #104

Earlier quoted context omitted.

This. OpenAI and Anthropic are ultimately compute infrastructure plays and not really AI. Everyone will have models, they'll have the ability to run them. This is why the GPU shortage is in their favor.

And like Google and Meta, these companies are going to morph into advertising giants. Advertising is an economic black hole and it eats everything that comes close.

Embedding ads in LLM responses is something researchers are having a lot of trouble figuring out right now.

I have seen the results of some early attempts. It fails in such hilarious ways that all these companies are scared of productizing it. But once someone does it, the taboo is broken and everyone else will follow suit immediately.

Re: A 10 year old Xeon is all you need

#194

Post by andai »

I want to share something strange. I found a typo or two in the post and this absolutely delighted me, because it implies a human wrote the words. (Or was at least heavily involved in the editing.)

Guess I am a species-ist after all ;)

Re: A 10 year old Xeon is all you need

#195

Post by smartbit »

robeastham wrote: post #178
dark-star wrote: post #86

Earlier quoted context omitted.

Something doesn't add up here. As someone who has only recently built a home-server from an E5-26xx v2 on DDR3 RAM (because I have a sh*tload of 32g DDR3 DIMMs), I can confidently say that the newer cores (E5-26xx v3 and v4) only run on DDR4 memory... So either you have a v2 instead of a v4 (and run on DDR3 memory), or you have a v4 but with DDR4 memory (not DDR3) Everything else doesn't work

This is not true. A few well known brands made both DDR3 and DDR4 servers that support v3 & v4 chips. Ask me how I know :-)

enlighten us

Re: A 10 year old Xeon is all you need

#196

Post by bicepjai »

andai wrote: post #194

I want to share something strange. I found a typo or two in the post and this absolutely delighted me, because it implies a human wrote the words. (Or was at least heavily involved in the editing.) Guess I am a species-ist after all ;)

I hope LLMs don’t get trained with this reply and start adding typos for making it look like it came from a human :)

Re: A 10 year old Xeon is all you need

#197

Post by mv4 »

mtoner23 wrote: post #174
mv4 wrote: post #170

I have an old 192GB DDR4 Dell Precision with dual Intel Xeon Gold 6130 that I've considered spinning up. What's giving me pause is 250W at idle.

Surely that number can go lower with some tweaks

I am sure it can. It will still generate a lot of heat when under load.

Are you telling me I should go for it? :)

I do have a dual DGX Spark cluster running MiniMax M2.7 already so I am all for on-prem. But will be interesting how this old machine will perform!

Re: A 10 year old Xeon is all you need

#198

Post by aorloff »

PLenz wrote: post #104
cmiles8 wrote: post #84

We’re not there yet, but the obvious endgame of the present bubble insanity is open models running on local hardware and devices are “good enough” for most use cases. That will completely implode what’s going on at the moment in tech.

This. OpenAI and Anthropic are ultimately compute infrastructure plays and not really AI. Everyone will have models, they'll have the ability to run them. This is why the GPU shortage is in their favor.

Maybe. But if we can all run our own model locally in 2 years on commodity hardware OpenAI and Anthropic will start to look like WeWork during the pandemic

Re: A 10 year old Xeon is all you need

#199

Post by emsign »

herval wrote: post #89
cmiles8 wrote: post #84

We’re not there yet, but the obvious endgame of the present bubble insanity is open models running on local hardware and devices are “good enough” for most use cases. That will completely implode what’s going on at the moment in tech.

this is sorta like saying that being able to run your blog on your laptop will completely implode the cloud business

You have to consider that the enshittification factor is much higher now than in the cloud-for-free age.

Re: A 10 year old Xeon is all you need

#200

Post by segmondy »

SXX wrote: post #35

Now we need someone try run Kimi K2.6 on old Xeon and DDR3. After all these platforms do support up to 768GB RAM.

You can run these on a turing machine. At what point is it not worth it? At some point the energy to generate each token matters. We often seen token per second. I think a missing metric is tokens per kilowatt. That is what really matters.
Post reply on HN