Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A 10 year old Xeon is all you need

point.free

Post reply on HN
Print view View original

141–150 of 301 posts

Re: A 10 year old Xeon is all you need

#141

Post by clusterhacks »

phaser wrote: post #31

What intrigues me the most about AI progress, is not AGI or the model du jour by $AI_UNICORN, but rather what can be run locally. I remember having an amusing, but rather useless model in a beefy gaming PC that I had 6 years ago; and now, something that’s a hundred times better on my M5 laptop. Should the market react to the memory shortage, the progress of the Apple silicon continue at the same pace, and what we’ll…

--what this means for the valuation of the AI companies

Probably nothing. Most users have no idea what an LLM is or how it runs. Anecdotally speaking, I see many LLM users default to whatever their day job provides to them. And even slightly more sophisticated users seem ok with paying for their openai or anthropic subscriptions.

Maybe we will see a small but dedicated group of open weight model users who prefer local llm, but everybody else will just consume from the big providers? The scenario might look something like OS choices today - a small, committed group of Linux users vs the vast majority of other users running Windows, MacOS, or Chrome?

Re: A 10 year old Xeon is all you need

#142

Post by coldtea »

IdiotSavage wrote: post #91
cmiles8 wrote: post #84

We’re not there yet, but the obvious endgame of the present bubble insanity is open models running on local hardware and devices are “good enough” for most use cases. That will completely implode what’s going on at the moment in tech.

I find that hard to believe. The AI companies will want to control what's possible and find new things to do that "need" their services. Otherwise it would be like Intel and Microsoft had decided in the year 2000 that computers are "good enough" now and we would have explored what's possible with that hardware ever since.

>Otherwise it would be like Intel and Microsoft had decided in the year 2000 that computers are "good enough" now and we would have explored what's possible with that hardware ever since.

That would be the dream... no fucking Electron! No lockdown modules.

Re: A 10 year old Xeon is all you need

#143

Post by zozbot234 »

ryandvm wrote: post #123
cmiles8 wrote: post #84

We’re not there yet, but the obvious endgame of the present bubble insanity is open models running on local hardware and devices are “good enough” for most use cases. That will completely implode what’s going on at the moment in tech.

I disagree. We are currently in a weird period where these frontier AI companies are losing tons of money even on the subscription-based AI models. It's just too compute intensive and there's no way most people are going to be buying the kind of hardware required to run $20 worth of inference every day. Sadly - it's going to be ads. Advertising is going to get in there and enshittify the whole thing because as always…

Most people are running a whole lot less than $20's worth of tokens per day on cloud platforms. (Is that assuming a frontier model? 1M output tokens per day?) Local hardware could easily take up that workload, at least the part of it that's non-time-critical.

Re: A 10 year old Xeon is all you need

#144

Post by adrian_b »

bflesch wrote: post #53
bflesch wrote: post #22

Might consider going for even older CPUs which don't have the Intel ME ring -3 thing which is full of backdoors

I appreciate the downvotes without any reasoning. It's a fact that newer Intel CPUs have Intel ME which was not in older CPUs and significantly increases attack surface if you are not living in a five eyes state.

In a server, you have to worry about the ME only if you also have an Intel Ethernet interface, which is connected to a potentially hostile network.

If that is not true, the ME cannot be controlled remotely.

The existence of the ME is much more worrisome in laptops, where the ME can be accessed remotely through WiFi. There, to be certain that there is no way for the ME to be accessed remotely you would have to disconnect or cut the internal antennas and use a USB dongle for WiFi.

Re: A 10 year old Xeon is all you need

#145

Post by coldtea »

IdiotSavage wrote: post #117
squidbeak wrote: post #110

Earlier quoted context omitted.

> Otherwise it would be like Intel and Microsoft had decided in the year 2000 that computers are "good enough" now and we would have explored what's possible with that hardware ever since. I think you've misunderstood what good enough means in the context - which is a model capable of completing the tasks assigned to it without having the breadth of full generalization. Your analogy breaks down because of this - we d…

I think you've misunderstood the analogy. Just ignore it, analogies mostly break down anyways. > a model capable of completing the tasks assigned to it The thing is, the "task assigned to it" is changing with improved capabilities. If everyone around you in 2036 is using general AI to do amazing stuff, you will probably have little interest in vibe coding slop like it's 2026.

>The thing is, the "task assigned to it" is changing with improved capabilities.

Only if you give in to fads and FOMO.

The core tasks people need change at a much smaller pace.

Re: A 10 year old Xeon is all you need

#146

Post by bbatha »

cafkafk wrote: post #6
fragmede wrote: post #3

Earlier quoted context omitted.

(purple on black is really hard to read) You say it runs "at reading speed". Have you benchmarked it?

> (purple on black is really hard to read) Noted, and agree (it looks like it has also already been clicked, which I dislike). I honestly I need to redo the themes. > You say it runs "at reading speed". Have you benchmarked it? At some point a few weeks ago, yes I think so, but I didn't write it down for some reason... so I'll have to find a time when it's not busy and do it again without a noisy system. Right now th…

What's time to first token? Raw throughput is usually not the problem in local setups in my experience.

Re: A 10 year old Xeon is all you need

#147

Post by enoint »

ryandvm wrote: post #123
cmiles8 wrote: post #84

We’re not there yet, but the obvious endgame of the present bubble insanity is open models running on local hardware and devices are “good enough” for most use cases. That will completely implode what’s going on at the moment in tech.

I disagree. We are currently in a weird period where these frontier AI companies are losing tons of money even on the subscription-based AI models. It's just too compute intensive and there's no way most people are going to be buying the kind of hardware required to run $20 worth of inference every day. Sadly - it's going to be ads. Advertising is going to get in there and enshittify the whole thing because as always…

The advertising future looks like that to me, too. Service proxies like OpenRouter might talk about price optimization, maybe some ad filtering. But I expect proxies will have malicious entries, too, surreptitiously altering agentic prompts.

Re: A 10 year old Xeon is all you need

#148

Post by Grazester »

throwaway2037 wrote: post #65
jansommer wrote: post #52

The E5-2620 v4 is great. Have been using it for 10 years now. Wanted to upgrade until I saw current prices. I have 64 GB ddr4. Paired it with rx 9060 xt 16 GB and games run as fast as ever. Perhaps the cpu is a slight bottleneck in DOOM The Dark Ages, but i'm at 60 fps, so no problem. Light llm on the gpu is a nobrainer, and it's cool to see that things can be tuned to run ok on the cpu. I bought 2667 v4 a month ago…

> The E5-2620 v4 is great. Have been using it for 10 years now. 10 years? Damn, that is a long time. I always assumed that heat-induced damage will kill a CPU after a certain amount of time (5-7 years). Am I wrong here? I assume yes. Or are CPUs must stronger/tougher than the bad old days?

Not my experience.

Re: A 10 year old Xeon is all you need

#149

Post by hylaride »

Scoundreller wrote: post #139
ryandvm wrote: post #123

Earlier quoted context omitted.

I disagree. We are currently in a weird period where these frontier AI companies are losing tons of money even on the subscription-based AI models. It's just too compute intensive and there's no way most people are going to be buying the kind of hardware required to run $20 worth of inference every day. Sadly - it's going to be ads. Advertising is going to get in there and enshittify the whole thing because as always…

Ads are usually the workaround where you don’t deliver enough value to get people to subscribe or payments are unavailable for some reason. It makes sense to show some ads and get some money at low volume (like a faraway reader wanting to read a story in your local newspaper) but taking money from regular users directly will pay much more. Newspapers are happy to cannibalize 99% of their ad revenue with a paywall if…

Newspapers had no choice after craigslist and later Google/Facebook took all their classified revenue.

LLMs may or may not be able to cover their costs with it. We'll see - I suspect product placement as recommendations will become a thing as it won't take as much GPU to give a "recommendation" on "the best widget for X". I firmly expect it to become enshittified the same way google and amazon search has.

And that's if LLMs don't become commodified.

Re: A 10 year old Xeon is all you need

#150

Post by trollbridge »

zozbot234 wrote: post #131
observationist wrote: post #128

Earlier quoted context omitted.

It's a huge difference. If you had AI sufficiently good running locally on a phone, you could devise workflows for things like basic digital hygiene, technical assistance, and tedious tasks like inbox management, image sorting, device updates, and so on. Privacy and security gets a big boost past some local competence threshold, and we're nearly there. Make the local AI competent enough to do good image generation an…

Phones and laptops are terrible devices for local AI, way too constrained by bad thermals and small batteries. MiniPC's (many of them using mobile hardware) don't have that particular issue, and can easily run on a 24/7 basis.

Phones are also a terrible place to run a radio, but there's a huge amount of benefit in figuring out how to do so.
Post reply on HN