Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A 10 year old Xeon is all you need

point.free

Post reply on HN
Print view View original

161–170 of 301 posts

Re: A 10 year old Xeon is all you need

#161

Post by enoint »

hylaride wrote: post #149
Scoundreller wrote: post #139

Earlier quoted context omitted.

Ads are usually the workaround where you don’t deliver enough value to get people to subscribe or payments are unavailable for some reason. It makes sense to show some ads and get some money at low volume (like a faraway reader wanting to read a story in your local newspaper) but taking money from regular users directly will pay much more. Newspapers are happy to cannibalize 99% of their ad revenue with a paywall if…

Newspapers had no choice after craigslist and later Google/Facebook took all their classified revenue. LLMs may or may not be able to cover their costs with it. We'll see - I suspect product placement as recommendations will become a thing as it won't take as much GPU to give a "recommendation" on "the best widget for X". I firmly expect it to become enshittified the same way google and amazon search has. And that's…

For agentic services, how would you be able to tell that you’ve been product-placed?

Re: A 10 year old Xeon is all you need

#162

Post by Majromax »

bboozzoo wrote: post #158
Majromax wrote: post #116

Earlier quoted context omitted.

From the prompt timings above, it seems like 'prompt eval time' is the equivalent to 'processing time for input tokens'. Hyperscalers can perform this evaluation very quickly because evaluation can be significantly parallelized. The layer `i` output of token `j` only requires access to the layer `i-1` output of all previous tokens, so a parallel frontier develops. Token (0,0) [(token, layer)] is processed first, then…

Seven tokens long input isn't very realistic, is it? For coding tasks it's normal for the input to be thousands or 10s of thousands. If it wasn't for prefix caching it'd be one miserable experience, but even then at the very best the input is often in hundreds each time. And don't even try to dump some logs into the prompt.

> Seven tokens long input isn't very realistic, is it?

The test prompt above was "Why is the sky blue?", so there's the seven tokens. I meant to highlight that because I'd expect processing of a thousand-token input to be faster per token than presented.

Re: A 10 year old Xeon is all you need

#163

Post by bflesch »

tryauuum wrote: post #90
bflesch wrote: post #53

Earlier quoted context omitted.

I appreciate the downvotes without any reasoning. It's a fact that newer Intel CPUs have Intel ME which was not in older CPUs and significantly increases attack surface if you are not living in a five eyes state.

How old are we talking?

IIRC it is pre-2008.

Re: A 10 year old Xeon is all you need

#165

Post by ethbr1 »

zamadatix wrote: post #57
Sweepi wrote: post #51

Earlier quoted context omitted.

"-t 8 matches physical cores. The machine has 16 SMT threads but only 8 cores. On a memory-bound workload, oversubscribing threads adds scheduling cost without adding throughput: the cores are waiting on DDR3, not on each other." But ... isnt that a classic use case for SMT? Giving T1 sth. to do while T0 is waiting on DDR(3) and vise-versa? I also dont understand the explanation of "--cpu-moe". If an expert has ~ 4.0…

> But ... isnt that a classic use case for SMT? Giving T1 sth. to do while T0 is waiting on DDR(3) and vise-versa? Waiting in terms of latency. When the bus is mostly empty and it takes a while to make a round trip it's great to try to find a few extra passengers to put on it. When the buses are all completely full adding the extra riders just makes the bus stop that much more chaotic.

This is ironically a pretty solid use case for (ex VLIW research) ILP-optimizing compilers.

Given knowable runtime hardware usage patterns (huge bursts of memory bandwidth saturation) and a single limited core/thread-shared resource (memory bandwidth), one could optimize for the constraint ahead of runtime.

Because most of the performance optimization levers you have available to pull are (a) trade compute for memory bandwidth (e.g. compression), (b) preload when memory bandwidth is available, (c) optimize the choice of what's in cache when, (d) align to cache size / memory boundaries.

Or tl;dr, try to approximate GPU ISAs at the CPU compiler level. (Which why would anyone but hobbyists, because everyone else just buys pallets of Nvidia/AMD or designs their own ML chips?)

Re: A 10 year old Xeon is all you need

#168

Post by bflesch »

s20n wrote: post #79
bflesch wrote: post #53

Earlier quoted context omitted.

I appreciate the downvotes without any reasoning. It's a fact that newer Intel CPUs have Intel ME which was not in older CPUs and significantly increases attack surface if you are not living in a five eyes state.

I agree with the first part. I think this article by FSF about Intel's ME summarizes the issue https://static.fsf.org/nosvn/blogs/Intel_ME_Carikli_article_... As for the second part, I am not sure about how living in a five eyes state would mitigate it. What do you mean by that?

As five eyes citizen you have at least some rights on paper and you can appeal to your government, but if you are foreigner these guys can go gloves off without any fear of retribution.

Try analyzing Epstein files and posting about it, they'll give you a proper penetration test of all your devices to see what you found out about their ex employee.

Nowadays even EU citizens migrating away from US cloud providers are a "national security issue".

Re: A 10 year old Xeon is all you need

#169

Post by layer8 »

enoint wrote: post #161
hylaride wrote: post #149

Earlier quoted context omitted.

Newspapers had no choice after craigslist and later Google/Facebook took all their classified revenue. LLMs may or may not be able to cover their costs with it. We'll see - I suspect product placement as recommendations will become a thing as it won't take as much GPU to give a "recommendation" on "the best widget for X". I firmly expect it to become enshittified the same way google and amazon search has. And that's…

For agentic services, how would you be able to tell that you’ve been product-placed?

Hidden advertising is illegal in most jurisdictions, so it has to be indicated to the user for each specific occurrence and hence be trackable anyway.
Post reply on HN