101–110 of 382 posts
> The measured result is a reference point, not a performance ceiling. Claude was here.
In all fairness, maybe it's just that they let some post-2022 recipe blogs get into the training runs around ~4.6-4.8 time
Dont want to crash the party here, but I am still sceptic about all those on-premise-llm-approaches. I think we strongly need something like that (shameless plug, I tried to build something around bitNet for the same reason: https://github.com/nickyreinert/bitNetRTR ). But at the end, all aproaches I saw, however genius they are: the actual results are always a mess. It's a better chat buddy, nothing else. It's e.g.…
You are not really accessing the biggest frontier model every time, and you're not really doing an end-to-end LLM request on each prompt.
I would go so far to say frontier models have peaked and improvements from here come from clever (or very elaborate) harnessing. "LLLMHs" - Large Large Language Model Harnessing !
I keep seeing more and more LLM models being loaded by incredibly under-powered machines. Is the GPU/memory crisis all lies? I get that running on an RTX 5090 will be much faster, but if we can use main memory instead of VRAM and get barely usable results, what is going on?
> The measured result is a reference point, not a performance ceiling. Claude was here.
They don't add anything of value, did the author use an LLM to fix his prose but no useless slop was added in the process: who cares ? Is the article useless slop: fine, downvote it to oblivion.
(1) For those not old enough to remember that wonderful practice please use your nearest LLM to find out or, you know, visit a library and do your own research.
12 tok/s and almost instant response on M1 Max Mac Studio (with faster SSD than laptops) are impressive – gives hope that large models may run locally from SSDs instead of memory.
What part of the optimization process gave you the biggest speed gain?
> It currently generates 5–6 tok/s on an 8 GB M2 MacBook Air and 31–35 tok/s on an M5 MacBook Pro. Where does this big a performance spread come from? I wouldn't naïvely expect SSD performance difference to be that big, and I would expect SSD performance to dominate...
My suspicion is that this is simply due to the M5 having more memory, and the OS already having most of the file cached. The M2 has more memory pressure and would cache fewer of the SSD reads If that's true, inference speed would be even lower if you have only 2GB total, including OS caches
The memory bandwidth is bumped up by 50%, and the size of the on-die system level cache is bumped up by 50% as well.