> SOTA LLMs locally
Shouldn't the headline be about running SOTA _local_ LLMs, as GLM 5.2 is nowhere near a SOTA LLM?
Jamesob's guide to running SOTA LLMs locally
151–160 of 193 posts
Re: Jamesob's guide to running SOTA LLMs locally
#152Earlier quoted context omitted.
Yes, "Like a car". LOL. You realise that many people in Europe and Asia do not own a car at all? Public transport, eBike / scooter, Tuk-Tuk, walk . The local LLM "privacy" war had been already lost.
You realise how many people in Europe do own a car? And how exactly has the privacy war been lost?
The privacy war has been lost in two ways (at least) 1) Running locally lobotomised models makes no sense; 2) as someone said here, the Gov will declare local AI a felony. And they will enforce it.
So those "many people" will buy V8 cars limited to V4 and declared illegal to drive without registration and license, even locally in your own yard, and they may go to jail if they attempt to activate other 4 cylinders.
Oh, wait ... isn't it how car laws work now? ;-)
Re: Jamesob's guide to running SOTA LLMs locally
#153Earlier quoted context omitted.
You realise how many people in Europe do own a car? And how exactly has the privacy war been lost?
The privacy war has been lost in two ways (at least) 1) Running locally lobotomised models makes no sense; 2) as someone said here, the Gov will declare local AI a felony. And they will enforce it. So those "many people" will buy V8 cars limited to V4 and declared illegal to drive without registration and license, even locally in your own yard, and they may go to jail if they attempt to activate other 4 cylinders. Oh…
Ludicrously paranoid take.
Re: Jamesob's guide to running SOTA LLMs locally
#154now the electric bill is the real subscription fee
Re: Jamesob's guide to running SOTA LLMs locally
#155FP4 is basically lossless
Re: Jamesob's guide to running SOTA LLMs locally
#156[dead]
Re: Jamesob's guide to running SOTA LLMs locally
#157Very useful. The whisper setup is something similar to what we have been using. The LLM setup though is outstanding.
i have the same Whisper Setup too!
Re: Jamesob's guide to running SOTA LLMs locally
#158Earlier quoted context omitted.
The privacy war has been lost in two ways (at least) 1) Running locally lobotomised models makes no sense; 2) as someone said here, the Gov will declare local AI a felony. And they will enforce it. So those "many people" will buy V8 cars limited to V4 and declared illegal to drive without registration and license, even locally in your own yard, and they may go to jail if they attempt to activate other 4 cylinders. Oh…
Ludicrously paranoid take.
I'm glad that you saw my point and I apologise for stepping on your ego. :-)
Re: Jamesob's guide to running SOTA LLMs locally
#159Earlier quoted context omitted.
Yeah but 4 bits very often loops needlessly. Which is not that bad because you do not pay for tokens. But you paid for hardware and you want use it for something useful. Q6 is better but then you have like 40t/s prefill. Which is really tiring. But at least it says sorry when you ask it what is wrong! I heard there is some extension for PI preventing that. I need to look into it. Otherwise I am quite happy.
You can fix looping with proper repetition penalties. Turn on the one called “DRY” that PeW invented and got merged into llama cpp
I added repetition penalty of two. I do not know. Maybe it is not applied correctly somehow by llama swap, that I am using, but I do not consider it that much of a nuisance to so I did not tried to fix it yet.
Do you have this DRY docs?
Re: Jamesob's guide to running SOTA LLMs locally
#160I play with local LLMs a lot. I've spent more on hardware than I should. I'm friends with a local group of people who have spent a lot more than I have. The warning I would have for everyone is to temper your expectations and read the fine print carefully. The big build in article starts off with a $40K budget and then includes 4 GPUs that are $12K each. For those doing the math, this build is going to cost more like…
I run Qwen3.6 on RTX4090, and it does amazing job for the most parts.
For coding task, one needs to break the session among multiple calls I made https://github.com/aka-rider/orqestra but it's possible to do the same in almost any modern harness directly.
The main idea is: - separate session that burns context on reading code and calling tools (context7, etc) -> markdown report "here are relevant patrts of code, docs" "with evidence" to prevent hallucinations
- separate session for planning (architect) - (critic architect) 1-3 times because small model skip over details - worker validator, again, the same reason
Qwen3.6 can run for hours looking for a complex bugs in read-only mode, and usually it gets it. Proposed fix would probably be hacky, but so as Sonnet's
Qwen3.6 can mechanically write code by Opus-made plan. You would have to prompt afterwards:
"Review your own changes. Any bugs? Cross-validate against the original plan - any gaps? Any violations of CLAUDE.md"
But again, I need to do this for Sonnet. But also I use local llms for reindexing knowledge base.
Grooming tickets: I can leave a caveman note "single panel for errors rendering, move all error messages" and come back to 90% ready specs with the end goal and context.