161–170 of 233 posts
Earlier quoted context omitted.
My problem is how hot they run. I'm on an m4 pro. Do you have the same issue?
I don't have the hardware but a often mentioned advice is to put your mac into energy saving mode - it still will work, a bit slower, but stays cool.
Steve Jobs would be proud. People really are holding their Apple hardware wrong.
I just got qwen 3.8 27b mlx running on my Macbook Pro and honestly I’m pretty blown away by how not-dumb it is.
It was actually great. I have like a non-AI box so to speak 8GB VRAM, co-incidentally from a gaming PC ... All the previous models that were "frontier level, just try it!" but wouldn't run at all in agentic mode, including previous Qwens, just disappointed, period. Then I ran then Qwen 3.8 27b and while it was super slow (4t/s) it literally one-shotted creating a usable "web search/pull" skill for `pi.dev` . while an…
Somewhat off topic, but I've started wondering if we can actually make local LLMs feel smarter than the frontier closed models, by post-training it for your specific use case. Say Company X has a software product which consists of a million lines of code, including a ticket for every bug and new feature for this piece of software. Then wouldn't it make sense to try using an open weight model but post-train it on that…
On the one hand it would definitely be useful for something like classifying support requests into priority. But would it be worth it to just train your own model? I guess one advantage is you could give it some well informed guidelines without training something on lots of data.
There is probably a better example between discrete labeling and code though.
Somewhat off topic, but I've started wondering if we can actually make local LLMs feel smarter than the frontier closed models, by post-training it for your specific use case. Say Company X has a software product which consists of a million lines of code, including a ticket for every bug and new feature for this piece of software. Then wouldn't it make sense to try using an open weight model but post-train it on that…
This is something I can reuse better.
Earlier quoted context omitted.
Wow. I tried to get Qwen3.8 4B to parse song lyrics and analyse them. Getting ollama running was a minute or two. However coming up with a prompt that didn't turn out total garbage was impossible. After wasting over an hour and I ended up getting Qwen side by side with Llama 3.2 3B, just to see if I was being stupid. Nope, it just looks like Llama is orders of magnitude better at this specific task for some reason).…
> Nope, it just looks like Llama is orders of magnitude better at this specific task for some reason This is almost every ML model, if the task isn't part directly or indirectly of the datasets they use for training it, then the model is gonna be pretty trash at it. What the big AI labs have over the smaller labs, is a huge amount of data and diverse set of tasks, hence they generalize better, but still not great. So…
Of course some things are not auto verifiable, and you'll have to give human judgement and input there, but you'll save a lot more time if you spend 1 week painstakingly writing checks for as many little things as possible and integrating them into the harness.
> And the comparisons in this post are not going to be running some 2.58-bit-gguf-in-ollama with a couple test prompts. Genuine question : is there something fundamentally wrong with Ollama ? I use Ollama because it is easy to set up and manage (and also because VLLM is not super Windows friendly). I thought the main advantage of VLLM was better concurrency management (better batching). But if the quality of the inte…
Maybe some were fixed, but: 1) Shipping with 2k default context window for the longest time, w/o any warning and being not easy to change (like any other setting). Totally made a lot of people think local LLMs are dumb as rocks. Just checked, still not fixed -- defaults to 4K if less than 24GB VRAM: https://docs.ollama.com/context-length 2) Registry mistrust due to Deepseek R1 naming. What model do you download/start…
This is fast becoming one of my top old-man-yells-at-clouds pet peeves.
Reported performance metrics are ONLY good for the exact model weights.
Quantizing a model, or changing it in any way, requires new evaluation to know how well it performs.
Quantizing a good model doesn't mean the quantized version is good.