71–80 of 150 posts
> The benchmark prompt was: > Write a compact Python function that parses a unified diff and returns the changed file paths. Then explain two edge cases. > Each benchmark generated about 128 tokens. Generating 128 tokens is probably not enough for good benchmark results. MTP speedup depends on how often the predicted tokens are accepted. In my experience, the very early output has a higher acceptance rate, so short t…
I assumed lmstudio is the obvious choice after ollama. Is there a reason lmstudio is not used widely ?
But there is an incentive not to use it if you want to write an article that uses only open-source tools, because it isn't.
>64 GB Thats the rub. I have an M4 with 48G. I wonder if it is worth testing this out. My past attempts (with Ollama and various LLMs) were too slow to use.
I have a M5 MAX with 128, local models are toys compared to hosted ones. I've spent a lot of time and money trying to make it work even 1/2 as well.
If you're seeking the kind of hands-off claude experience, obviously not. They are slow.
If you want to learn how these things work, train them locally, tinker, play with the code, grasp the fundamentals, or just out of sheer bloody-mindedness and principle refuse to tether the functioning of your application to a cloud API...
Earlier quoted context omitted.
https://en.wikipedia.org/wiki/Is_Google_Making_Us_Stupid%3F https://newsletter.pessimistsarchive.org/p/when-educators-mo... New decade, same old argument. It's not > "Claude, think for me" It's > "Claude, be my subordinate and get this done for me" Instead of complaining on the sidelines, I'm getting a shit ton of work done.
> Instead of complaining on the sidelines, I'm getting a shit ton of work done. Nah, you are just producing a bunch of slop and hope that nobody notices.
My biggest pet peeve with all these articles on local AI is the only thing they talk about is tokens per second. No one mentions the quality of the answers. No one. I don't mind waiting a little longer if the quality is better. Quickly serving me slop doesn't make it more useful. Are people really only looking at tokens per second?
Earlier quoted context omitted.
Yes. -hfd for the draft model.
Nice, was wondering if there was a flag for the draft as well. Not knocking huggingface-cli, just find it's much easier for people to try out this stuff when they can just mise use --global github:ggml-org/llama.cpp LLAMA_CACHE="models" llama-server \ -hf unsloth/gemma-4-26B-A4B-it-qat-GGUF:UD-Q4_K_XL \ --host 0.0.0.0 \ --port 11434 \ ...
—no-mmproj
I have used omlx.ai with great success to both download multiple mlx models (including gemma and qwen) suited for my hardware AND to be able to automagically launch both open-source and close-source (claude code, codex) harnesses using these models. All from a web or desktop UI You would not need to follow a blog post with omlx IMHO
In case anyone is looking for a sandbox to go with oMLX and Pi: https://github.com/Dotnaught/pi-sandbox
sbx policy set-default open
> The benchmark prompt was: > Write a compact Python function that parses a unified diff and returns the changed file paths. Then explain two edge cases. > Each benchmark generated about 128 tokens. Generating 128 tokens is probably not enough for good benchmark results. MTP speedup depends on how often the predicted tokens are accepted. In my experience, the very early output has a higher acceptance rate, so short t…
[flagged]
Alas, this video appears not have been linked to the text that describes it. Perhaps I should ask an AI to generate an artistic rendering of the author's description.
It feels like a GPT-4 class model in terms of "stored knowledge" but is better at long-horizon tool calling than any of the GPT-4 class models.
Running on a 128GB MBP M4 Max, I'm getting ~24 t/s on generation and ~200 t/s on prefill. I was expecting it to feel slow, and it certainly does when e.g. generating code, but it's surprisingly useful as a "machine orchestrator" for simple tasks.
For non-agentic usecases, it's a decent enough model to converse with, and has the benefit of being entirely self-contained/private.