551–560 of 682 posts
even 30B model is too large to large on local device (low end). meta should provide free hosted model api to use it.
I agree with their open-source model approach, but actually trusting Meta… to protect my privacy and my data… when it’s running on my personal hardware…
Not . In . A . Million . Years - that ship has sailed
https://xcancel.com/finkd/status/2086755195535413696 "... Soon we'll also release the weights for Muse Spark 1.2, our latest foundation model..." This is bigger news - good for self hosting enthusiasts and a strategically sound move for Meta. Any push towards 'anti Chinese' models will directly benefit Meta as the competition on the frontier open-weights American models is almost non-existent. Meta will have no probl…
Earlier quoted context omitted.
It’s rather amusing to me to read comments like this, and then simultaneously whenever a Chinese company or team releases open-weight models or whatever there is a giant round of applause, America is so behind, and there’s nothing but positive things to say about the intelligent, creative, and well-intentioned Chinese engineers (which is true, America certainly doesn’t have a monopoly on great people). Don’t you know…
The GP is claiming Meta is an awful company for what they have done and how they continue to treat their employees. That’s a perfectly ok opinion to hold, and many seem to agree. Have DeepSeek, Moonshot, or the other Chinese AI companies done such things that attract moral outrage?
Earlier quoted context omitted.
I don't really use the Qwen 3.6 27B though I do test the variants (Bonsai, ThinkingCap). I really like the 3.6 35B A3B for experiments, and it seems OK, but as you say, it spins round in thinking loops more than say the 26B Gemma 4 does. If Muse doesn't actually-wait itself as much it will be very interesting. I am just downloading it to run my small tests.
I have a custom A3B proxy that caps its thinking off. It is a known issue with the model that Qwen themselves documented but is almost never addressed in any harnesses. I also patched up a few other known bugs in the proxy. I highly recommend you shim A3B and when it hits 2K thinking tokens inject (paraphrasing) 'Time to wrap it up bud! Get to work' into its thinking stream. It almost always gets to work. If it needs…
It had not occurred to me that you could nudge it to stop thinking with a proxy. Nice idea.
Will favourite your comment and come back to it.
ETA: Incidentally you've helped me put into words the difference between the way Muse Glimmer thinks to the way Qwen thinks. There is a clear sense of urgency in Glimmer's thinking traces.
> Muse Glimmer is a 30-billion-parameter model optimized for always-on local agent workflows. It’s small enough to run on a Mac or PC with a single consumer GPU, enabling use cases that range from local agents and function calling, to local coding, and LLM-as-a-judge evaluation. The next iteration in LLM products is a 24/7 thinking loop where the claude-code like thing gets input continuously from your wearable, noti…
Earlier quoted context omitted.
It’s really interesting timing, Qwen over thinking is what kills it for me. I’m just glad we have more options in this size class now.
I've been using Qwen3.6 35B A3B, and with reasoning turned on, I'd say 2/3 (give or take) of the tokens for a response are thinking tokens. Which at 70+ tps locally, that isn't that awful. I run an 80k context across 4-10 "agents" for my solo TTRPG, where Qwen is the GM, each NPC at a location, the director, and the narrator. Each turn is about 45-60 seconds to generate all of the various responses. The GM and direct…
https://xcancel.com/finkd/status/2086755195535413696 "... Soon we'll also release the weights for Muse Spark 1.2, our latest foundation model..." This is bigger news - good for self hosting enthusiasts and a strategically sound move for Meta. Any push towards 'anti Chinese' models will directly benefit Meta as the competition on the frontier open-weights American models is almost non-existent. Meta will have no probl…
Had Deepseek V4 Flash 0731 not launched, their latest Laguna release was really intelligent at non-coding tasks and it would have been my go-to model for my local workloads.
Earlier quoted context omitted.
I use llama.cpp w/ llama-swap https://github.com/ggml-org/llama.cpp https://github.com/mostlygeek/llama-swap
FYI, llama-server can now be run in router mode so llama-swap is probably only needed for more exotic scenarios.
Earlier quoted context omitted.
Friends Don't Let Friends Use Ollama https://news.ycombinator.com/item?id=47788385
When that article was first published , I started looking into llama.cpp. With the help of an LLM I researched the knobs to turn that made most sense. Things went from "local models are useless unless you have a 512GB GPU I guess" to "oh.. I can do a lot of stuff locally now!"