351–360 of 382 posts
Earlier quoted context omitted.
I haven't tried it but it should work! You can try it and share your results, it would be really appreciated I tried it on my wife's M1 MacBook Air 512GB and it gets 4–5 tok/s Also, it must be easy to adjust for iPhones and iPads in theory
Confirmed on my Neo! Got 4.5 tokens / second sustained.
Earlier quoted context omitted.
The term "AI slop" is thought-terminating. A more nuanced approach: read it and decide for yourself on merits, rather than vibes.
Life is short. Do you want to spend it reading text that was evidently generated by a machine? There is an opportunity cost to reading "slop."
Earlier quoted context omitted.
The term "AI slop" is thought-terminating. A more nuanced approach: read it and decide for yourself on merits, rather than vibes.
Saying that something is thought terminating is thought terminating, it's the laziest "I win" bullshit approach ever. A more nuanced approach: don't produce slop and people won't dismiss it as lazy bullshit either.
Look, there's is a wide variety of work being produced with AI, all the way from exceptional professional work to total trash done by amateurs. Painting all over those efforts with the same brush of "AI slop" attempts to avoid the thought necessary to process the nuance in each individual situation. In fact, folks that use "AI slop" enjoy being able to dismiss AI output as quickly as possible; they seem to be quite happy to forgo whatever insights might be present in such work. But let's not for a moment pretend it's not a crappy heuristic.
Through this lens, dunking on a piece of prose because it has some trace of LLM processing seems both useless and uninsightful, which is why I'm rallying against it as thought-terminating. Do the thinking to determine whether what you're reading is valid. Saying that it has tells that an LLM might have contributed is not sufficient evidence to do that, and it's also something anyone can do, it requires no skill or insight, and makes for boring discussion. Zero curiousity, 100% dismissive.
Earlier quoted context omitted.
> I always wondered why we need to shove the entire model into memory, I don't care who King Charles is every single time. That's kind of the problem, isn't it? How do you know which part of the model to put in memory? You have to make a per-parameter decision of whether or not it's worth it to have it in memory or whether the value should just be treated as zero. Then you have to "re-link" the layers of the model to…
Would an extension to `madvise` to say: "page this region in/out as a whole" help here? Engine could define memory ranges representing each expert and leave paging to the OS (tho' "paging" at this point becomes more similar to swapping in granularity...).
Earlier quoted context omitted.
> I always wondered why we need to shove the entire model into memory, I don't care who King Charles is every single time. That's kind of the problem, isn't it? How do you know which part of the model to put in memory? You have to make a per-parameter decision of whether or not it's worth it to have it in memory or whether the value should just be treated as zero. Then you have to "re-link" the layers of the model to…
I know very little about this but it seems like the kind of thing that can and eventually will be solved computationally, not by people figuring out what a parameter or group of parameters represent
> The measured result is a reference point, not a performance ceiling. Claude was here.
Everyone's on high alert. Maybe writing will get better!
Can the same be done with qwen3.6-35b-a3b?
Yeah, the same ideas should work for qwen. You can try porting this engine to use Owen. Owen 3.6-35b-a3b was my initial idea, but I switched to Gemma because of its simpler architecture and kernels
Would be awesome if it ran Qwen (the MoE probably won't squeeze that low, but...). This because I have hardly been able to use Gemma for any sort of useful coding.
Gemma 4's tool calling was recently fixed; that was the main issue with agentic use in my experience. Otherwise IMO it codes about as well as the Qwen MoE for PHP and SQL. It's a fully impressive model (though it is not as mindbendingly impressive as the 12B, which is outrageously good for its footprint)