261–270 of 411 posts
Earlier quoted context omitted.
Till now I was using successfully Qwen 3.5 and Gemma 4 at a reasonable speed
I think (maybe I missed something) that identical size and quant versions of Qwen 3.5 and 3.8 should run at the same speed. It’s the exact same architecture.
Earlier quoted context omitted.
Tried yesterday on my own laptop (a UltraCore 7 255H without dedicated GPU,with 32 GB RAM), it wasn't even starting thinking, even on a small context window (65k)
Have you tried with different amounts for the "reasoning_effort (xhigh|medium|low)" parameter? Or the " | | " tags: apart from this template detail, it is not immediately clear if reasoning_effort is deterministic (API) or is prompt engineering.
Earlier quoted context omitted.
Tried yesterday on my own laptop (a UltraCore 7 255H without dedicated GPU,with 32 GB RAM), it wasn't even starting thinking, even on a small context window (65k)
i have same 255h and i was able to run it with low token speed 6-8tg/s with approx similar context window 60k
Earlier quoted context omitted.
Google provided incorrect settings and an imperfect template. Unsloth modified the template and then finetuned their own version of the model to optimize for some benchmarks as a means of validating quants. Google and Llama.cpp then adopt template changes by default, so anyone downloading the new model or even using the original model will now automatically be using it incorrectly. Llama.cpp also uses the same infere…
I am very much a beginner to local LLM stuff and I find it incredibly hard to figure out how to run models optimally with the correct settings for my hardware. The number of different variations of the same model and how each quant work is super confusing as well. When I tried to run llama.cpp directly I was getting max 9tk/s on qwen3.5-9B, then I tried LM Studio with the same model and got 77tk/s. I haven't figured…
Earlier quoted context omitted.
Ok, you and I can easily spot LLM text. So what? The Turing test has still been passed, as is clear by people falling in love with ChatGPT, not believing something is AI, and by continuously claiming this or that is a bot. People, many of them at least, cannot make this distinction anymore. You can, I can, but people as a whole are having problems with that.
>You can, I can Even this (assuming it's even true) will likely not be true in some near-term future. >continuously claiming this or that is a bot I see it as a contemporary form of religious thinking. Like (say) pilgrims seeing blood on a statue of the virgin, plenty of people are now seeing the hand of AI in everything they read. If you want to see something hard enough, it tends to become magically visible.
Earlier quoted context omitted.
Is the hardware really getting better? It feels performance per watt is not getting better at all which is the metric that will matter eventually when supply-demand stabilizes. As it is, it seems the improvements are about making the hardware cheaper (as in capex, not opex). This is just feels from me from what I hear on the news and see on the products though.
Solar power and batteries are getting cheaper and cheaper at the moment. So Watts should become cheaper in the long run. Especially when chips are becoming cheaper (in the capex sense), then you can afford to only run them when power is cheap. Btw, from where do you take the notion that performance per Watt ain't increasing? We are also still using what's more or less general purpose GPU hardware; we could get a lot…
GPUs have been getting physically bigger with huge heatsinks and fans to support those bigger dies power consumption. Just compare the TDPs:
2020 RTX 3090: 350W
2022 RTX 4090: 450W
2025 RTX 5090: 575W
Bigger dies means lower capex of course, but the similar opex (maybe slightly lower as there is less physical hardware to maintain).
I seen some specialized hardware like google's TPUs. Not sure how they compare on performance per watt with GPUs though. Regardless the manufacturing processes are still the same (EUV) which is the thing that hasn't been improving. A fully optimized specialized hardware can at most deliver a single-time linear improvement (that could be very significant, for example 30% is still huge of course) and then little compared to normal GPUs.
I don't think renewable power generation is going to massively reduce costs for data centers, especially considering power transmission hasn't meaningfully reduced in cost. If anything the only thing that I think will have significant impact for data centers would be dedicated nuclear power plants physically located right next to the data center.
In fact I expect power generation to get more expensive as demand can increase faster than supply can be established. I imagine setting up new solar farms and transmission lines to be significantly harder (as in, takes longer time due to approvals and so on) than new data centers (which requires a single large location and I assume less approvals).
Earlier quoted context omitted.
>You can, I can Even this (assuming it's even true) will likely not be true in some near-term future. >continuously claiming this or that is a bot I see it as a contemporary form of religious thinking. Like (say) pilgrims seeing blood on a statue of the virgin, plenty of people are now seeing the hand of AI in everything they read. If you want to see something hard enough, it tends to become magically visible.
The most interesting part of your reply is that you're not challenging the claim that the Turing test has been passed. I think it's a given, by now.
Earlier quoted context omitted.
> The reason thinking seems to work is that it's just trying to find an optimum outside the local optimum, and the thinking trace helps find it. Yes, I think I finally have an intuitive sense for that. But surely on a longer prompt it is still better for the final response if the thinking has at least brushed past all of the prompt? One of the things I witnessed with xhigh is that while the thinking trace starts out…
Sure, but interrupting it isn't going to stop it's next output or thought from continuing the progress. I guess if you're literally in a greenfield project, at the start, and it has nothing else to look at, then you're right; but everything I've been doing is taking existing stuff and improving it. So my advice might have a bad edge case that only exists immediately. My setup is to tweak and improve on existing code…
It is sort of frustrating and unnerving to me how much of this is dependent on application and usage, but it is occasionally fascinating.
I run mine on an M5 Max with just 48GB of (V)RAM, and it fits nearly twice in Q4. Works perfectly. I'm kinda glad I didn't spend the extra $2400 to get 128. We don't really need more... and that's a good thing (tm). God knows I thought about it in store. But I thought... maybe this year will be the year of the local model? Maybe soon we won't need that much RAM? I was right. The fact that it runs at 15tk/s in power s…
Have you looked at using oMLX? https://omlx.ai/