371–380 of 411 posts
Earlier quoted context omitted.
On a consumer hardware it would be nicer. With no GPU/iGPU or a 6-8GB VRAM.
It would be somewhat slow on a CPU only machine, but it still works. Besides, Macbooks with 32GB RAM is consumer hardware, just maybe on the higher end.
Earlier quoted context omitted.
I don't think that's true. If a context contains a statement followed by something opposing that statement, that will confuse the model. So "actually wait, I really need to" is there to signal that the previous thinking may be flawed and that what follows is a new attempt. It's good to remember that LLMs have no more state then what they can derive from the context up til any point. So if that context is hard to inte…
As long as the model itself understands that when it says "foo bar" in its own reasoning trace it means "pursue theory foo; no, this didn't lead anywhere, let's backtrack and pursue theory bar", where's the loss in clarity?
Earlier quoted context omitted.
Not if an LLM over chat can fool most people they're talking to a human (which it can), where the TI-99 speech synthesizer voice absolutely can not.
Does that mean it's intelligent? To me it just means they can brilliantly fake human conversation - the original design goal of Large Language Models. It's really easy to tell if you're talking to an LLM if you ask a question that requires actually knowing things, not going for the first search result of a tool call or whatever most popular answer was embedded in the weights. For this reason even the most sophisticat…
That's exactly the criteria we used to assume for over half a century for it finally being intelligent: the Turing Test.
And what does "brilliantly fake human conversation" even mean if not some kind of intelligence? It's like saying "He is not good at math, he just brilliantly proves theorems".
Earlier quoted context omitted.
I think the point is that if people are able to run inference on their laptops batch size efficiency won’t matter. And before that, businesses will be able to get decent results with dedicated inference hardware.
Jevons paradox: large purpose-fit data centers increase efficiency such that you can use AI in more places, and use more tokens for those tasks. The future is not a single chat bot session of bs=1. The future is many agents performing many tasks in parallel for a single user. Large GPU clusters will always have the edge in efficiency.
To what extent can the overthinking be reduced by a better system prompt? For instance, telling it not to overthink, embellish or snazz anything up, and just directly deliver the most what was requested without in the plainest form to build on in future iterations.
I tried "render an svg of five intersecting squares. don't overthink this" and... it overthought it: https://static.simonwillison.net/static/2026/cors-chat.webp
Thinks about it all the time.
Earlier quoted context omitted.
As long as the model itself understands that when it says "foo bar" in its own reasoning trace it means "pursue theory foo; no, this didn't lead anywhere, let's backtrack and pursue theory bar", where's the loss in clarity?
How could it understand, if there's no distinction between that and "from foo logically follows bar"? If the notes it's taking for itself don't make sense, it might just as well not take them.
I wonder if this can be fixed with LORAs.
I had to fix this on 35B A3B -- I have a proxy that just shuts it down if it gets to 2K thinking tokens and injects something like "We have thought enough, let's begin working." and it almost always finishes the turn then. It rarely needs more than 2K thinking tokens and if it does there is always next turn. I would need to see what 27B is actually doing, but these smaller Qwen models seem prone to this.
Earlier quoted context omitted.
The updated version will handle tool calling better by default, but the reasoning quality is no longer preserved and is mutilated quite badly.
So then how do you run it unmutilated?
Then create your own reasoning tests to verify that it is working correctly. You can set a specific seed value to make sure the generation is the same every time, that way you can identify any tokens that are different.
Afterwards, try making small incremental changes to the template and validate your tests each time in order to try to adopt the improvements from the newer templates. If the reasoning quality degrades, undo your changes and try again or test alternative solutions.
Earlier quoted context omitted.
Solar power and batteries are getting cheaper and cheaper at the moment. So Watts should become cheaper in the long run. Especially when chips are becoming cheaper (in the capex sense), then you can afford to only run them when power is cheap. Btw, from where do you take the notion that performance per Watt ain't increasing? We are also still using what's more or less general purpose GPU hardware; we could get a lot…
Like I said, just feels I have from the consumer-hardware space. For several generations of GPU now most improvements come from packing more transistors into a larger die than packing more transistors closer to each other. GPUs have been getting physically bigger with huge heatsinks and fans to support those bigger dies power consumption. Just compare the TDPs: 2020 RTX 3090: 350W 2022 RTX 4090: 450W 2025 RTX 5090: 5…
You can also look at what's been happening in mobile and especially with Apple's integrated processors. They are more power constrained, so people worried more about power there.
Earlier quoted context omitted.
Like I said, just feels I have from the consumer-hardware space. For several generations of GPU now most improvements come from packing more transistors into a larger die than packing more transistors closer to each other. GPUs have been getting physically bigger with huge heatsinks and fans to support those bigger dies power consumption. Just compare the TDPs: 2020 RTX 3090: 350W 2022 RTX 4090: 450W 2025 RTX 5090: 5…
If you have somewhere to put them, you can get 2 440W panels for ~$350 if you're okay with intermittent or mostly daytime use. Or for ~$130/kWh you can extend that with batteries. Then you need an inverter or some kind of regulator, but all in your fully capitalized power is still less than an AMD or Intel GPU and a lot less than an nVidia GPU for home use (the context of this thread is how intelligence is not limite…
I don't think the main use case is for a human to directly interact with the raw token stream. You probably want reasoning and you want the thing to be able to program on its own. That uses way more tokens than you can read.