Interesting. I use Flash for making the plans and GPT for execution.
Depending on the language you're writing in and the problem domain, the smaller models can do dramatically better or worse. I suspect in the future we'll see language-specific small models. "Coding" is still pretty broad as an activity. It'd be nice to be able to load up a model specific to, say, class-based Python and run it on-device.
Harmonic's Aristotle is a sort-of language specific model for Lean, if you want to see the future you described today.
How do you define intelligence? I encounter that kind of sentiment all too often, and I have to assume we go by wildly different understanding of what that might entail.
My definition is that I can be much less precise with AI the more intelligent it is. It can extract the intent from my fuzzy description of the problem. Which means I can offload some of the thinking effort. It wasn't possible a couple years ago. I used to make fun of people who were trying to get ChatGPT to think about the problem when all it could do was write code from the pseudocode you provide. But now I can say…
Sounds like a good working definition in the context you are using it in.
> But now I can say: "Look at the latest log and make a plan to fix". And it takes it from there.
I usually tell the agents to first work on reliably reproducing the problem in the log, and only then even start thinking about a fix.
How do you define intelligence? I encounter that kind of sentiment all too often, and I have to assume we go by wildly different understanding of what that might entail.
My definition is that I can be much less precise with AI the more intelligent it is. It can extract the intent from my fuzzy description of the problem. Which means I can offload some of the thinking effort. It wasn't possible a couple years ago. I used to make fun of people who were trying to get ChatGPT to think about the problem when all it could do was write code from the pseudocode you provide. But now I can say…
I think this is the best and most useful way to measure model intelligence. In my experience it's what really sets apart the capable models from the best. A small model can be RL trained to be extremely good at programming or narrow problem solving for its size (eg 5.6 Luna, DS4 Flash, Qwen 3.6 27B), but even Luna is IME comparatively awful at understanding intent and making good decisions with limited guidance.
I don’t understand why people are calling these transformers models non-deterministic? Are you referring to the temperature parameter? I haven’t played with transformer internals in a while but my understanding is that if the temperature is fixed at a value where the top logit is always picked, then because they weights are fixed, the exact same input should produce the exact same output. Am I missing something?
I think people are wrapping that across the English language. In English, these two tasks are exactly the same: "Would you hand me that item?" "Please hand me that item" But when posed to the LLM, they generate different outputs. One character difference in the prompt might be a whole different output. People who aren't programmers mostly don't know that there's any difference. They asked for the same thing, it knows…
I'm not sure that's true. Sure, in the end I might hand them the item, but my thoughts about what they said will be different. I think you have to consider my thoughts "output" for this comparison to be valid.
IME I can't trust it to write it's own plans from a spec, but if I give it a detailed execution plan written by Opus, it's fast and cheap (if chatty) in executing it.
Interesting. I use Flash for making the plans and GPT for execution.
flash for plans?! i don't understand why you wouldnt use something far stronger for the most load bearing point of the project
Why does this link to OpenRouter, which has no useful information on its own? Linking to the official API or the benchmarks would make more sense: - https://api-docs.deepseek.com/ - https://x.com/ChrisGPT/status/2087572834650407024/photo/1 (officially posted on WeChat, this is just one of many reposts)
Don’t you find the official document website was out of service for a long time since the new model was published soon?
50% cache hit is really low - in a standard agentic loop you should expect like 99%+ cache hit percentage (which should also lower that $12.50 to like a couple of $ for the same amount of tokens). If you're using a customised harness you should make sure you don't have something that's e.g. changing your system prompt on some requests or rewriting history - it can be tempting to do stuff like strip old thinking token…
In my experience, that's the OpenRouter tax. Even a session that does everything right to remain sticky ends up getting moved between providers on a few requests, which bills you the full context as input every time the switch happens. I assume it's done as load balancing/latency mitigation, but it's put me off of OpenRouter for my use cases (limited use, limited need for changing models).
It is a pain from openRouter if you don't define your providers correctly, but for DeepSeek, surely not- the weights aren't released yet and there's only one provider, DeepSeek.
Maybe it's me but I don't see how DS Flash is better than GLM at all, much less by a huge gap. I'd probably protest less against Fable and Opus being put at the same level than many would, but there's no denying the two models are a very different experience from each other. I guess where I'm going is no one should pick a model by the benchmarks.
I'm not the most LLM-savvy person around, and I'm not gonna say I've put a ton of effort into practically compared these open models. But, a month or two ago I did do some "practical evaluates" testing GLM 5.2 versus DSv4 (flash/pro) with OpenCode's subscription with some late 80s Unix clone-type work, and this jives with my experience. GLM ended up being far slower, and far more expensive, for approximately the same…
Exactly the same experience. I really loved GLM5.2 for a while, but after trying it again after riding DSv4 flash (new) for a while, they're mostly at the same capabilities, with GLM being slower and much, much more expensive. A task cost me 2$ where it did very wrong, whereas Flash nailed it almost instantly for like a rounding error on my billing page.