DeepSeek V4 Pro 0813

For me, flash 0731 was much better in omp/opencode than in Pi.

Anyway, it might be so that they are rolling out deployment. There haven't been an official announcement post yet (this submission is a link to openrouter). Some people have been saying they are getting results worse than GLM-5.1, that's obviously broken.

the breadth and width of the universe of oneshot challenges are all arbitrary. It's unsurprising different workflows oneshot better than others.

All the more reason to favor local models under your control, as once you find that sweet spot model, no one can change it, upgrade it, align it, take it down or otherwise harm the time investment you made it making it your own.

I can't really believe no one understands, after decades, how valueable a rock solid development environment is.

Because complex problems can be decomposed (a skill in itself) into easy parts and hard parts; and the hard parts are almost always bottlenecked on understanding concepts and principles (i.e. things that are either in a model's weights, or not), not on having certain facts available. Models can solve complex problems insofar as they can decompose those problems, and have learned the concepts and principles relevant to approaching the hard parts of those problems.

Whereas tool-use isn't a capability problem, but a context problem: the thing that makes models fail by default is that they have no idea, when first summoned out of the aether, what kind of conversation they're having, who it's with, what that person is trying to do, what tools they have available, and how those tools can be invoked.

Think of the difference between how you'd respond to a casual programming question asked by a person sitting next to you on a flight, vs. a programming question asked of you by someone you're pair-programming with with your IDE open in front of you. Now imagine waking up blind and deaf and needing to discern which of the two situations you're in. LLMs know how to approach both of these problem-contexts (and more besides), but they need to be given context to know which problem-context they're in (and everything else about that problem-context: which IDE they're using, which OS it's installed on, what other tools are installed+accessible, etc.)

And before you say "but why can't they just experiment to figure these things out" — if you think about it, knowing how to interface with a shell and an IDE are bootstrapping requirements for any kind of experimentation, in about the same way that "knowing how to open your eyes and move your head" is a bootstrap requirement for a human gaining information about the world around them. These capabilities are necessary to explore the world to "discover" and "probe" other capabilities.

---

Also, a lot of the work LLMs do "needs" (i.e. is heavily improved by the use of) some kind of structured scratchpad, that they have been trained to manipulate and "look at" through tool-use. Even for a human who could accurately visualize a canvas based on a coordinate system, you still wouldn't expect said human to succeed at the pelican test if they had to write the SVG entirely in their head and then write it out sequentially with no rewinding to fix mistakes. You'd expect them to ask for at least a whiteboard, if not a text editor, to be able to write and rewrite the SVG XML.

(Really, they'd ideally want to run the SVG and look at it to see how close it is, and optimize that way. I'm not sure if we're letting LLMs do that part in the classical pelican test. It feels like that would vaguely violate the "zero-shot"-ness of the test, though I'm not sure if we're currently considering a conversation to be "zero-shot" if it involves the model iteratively interacting with a third-party system [such that there are repeated model -> system -> model conversation turns] but holding off to responding to the user until they think they've fully solved the problem.)

---

And also, on a lower level, all of these external capabilities are getting exposed to the LLM through MCP. Models can and do understand how to speak MCP itself. But there's no standard for how a given harness's capabilities (e.g. "execute command line in new shell session", "send patch edit command to active tab in IDE", etc) should be modelled to be exposed through MCP, either in their encoding or in their semantics. There's no MCP equivalent of WASM's WASI meta-standard, such that models could learn these specs and "assume by default" that things work like them until told otherwise; and nor are there even open harnesses that LLMs could learn about during training, and through them, learn some de-facto MCP-endpoint specs. Instead, there are mostly just proprietary harnesses, that hide all that info from public access, sharing it only with the LLM during inference, and even then, only at the moment the LLM needs it.

so?????