Worse than Luna but more expensive than Luna. Sticking with Luna without sending my data to Deepseek (China)
At least you can run it for relatively cheap hardware. I guess OpenAI doesn't let you do that.
DeepSeek V4 Pro 0813
161–170 of 493 posts
Re: DeepSeek V4 Pro 0813
#162Earlier quoted context omitted.
If the model cannot figure out simple and ubiquitous tools, how is it supposed to figure out complex problems? All of the good models basically work with any harness, including giving them a single "shell command" tool. They can just figure things out.
When it comes to quality of outcome, since at least Feburary, the harness has almost equal, if not more weight than the model itself. It's no longer "which model is the best?" it's "which model + harness is the best?" I get drastically different tool call failure rates using Claude SDK vs OpenCode using Qwen 3.6 models
It's worth nothing that recent Claude models seem to have gotten worse at tool calling outside of Claude Code and the SDK: https://lucumr.pocoo.org/2026/7/4/better-models-worse-tools/
Re: DeepSeek V4 Pro 0813
#163Earlier quoted context omitted.
Geometric mean of all these benchmarks : * GPT-5.6 Sol: 65.5 * Fable 5 (w/ fallback): 64.5 * Opus 5: 64.0 * DS-V4-Pro 0813: 62.5 * Kimi-K3: 62.3 * DS-V4-Flash 0731: 55.8 * GLM-5.2: 47.3
Maybe it's me but I don't see how DS Flash is better than GLM at all, much less by a huge gap. I'd probably protest less against Fable and Opus being put at the same level than many would, but there's no denying the two models are a very different experience from each other. I guess where I'm going is no one should pick a model by the benchmarks.
In my little social circle DS4F generally substitutes for GLM 5.2 except it's the next best thing to free.
Re: DeepSeek V4 Pro 0813
#164Earlier quoted context omitted.
+1, a single test means little.
I don't think so. I specifically kept this PR to test model capabilities, and I've already tested a bunch of models. Current test results show that the more advanced the model is, the easier it passes. For example, GPT-5.5 Medium fails the test(has bug), but High passed.
They are causal autoregressive models, the output is sensitive even to the implementation nuances in inference. Even 1 token that's badly selected could throw off the entire answer.
Re: DeepSeek V4 Pro 0813
#165Benchmarks: | Benchmark | DS-V4-Pro | DS-V4-Flash | DS-V4-Pro | DS-V4-Flash | GLM-5.2 | Kimi-K3 | Opus-4.8 | Fable 5 | | | 0813 | 0731 | Preview | Preview | | | | (w/ fallback) | |--------------------------|-----------|-------------|-----------|-------------|-----------|-----------|-----------|---------------| | HLE (wo/w tools) | 42.7/60.0 | 37.8/51.5 | 37.7/48.2 | 34.8/45.1 | 40.5/54.7 | 43.5/56.0 | 49.8/57.9 | 53.…
The most interesting part of this is how Flash scores almost as well on all of them.
Haven't tried the new DeepSeek models but I'm assuming the difference is more than these numbers show!
Re: DeepSeek V4 Pro 0813
#166Even though cost-per-token is low, Deepseek v4 tends to burn an immense number of tokens to accomplish tasks.
It still ends up being one or two orders of magnitude cheaper per task on benchmarks.
Re: DeepSeek V4 Pro 0813
#167This model is not very good at coding, but it is quite good at research, evaluation and action, I don't write code, but it really goes head-to-head with the most expensive models in searches such as stock market and forex
[dead]
Re: DeepSeek V4 Pro 0813
#168I'm Satisfied with this model (in opencode)
Re: DeepSeek V4 Pro 0813
#169Just tested through openrouter.. gave exactly same task.. the task was to scan existing repo, and generate a single docker-compose file to deploy behind a caddy server, where certain port ranges are already used, the service demands widlcard certificates to be provisioned from outside, and postgre needs to be built-in one... Tested this model, and gpt-5.6-terra-high. Results: this one had few issues. terra: none. The…
Terra has not been able to do any of the technical tasks I've asked of it correctly. I'm surprised others get use out of it. Anything below Sol high tends to give me mostly unreliable results. I'm using codex as my main harness but maybe it performs better with a different one.
I am seeing essentially deterministic results with Terra running a custom browser automation agent across >100 interaction events.
The harness is everything. If I just threw something like Codex at this and said "good luck" I wouldn't make it beyond 5-10 interactions. I tried that already. Carefully designing the views and tools over the environment is where you can go from 50% to 99.9999%.
Re: DeepSeek V4 Pro 0813
#170Just tested through openrouter.. gave exactly same task.. the task was to scan existing repo, and generate a single docker-compose file to deploy behind a caddy server, where certain port ranges are already used, the service demands widlcard certificates to be provisioned from outside, and postgre needs to be built-in one... Tested this model, and gpt-5.6-terra-high. Results: this one had few issues. terra: none. The…
wait for Deepseek Harness (yes it is the official name) release then try again. for your kind of task, harness tools matter.
I use pi harness with codex and all the tool calls are custom delegate extensions, I mean ALL(for security checks), i get consistently good results from sol on high and xhigh reasoning. I don't believe harness should matter because its at most just a way to abstract tool calls and maybe the system prompt. Training on the tool calls results should not(and in codex's case does not matter)