81–90 of 318 posts
> Treating harnesses as solved, or even inconsequential, is very short-sighted Is it possible that burning extra tokens is the point, since they get paid more?
> Treating harnesses as solved, or even inconsequential, is very short-sighted Is it possible that burning extra tokens is the point, since they get paid more?
On first principles it would seem that the "harness" is a myth. Surely a model like Opus 4.6/Codex 5.3 which can reason about complex functions and data flows across many files would trip up over top level function signatures it needs to call? I see a lot of evidence to the contrary though. Anyone know what the underlying issue here is?
Earlier quoted context omitted.
At this point subsidizing Chinese open-weights vendors by paying for them is just the right thing to do. Maybe they too might go closed-weights when they become SotA, but they're now pretty close and haven't done it.
I am wondering what kinds of harness are best for GLM, Deepseek, Qwen, Kimi.
Edit
Checking ohmypi The model has access to str replace too so this is just a edit till
Earlier quoted context omitted.
Also, yes, I'm aware that I use a lot of "its not just X, its Y." I promise you this comment is entirely human written. I'm just really tired and tend to rely on more wrote rhetorical tropes when I am. Believe me, I wrote like this long before LLMs were a thing.
why the long -'s
As Emacs has a built-in tree-sitter package, I implemented this same idea. I created gptel tools like tree_sitter_list_nodes, tree_sitter_get_nodes, tree_sitter_update_nodes, tree_sitter_insert_before_node and tree_sitter_insert_after_node. The "list" tool returns a list of AST nodes with first line number, first line content and node hash. The LLM can then use "get" to collect interesting nodes in their entirety and "update" to update a list of nodes identified by hash with new content (var/function bodies).
Worked like a charm.
I really enjoyed this article. I think the author is precisely right and I've been saying this for a long time. There's a ton of extremely interesting low hanging fruit that can vastly improve the effectiveness of even currently existing models hiding in how we design our agent harnesses; enough to — at least until we hit diminishing returns — make as much or more of a difference than training new models! I think one…
Once you begin to see the “model” as only part of the stack, you begin to realize that you can draw the line of the system to include the user as well. That’s when the future really starts hitting you.
On first principles it would seem that the "harness" is a myth. Surely a model like Opus 4.6/Codex 5.3 which can reason about complex functions and data flows across many files would trip up over top level function signatures it needs to call? I see a lot of evidence to the contrary though. Anyone know what the underlying issue here is?
If smaller labs (Zai, Moonshot, deepseek, mistral..) get together and embrace a harness, like opencode for example, as a consortium just by the power of "evolution across different environments" they might hit jackpot earlier than bigger labs.