if you want to quickly try it on codex https://github.com/tartavull/hashline
Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed
281–290 of 318 posts
Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed
#282I really enjoyed this article. I think the author is precisely right and I've been saying this for a long time. There's a ton of extremely interesting low hanging fruit that can vastly improve the effectiveness of even currently existing models hiding in how we design our agent harnesses; enough to — at least until we hit diminishing returns — make as much or more of a difference than training new models! I think one…
[deleted]
Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed
#283Look at Serena. Does something on this line.
> it can use code-centred tools like find_symbol, find_referencing_symbols and insert_after_symbol.
Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed
#284[deleted]
Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed
#285The harness is the model "body", it's weight the cognition. Like in nature they develop together and the iteration of natural selection works at both. If smaller labs (Zai, Moonshot, deepseek, mistral..) get together and embrace a harness, like opencode for example, as a consortium just by the power of "evolution across different environments" they might hit jackpot earlier than bigger labs.
But they rely on distilling the output of american leader models. Which will probably train against their own harness. Someone has to do the baseline training, development, and innovation. it can't be clones all the way down
It goes the other way around as well. DeepSeek has made quite a few innovations that the US labs were lacking (DSA being the most notable one). It's also not clear to me how much of distilled outputs are just an additional ingredient of the recipe rather than a whole "frozen dinner" so to speak. I have no evidence to say it's one way or the other, but my guess is the former.
Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed
#286It seems like agentic (or atleast AI-assisted coding) is the future. And we will be increasingly relying on these models to earn our livelihood. Is anyone else worried at how easily Anthropic/Google/OpenAI can basically cut you off if you do something they don't like?
It's a very concerning future. I would love to live in a world where we could simply stop them from doing that, but for the moment, the best hedge appears to be the Chinese open weight models that can't be put back in the box and provide the valuable market function of commodifying the encoded knowledge of these models (which in and of itself was derived from knowledge not created by the frontier lab).
Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed
#287I really enjoyed this article. I think the author is precisely right and I've been saying this for a long time. There's a ton of extremely interesting low hanging fruit that can vastly improve the effectiveness of even currently existing models hiding in how we design our agent harnesses; enough to — at least until we hit diminishing returns — make as much or more of a difference than training new models! I think one…
My Weird Hill is that we should be building things with GPT-4. I can say unironically that we haven't even tapped the full potential of GPT-4. The original one, from 2023. With no reasoning, no RL, no tool calling, no structured outputs, etc. (No MCP, ye gods!) Yes, it's possible to build coding agents with it! I say this because I did! Forcing yourself to make things work with older models forces you to keep things…
reminds me of the Nintendo strategy “lateral thinking with withered technology”
Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed
#288The harness is the model "body", it's weight the cognition. Like in nature they develop together and the iteration of natural selection works at both. If smaller labs (Zai, Moonshot, deepseek, mistral..) get together and embrace a harness, like opencode for example, as a consortium just by the power of "evolution across different environments" they might hit jackpot earlier than bigger labs.
Mistral recently came out with their own harness (vibe) and I feel like it was a massive missed opportunity vs throwing in with with aider or opencode.
Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed
#289I enjoyed the post. Sometimes small hacks go a long way. I still feel like the game changer will be models outputting just enough code for replacement and harness replacing it efficiently rather than whole code with find and replace hacks.
Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed
#290Earlier quoted context omitted.
My Weird Hill is that we should be building things with GPT-4. I can say unironically that we haven't even tapped the full potential of GPT-4. The original one, from 2023. With no reasoning, no RL, no tool calling, no structured outputs, etc. (No MCP, ye gods!) Yes, it's possible to build coding agents with it! I say this because I did! Forcing yourself to make things work with older models forces you to keep things…
> semantic > grep def Once you get to a codebase beyond a certain size, that no longer works. I've for one found Serena https://github.com/oraios/serena , which you can install from right within Claude, to be a fairly fantastic code-interaction tool for LLM's. Both semantic search as well as editing. And with way less token churn.
This is definitely a cool finding.
Have you investigated more on this topic? like, anything similar in concept that competes with Serena? if so, have you tested it/them? what are your thoughts?