Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

blog.can.ac

Post reply on HN
Print view View original

161–170 of 318 posts

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#161

Post by aghilmort »

azinman2 wrote: post #124
renewiltord wrote: post #109

Earlier quoted context omitted.

Forces you to read after every write. E.g. you edit line 15 to be two lines. Then now you need arithmetic for later vs earlier lines or you need to read full file to reindex by line number.

Good point! I just wonder how unique these hashes will be if only 2 characters. It seems like the collision rate would be really high.

we dug into those sorts of questions with hypertokens, a robust hash for lines, code, tables/rows or any in-context token tagging to give models photographic memory

one mechanism we establish is that each model has a fidelity window, i.e., r tokens of content for s tag tokens; each tag token adds extra GUID-like marker capacity via its embedding vector; since 1,2,3 digit numbers only one token in top models, a single hash token lacks enough capacity & separation in latent space

we also show hash should be properly prefix-free, or unique symbols perp digit, e.g., if using A-K & L-Z to hash then A,R is legal hash whereas M,C is not permitted hash

we can do all this & more rather precisely as we show in our arXiv paper on same; next update goes deeper into group theory, info theory, etc. on boosting model recall, reasoning, tool calls, etc. by way of robust hashing

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#162

Post by aghilmort »

a11r wrote: post #24

This is very nicely done. We have seen the same issue at a higher level of getting separators right when generating multiple files in a single inference call.

curious: wdym by "getting separators right when generating multiple files in a single inference call"

context: created hypertokens an even more robust hashing mechanism to create context-addressable memory (CAM), one cheat code is make them prefix-free, lots of others that get deep into why models work the way they do, etc.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#163

Post by vardalab »

ahofmann wrote: post #80
fazgha wrote: post #71

Earlier quoted context omitted.

[flagged]

Em dashes are used often by LLMs, because humans use them often. On mac keyboards its easily typed. I know this is oversimplifying the situation, but I don't see the usefulness of the constant witch-hunting for allegedly LLM-generated text. For text we are long beyond the point, where we can differenciate between human generated and machine generated. We're even at the point, where it gets somewhat hard to identify m…

Yeah, I agree with you. I'm so tired of people complaining about AI-generated text without focusing on the content. Just don't read it if you don't like it. It's another level of when people complain how a website is not readable for them or some CSS rendering is wrong or whatever. How does it add to the discussion?

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#164

Post by drob518 »

logicprog wrote: post #66
logicprog wrote: post #65

I really enjoyed this article. I think the author is precisely right and I've been saying this for a long time. There's a ton of extremely interesting low hanging fruit that can vastly improve the effectiveness of even currently existing models hiding in how we design our agent harnesses; enough to — at least until we hit diminishing returns — make as much or more of a difference than training new models! I think one…

Also, yes, I'm aware that I use a lot of "its not just X, its Y." I promise you this comment is entirely human written. I'm just really tired and tend to rely on more wrote rhetorical tropes when I am. Believe me, I wrote like this long before LLMs were a thing.

That's what all the AIs have been trained to say.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#165

Post by jahala »

hedgehog wrote: post #136
jahala wrote: post #131

I implemented this hash (read and edit) approach in tilth if you want to test it out. https://github.com/jahala/tilth its on npm and cargo: - cargo install tilth - npx tilth then tilth install claude-code/windsurf/cursor --edit (--edit flag is needed) I made "tilth" a few days ago, since I'm consistently trying to get the LLMs to use tools more efficiently and spend less tokens doing it -- original tilth post from Mo…

You might find it useful for markdown as well, especially if you add support for section-based addressing (e.g. cat or replace a section at a time). Section-based addresses are nice because they tend to be stable across versions.

Great idea - Just implemented this.

(Already published on cargo, on npm in a few mins).

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#166

Post by HarHarVeryFunny »

babkayaga wrote: post #133

Still weird to me that most people are not just giving an LLM an access to an editor, forcing it to write shell scripts to edit files. Shrug.

That's not quite how it works, and anyways if the model can't generate an accurate find/replace string, why would you expect it to do any better generating accurate commands to drive your editor (assuming it knew how do do that in the first place) ?!

The way edits happen is that the agent (local) first tells the model (typically remote) that it has an edit tool (e.g. taking parameters file name, find string and replace string). If the model decides it wants to edit a file then it'll invoke this edit tool, which just results in a blob of JSON being put in the model's response specifying the edit (filename, etc). The agent then receives the response, intercepts this JSON blob, sees that it is an edit request and does what is asked.

The problem the article is describing is that the edit request (tool invocation) generated by the model isn't always 100% accurate. Even if the agent told the model it had a tool to invoke an actual editor, say sed, assuming the model knew how to use sed, this is still going to fail if the edit request cannot be interpreted literally by the editor (due to being inaccurate).

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#167

Post by drob518 »

aeon_ai wrote: post #69
logicprog wrote: post #65

I really enjoyed this article. I think the author is precisely right and I've been saying this for a long time. There's a ton of extremely interesting low hanging fruit that can vastly improve the effectiveness of even currently existing models hiding in how we design our agent harnesses; enough to — at least until we hit diminishing returns — make as much or more of a difference than training new models! I think one…

Once you begin to see the “model” as only part of the stack, you begin to realize that you can draw the line of the system to include the user as well. That’s when the future really starts hitting you.

That's next-year's problem.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#168

Post by dolebirchwood »

fazgha wrote: post #71
logicprog wrote: post #65

I really enjoyed this article. I think the author is precisely right and I've been saying this for a long time. There's a ton of extremely interesting low hanging fruit that can vastly improve the effectiveness of even currently existing models hiding in how we design our agent harnesses; enough to — at least until we hit diminishing returns — make as much or more of a difference than training new models! I think one…

[flagged]

I really despise that people like you ruined em dashes for the rest of us who have enjoyed using them.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#169

Post by MrGreenTea »

azinman2 wrote: post #124
renewiltord wrote: post #109

Earlier quoted context omitted.

Forces you to read after every write. E.g. you edit line 15 to be two lines. Then now you need arithmetic for later vs earlier lines or you need to read full file to reindex by line number.

Good point! I just wonder how unique these hashes will be if only 2 characters. It seems like the collision rate would be really high.

The author writes that these hashes are 2 or 3 characters long. I assume depending on the line count. That's good for almost 48k lines. You have other issues then.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#170

Post by woah »

Seems like a very cool technique, but also very oversold. He's seeing a 5% improvement on a find and replace benchmark of his own devising and saying stuff like this in the blog post:

> Here is why that is backwards. I just showed that a different edit format improves their own models by 5 to 14 points while cutting output tokens by ~20%. That’s not a threat. It’s free R&D.

He makes it sounds like he got a 5-14% boost on a top level benchmark, not 5% improvement on a narrow find and replace metric. Anecdotally, I don't usually have a lot of issues with editing in Claude Code or Cursor, and if there is an issue the model corrects it.

Assuming that it costs double the tokens when it has to correct itself, and find and replace errors are as prominent in actual day to day use as his benchmark, we're talking a 5% efficiency gain in editing token use (not reasoning or tool use). Given that editing must be less than 1/3 of the token use (I assume much less?), we're talking an overall efficiency gain of less than 1%.

This seems like a promising technique but maybe not a high priority in efficiency gains for these tools. The messianic tone, like assuming that Google cut off his access to suppress his genius editing technique rather than just because he was hammering their API also leaves a bad taste, along with the rampant and blatant ChatGPTisms in the blog post.

Post reply on HN