[dead]
Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed
201–210 of 318 posts
Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed
#202This makes sense to me because I've been having very accurate results with models from even 2+ years ago... but I had to "hold them right." Even when reasoning models and coding agents were just a gleam in Altman's and Amodei's eyes, I could tell a lot of the unrealized gains lay in building the right tools, harnesses and guardrails to manage the context and guide the model. (Relevant subthread as example: https://news.ycombinator.com/item?id=44171519)
But this article hints at deeper wins to be had. Consider that these models are operating on source code, which is a verbose, noisy, textual serialization of the intended syntax / semantic trees. TFA improves accuracy by retro-fitting some structure onto the text. But what if models could operate directly on these underlying structures themselves?
As a data point, there are projects like OpenRewrite, which encode a ton of information, from formatting to types with globally resolved dependencies for each symbol in what they call a "Lossless Semantic Tree", so that there is ~0 ambiguity about the code. When I worked with OpenRewrite (in the era before LLMs, how quaint!) compared to other tools, it produced the best results for code transformations with the highest fidelity to the surrounding code.
Now imagine if the agent has access to such detailed information. It would not have to waste tokens figuring incidental things out like formatting. Although I haven't tested it out myself, I believe Moderne (the maintainers of OpenRewrite) when they say that agents armed with LST-based tools make extremely accurate changes.
This is essentially the same reason why the answer to "Which is better, Vim or Emacs?" is "IntelliJ."
Now consider that these models are STILL operating on text as an input and output mode! What if they were multi-modally trained on source code and docs and their syntax / semantic trees? I don't even know what this would look like, but I'd bet this would produce the most accurate coding models ever -- probably neurosymbolic in the truest sense.
Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed
#203Earlier quoted context omitted.
Good point! I just wonder how unique these hashes will be if only 2 characters. It seems like the collision rate would be really high.
we dug into those sorts of questions with hypertokens, a robust hash for lines, code, tables/rows or any in-context token tagging to give models photographic memory one mechanism we establish is that each model has a fidelity window, i.e., r tokens of content for s tag tokens; each tag token adds extra GUID-like marker capacity via its embedding vector; since 1,2,3 digit numbers only one token in top models, a single…
For others, here's the paper: https://arxiv.org/abs/2507.00002
Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed
#204Earlier quoted context omitted.
how do you make them?
On macOS, Option+Shift+- and Option+- insert an em dash (—) and en dash (–), respectively. On Linux, you can hit the Compose Key and type --- (three hyphens) to get an em dash, or --. (hyphen hyphen period) for an en dash. Windows has some dumb incantation that you'll never remember.
For Windows it's just easier to make a custom keyboard layout and go to town with that: https://www.microsoft.com/en-us/download/details.aspx?id=102...
Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed
#205Earlier quoted context omitted.
2026 is the year of the harness.
Already made a harness for Claude to make R/W plans, not write once like they are usually implemented. They can modify themselves as they work through the task at hand. Also relying on a collection of patterns for writing coding task plans which evolves by reflection. Everything is designed so I could run Claude in yolo-mode in a sandbox for long stretches of time.
Link?
Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed
#206I really enjoyed this article. I think the author is precisely right and I've been saying this for a long time. There's a ton of extremely interesting low hanging fruit that can vastly improve the effectiveness of even currently existing models hiding in how we design our agent harnesses; enough to — at least until we hit diminishing returns — make as much or more of a difference than training new models! I think one…
My Weird Hill is that we should be building things with GPT-4. I can say unironically that we haven't even tapped the full potential of GPT-4. The original one, from 2023. With no reasoning, no RL, no tool calling, no structured outputs, etc. (No MCP, ye gods!) Yes, it's possible to build coding agents with it! I say this because I did! Forcing yourself to make things work with older models forces you to keep things…
The problem with these exercises is always: I have limited time and capacity to do things, and a fairly unlimited number of problems that I can think of to solve. Coding is not a problem I want to solve. Prompt engineering is not a problem I want to solve.
If I do things for the love if it, the rules are different of course. But otherwise I will simply always accept that there are many things that improve around me, that I have no intimate knowledge of and probably never will, and I let other people work them out and happily lean on their work to do the next thing I care about, that is not already solved.
Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed
#207You can improve a lot the success rate with providing HELM and clear instructions with the tool description.
Over a year ago had a lot of issues and the description and example was the difference between 30-50% failure to 1%!
So I'm surprised a bit about the point. May be I'm missing it.
Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed
#208Seeing all these 'coding' benchmarks reminds me that people still don't understand what coding means in practice. People still think one-phase puzzle-solving is coding. Real coding almost always has multiple phases which build on top of one another. There is an architectural component which is missed here - and the sheer number of phases/layers is actually where most of the complexity comes from.
Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed
#209Google banning you for benchmarking is crazy, are you sure that's the cause? How would they even know you are benchmarking?
Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed
#210I really enjoyed this article. I think the author is precisely right and I've been saying this for a long time. There's a ton of extremely interesting low hanging fruit that can vastly improve the effectiveness of even currently existing models hiding in how we design our agent harnesses; enough to — at least until we hit diminishing returns — make as much or more of a difference than training new models! I think one…
2026 is the year of the harness.
2027 is the year of the "maybe indeterminism isn't as valueable as we thought"