Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed
1–10 of 318 posts
Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed
#2I feel the baseline comparison should be relative to the intuitive and simple "line-numbers only" schema.
It's less token heavy than the proposed hash approach, and I don't think frontier LLMs hallucinate line numbers if each line in the context is prefixed with them.
Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed
#3I feel the baseline comparison should be relative to the intuitive and simple "line-numbers only" schema. It's less token heavy than the proposed hash approach, and I don't think frontier LLMs hallucinate line numbers if each line in the context is prefixed with them.
The issue is when the file changed between when the LLM read the file and when it wrote to the file. Just using line numbers will clobber a file if that happens. The hashes prevent that from being an issue.
Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed
#4I feel the baseline comparison should be relative to the intuitive and simple "line-numbers only" schema. It's less token heavy than the proposed hash approach, and I don't think frontier LLMs hallucinate line numbers if each line in the context is prefixed with them.
The issue is when the file changed between when the LLM read the file and when it wrote to the file. Just using line numbers will clobber a file if that happens. The hashes prevent that from being an issue.
Point taken.
Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed
#5I wonder if we'll get to "VI for LLMs" - if the model was trained on using that kind of text navigation and you show context around cursor when it navigates.
Would also be worth having special tokens for this kind of navigation.
Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed
#6The harness matters far more than most people think. This post about the CORE benchmark where Opus’ score almost doubled when they switched to Claude Code from their own harness. https://x.com/sayashk/status/1996334941832089732
Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed
#7The harness matters far more than most people think. This post about the CORE benchmark where Opus’ score almost doubled when they switched to Claude Code from their own harness. https://x.com/sayashk/status/1996334941832089732
Which, IMHO, should be why we should be able to change them freely or make our own. Being locked into a specific harness because you pay 20 bucks per month vs. pay-per-use ... is kinda dumb.
Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed
#8I wonder if we'll get to "VI for LLMs" - if the model was trained on using that kind of text navigation and you show context around cursor when it navigates. Would also be worth having special tokens for this kind of navigation.
I bet it’s good enough at VI already
Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed
#9I use small model I like to give them TOC more than lines wonder how it'd stack up with the hashline approach
read_toc tool:
...
{
"name": "mcp",
"qualified_name": "mcp",
"type": "constant",
"docstring": null,
"content_point": "src\\mcps\\code_help\\server.py::17::18::python::mcp",
"is_nested": false
},
{
"name": "handler",
"qualified_name": "handler",
"type": "constant",
"docstring": null,
"content_point": "src\\mcps\\code_help\\server.py::18::19::python::handler",
"is_nested": false
},
update_content tool:
{
"content": "...",
"content_point": "src\\mcps\\code_help\\server.py::18::19::python::handler",
"project_root": ....
}Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed
#10Great work, but concurrency is lost.
With search-replace you could work on separate part of a file independently with the LLM. Not to mention with each edit all lines below are shifted so you now need to provide LLM with the whole content.
Have you tested followup edits on the same files?