Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

blog.can.ac

Post reply on HN
Print view View original

151–160 of 318 posts

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#151

Post by znnajdla »

parhamn wrote: post #141
znnajdla wrote: post #138

Earlier quoted context omitted.

How hard is it to for you to assemble a piece of IKEA furniture without an allen wrench, screwdriver, and clear instructions, vs with those 3?

You didn't read the article it seems (or the analogy is a bad one). The differences are much more subtle than having a screwdriver or not.

I did read the article quite enthusiastically and my practical experience confirms the same. Sure the difference is more subtle. But my point was, an "agent" whether human or AI can be a lot more productive with better tools. This guy found a better screwdriver than the most commonly used one. That's amazing and nothing from "first principles" denies that a better tool harness would mean better/faster/more correct AI agents.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#152

Post by jahala »

kachapopopow wrote: post #145
jahala wrote: post #131

I implemented this hash (read and edit) approach in tilth if you want to test it out. https://github.com/jahala/tilth its on npm and cargo: - cargo install tilth - npx tilth then tilth install claude-code/windsurf/cursor --edit (--edit flag is needed) I made "tilth" a few days ago, since I'm consistently trying to get the LLMs to use tools more efficiently and spend less tokens doing it -- original tilth post from Mo…

benchmarks vs grep?

tilth isn’t trying to replace grep for raw text search — for that, it wraps ripgrep internally so perf is comparable. It’s about reducing round-trips and giving the agent a verified edit workflow, not faster search.

Instead of cat + grep + manual line counting, one tool call returns a structural outline of a large file, lets you drill into sections, and since this last update also returns hashline-anchored output that an edit tool can target.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#153

Post by StilesCrisis »

ahofmann wrote: post #80
fazgha wrote: post #71

Earlier quoted context omitted.

[flagged]

Em dashes are used often by LLMs, because humans use them often. On mac keyboards its easily typed. I know this is oversimplifying the situation, but I don't see the usefulness of the constant witch-hunting for allegedly LLM-generated text. For text we are long beyond the point, where we can differenciate between human generated and machine generated. We're even at the point, where it gets somewhat hard to identify m…

I might not be able to spot ALL AI generated text, but I can definitely spot some. It's still kind of quirky.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#154

Post by kachapopopow »

jahala wrote: post #152
kachapopopow wrote: post #145

Earlier quoted context omitted.

benchmarks vs grep?

tilth isn’t trying to replace grep for raw text search — for that, it wraps ripgrep internally so perf is comparable. It’s about reducing round-trips and giving the agent a verified edit workflow, not faster search. Instead of cat + grep + manual line counting, one tool call returns a structural outline of a large file, lets you drill into sections, and since this last update also returns hashline-anchored output tha…

well yah, that's what I mean how better is it versus cat + grep + manual line counting. Agents tend to perform worse with niche tools

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#155

Post by canadiantim »

senordevnyc wrote: post #114
robbomacrae wrote: post #63

Earlier quoted context omitted.

Indeed. The biggest waste might be the overuse of MCP for everything. Sure it makes the initial development easier but then for every connection you're using a hundred billion dollar parameter model to decide how to make the call when it's usually completely unnecessary and then prone to random errors. MCP is the hammer that can make literally everything look like a nail...

I see this ranting against MCP all the time, and I don't get it, maybe I'm missing something. I'm currently using an MCP in Cursor to give agents read-only access to my staging and prod databases, as well as BugSnag's MCP so it can look up errors that happen in those environments. It works great . What should I be using for this if not MCP?

agent skills, or use claude code to iteratively condense an MCP you want to use into only its most essential tools for your workflow

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#156

Post by co_king_3 »

logicprog wrote: post #66
logicprog wrote: post #65

I really enjoyed this article. I think the author is precisely right and I've been saying this for a long time. There's a ton of extremely interesting low hanging fruit that can vastly improve the effectiveness of even currently existing models hiding in how we design our agent harnesses; enough to — at least until we hit diminishing returns — make as much or more of a difference than training new models! I think one…

Also, yes, I'm aware that I use a lot of "its not just X, its Y." I promise you this comment is entirely human written. I'm just really tired and tend to rely on more wrote rhetorical tropes when I am. Believe me, I wrote like this long before LLMs were a thing.

No one here will accuse you of being an AI unless they're trying to dehumanize you for expressing anti-AI sentiment.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#157

Post by uriegas »

I do agree with his identification of the problem: sometimes agents fail because of the tools around it and not because of the model's reasoning. However, for the failing tests I think he is not making the distinction between a failed test due to a harness failure or due to a reasoning failure. It would be nice if someone analyzed that from the data set.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#158

Post by noupdates »

logicprog wrote: post #65

I really enjoyed this article. I think the author is precisely right and I've been saying this for a long time. There's a ton of extremely interesting low hanging fruit that can vastly improve the effectiveness of even currently existing models hiding in how we design our agent harnesses; enough to — at least until we hit diminishing returns — make as much or more of a difference than training new models! I think one…

I was just looking at the SWE-bench docs and it seems like they use almost an arbitrary form of context engineering (loading in some arbitrary amount of files to saturate context). So in a way, the bench suites test how good a model is with little to no context engineering (I know ... it doesn't need to be said). We may not actually know which models are sensitive to good context-engineering, we're simply assuming all models are. I absolutely agree with you on one thing, there is definitely a ton of low hanging fruit.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#159

Post by 0x457 »

znnajdla wrote: post #138
parhamn wrote: post #76

On first principles it would seem that the "harness" is a myth. Surely a model like Opus 4.6/Codex 5.3 which can reason about complex functions and data flows across many files would trip up over top level function signatures it needs to call? I see a lot of evidence to the contrary though. Anyone know what the underlying issue here is?

How hard is it to for you to assemble a piece of IKEA furniture without an allen wrench, screwdriver, and clear instructions, vs with those 3?

Well, I assembled Alex once without instruction and with impact driver and hammer last year. Hardest part was to make tools fit.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#160

Post by keybored »

cruffle_duffle wrote: post #64
znnajdla wrote: post #40

My experience as well. People worry our profession is being reduced to "prompt engineer", but actually I get the feeling that programming will soon be mainly about designing and building harnesses for specific tasks.

The more I dive into this space the more I think that developers will still be in heavy demand—just operating in a different level of abstraction most of the time. We will need to know our CS fundamentals, experience will still matter, juniors will still be needed. It’s just that a lot of time time the actual code being generated will come from our little helper buddies. But those things still need a human in the sea…

> The more I dive into this space the more I think that developers will still be in heavy demand—just operating in a different level of abstraction most of the time. We will need to know our CS fundamentals, experience will still matter, juniors will still be needed. It’s just that a lot of time time the actual code being generated will come from our little helper buddies. But those things still need a human in the seat to drive them.

It’s disheartening that programmers are using this advanced, cutting-edge technology with such a backwards, old-fashioned approach.[1]

Code generation isn’t a higher level abstraction. It’s the same level but with automation.

See [1]. I’m open to LLMs or humans+LLMs creating new abstractions. Real abstractions that hide implementation details and don’t “leak”. Why isn’t this happening?

Truly “vibe coding” might also get the same job done. In the sense of: you only have to look at the generated code for reasons like how a C++ programmer looks at the assembly. Not to check if it is even correct. But because there are concerns beyond just the correctness like code gen size. (Do you care about compiler output size? Sometimes. So sometimes you have to look.)

[1]: https://news.ycombinator.com/item?id=44163821

Post reply on HN