This matches my experience exactly. I’ve been building an MCP server with 82 tools and spent weeks on infrastructure testing. Switching from a Docker-based Cloudflare Tunnel to a native tunnel process took my tool call success rate from ~50% to 100% — same model, same tools, same prompts. The harness isn’t just important, it’s often the dominant variable.
Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed
311–318 of 318 posts
Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed
#312Earlier quoted context omitted.
> semantic > grep def Once you get to a codebase beyond a certain size, that no longer works. I've for one found Serena https://github.com/oraios/serena , which you can install from right within Claude, to be a fairly fantastic code-interaction tool for LLM's. Both semantic search as well as editing. And with way less token churn.
This is definitely a cool finding. Have you investigated more on this topic? like, anything similar in concept that competes with Serena? if so, have you tested it/them? what are your thoughts?
I actually just enhanced my `codescan` project to exceed Serena in some ways
https://github.com/pmarreck/codescan
Essentially zero-install, no MCP, just tell your agent about its CLI, have Ollama running with a particular embeddings model and boom
now I just need to set up Github Actions (ugh) so people can actually download artifacts
Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed
#313Earlier quoted context omitted.
> semantic > grep def Once you get to a codebase beyond a certain size, that no longer works. I've for one found Serena https://github.com/oraios/serena , which you can install from right within Claude, to be a fairly fantastic code-interaction tool for LLM's. Both semantic search as well as editing. And with way less token churn.
This is an interesting one - thanks for sharing!
I actually just enhanced my `codescan` project to exceed Serena in some ways
https://github.com/pmarreck/codescan
Essentially zero-install, no MCP, just tell your agent about its CLI, have Ollama running with a particular embeddings model and boom
now I just need to set up Github Actions (ugh) so people can actually download artifacts
Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed
#314Arguably I would think that the last year was mainly inner harness improvement instead model improvement but I could be wrong, just feels like that to me
We can measure this by looking at the same harness applied to different models, e.g. the very plain Terminus: https://www.tbench.ai/leaderboard/terminal-bench/2.0?agents=... Models have improved dramatically even with the same harness
I mean that just the way it tackles task in the core is generated differently, like inner harness, through system prompt or deeper root. F.e. Instead of answering instantly it goes through a pre-defined steps which strategy should be done, split task, use thinking tokens, use tools etc.
Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed
#315[dead]
Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed
#316[dead]
Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed
#317We build MCP servers that wrap fund APIs. The biggest performance variable we’ve found isn’t the model, it’s how much domain context the harness provides before the model has to reason. Same model, generic prompt versus one loaded with our procedural docs - wider gap than switching between model generations. Which surprised me.
The post’s framing is right but undersells what the harness actually does in production. It’s your trust layer: what can the model touch, what can’t it, how cheaply do you recover when it gets something wrong. We spend something like 70% of engineering time on the recovery path, not the inference. Whether that ratio is right I’m not sure, but it’s where we’ve ended up.
On MCP overhead downthread: real, yes. In regulated environments you need the audit trail and the kill switch, and a tool boundary is how you get those. The unsolved part is keeping the protocol thin enough that you’re not burning tokens on ceremony.
Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed
#318Earlier quoted context omitted.
This is definitely a cool finding. Have you investigated more on this topic? like, anything similar in concept that competes with Serena? if so, have you tested it/them? what are your thoughts?
I actually just enhanced my `codescan` project to exceed Serena in some ways https://github.com/pmarreck/codescan Essentially zero-install, no MCP, just tell your agent about its CLI, have Ollama running with a particular embeddings model and boom now I just need to set up Github Actions (ugh) so people can actually download artifacts
@pmarreck, Serena developer here. We invite you to contribute to Serena in order to make it better. Serena is free & open-source, and it already robustly addresses the key issues preventing coding agents from being truly efficient even in complex software development projects (while being highly configurable).
We don't believe CLI is the way to go though, because advanced code intelligence simply cannot be spawned on the fly and thus benefits from a stateful process (such as a language server or an IDE instance).