Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

blog.can.ac

Post reply on HN
Print view View original

51–60 of 318 posts

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#51

Post by disgruntledphd2 »

Aurornis wrote: post #44
horsawlarway wrote: post #14

Earlier quoted context omitted.

Also another place where having it change out from underneath you can drastically alter the quality of your work in unexpected ways. Like most things - assume the "20/100/200" dollar deals that are great now are going to go down the enshitification route very rapidly. Even if the "limits" on them stay generous, the product will start shifting to prioritize things the user doesn't want. Tool recommendations are my imm…

> Like most things - assume the "20/100/200" dollar deals that are great now are going to go down the enshitification route very rapidly. I don’t assume this at all. In fact, the opposite has been happening in my experience: I try multiple providers at the same time and the $20/month plans have only been getting better with the model improvements and changes. The current ChatGPT $20/month plan goes a very long way ev…

> Inference costs are only going to go down from here and models will only improve. I’ve been reading these warnings about the coming demise of AI plans for 1-2 years now, but the opposite keeps happening.

This time also crosses over with the frontier labs raising ever larger and larger rounds. If Anthropic IPO (which I honestly doubt), then we may get a better sense of actual prices in the market, as it's unlikely the markets will continue letting them spend more and more money each year without a return.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#52

Post by kachapopopow »

tosh wrote: post #30

Shows how much room for improvement there is on the harness level. Agents waste a lot of tokens on editing, sandboxes, passing info back and forth from tool calls and subagents. Love the pragmatic mix of content based addressing + line numbers. Beautiful.

you can actually go the other way and spend more tokens to solve more complex problems (multi-agent) by letting agents work with smaller problems

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#53

Post by brendanmc6 »

chrisweekly wrote: post #36

Great post. A few choice quotes: > Often the model isn’t flaky at understanding the task. It’s flaky at expressing itself. You’re blaming the pilot for the landing gear. > The model is the moat. The harness is the bridge. Burning bridges just means fewer people bother to cross. Treating harnesses as solved, or even inconsequential, is very short-sighted. > The gap between “cool demo” and “reliable tool” isn’t model m…

You’re absolutely right! This isn’t your average engineering advice— it’s like painting the reader a vivid tapestry of the author’s mind.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#54

Post by esafak »

brendanmc6 wrote: post #53
chrisweekly wrote: post #36

Great post. A few choice quotes: > Often the model isn’t flaky at understanding the task. It’s flaky at expressing itself. You’re blaming the pilot for the landing gear. > The model is the moat. The harness is the bridge. Burning bridges just means fewer people bother to cross. Treating harnesses as solved, or even inconsequential, is very short-sighted. > The gap between “cool demo” and “reliable tool” isn’t model m…

You’re absolutely right! This isn’t your average engineering advice— it’s like painting the reader a vivid tapestry of the author’s mind.

Please stop; I just can't any more! Yes, I'm absolutely right.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#55

Post by bri3d »

animan wrote: post #13

What was the point of Claude code or Gemini banning the OP? Why would they care about how IDEs use the underlying API?

When you buy a subscription plan, you’re buying use of the harness, not the underlying compute / tokens. Buying those on their own is way more expensive. This is probably because:

* Subscriptions are oversubscribed. They know how much an “average” Claude Code user actually consumes to perform common tasks and price accordingly. This is how almost all subscription products work.

* There is some speculation that there is cooperative optimization between the harness and backend (cache related etc).

* Subscriptions are subsidized to build market share; to some extent the harnesses are “loss leader” halo products which drive the sales of tokens, which are much more profitable.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#56

Post by evolly »

My experience exactly! I’ve recently become so tired of the Claude harness that I switched to OpenCode (which is extremely good compared to Claude). However, OpenCode is also tedious to change, and it inherits all the “good stuff,” like treating agents as Markdown files and all the dancing around with hooks/plugins/skills scattered all over the place. Getting stuck again and again, I’ve ultimately come to the conclusion that this must be solved by writing my own damn coding agent, with extensibility that’s acceptable for real-world engineering.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#60

Post by HumanOstrich »

evolly wrote: post #56

My experience exactly! I’ve recently become so tired of the Claude harness that I switched to OpenCode (which is extremely good compared to Claude). However, OpenCode is also tedious to change, and it inherits all the “good stuff,” like treating agents as Markdown files and all the dancing around with hooks/plugins/skills scattered all over the place. Getting stuck again and again, I’ve ultimately come to the conclus…

Give Pi[1] a try. Comes pretty barebones out of the box, yet still provides a decent default experience. Extension points are all TypeScript if you want. There are a lot of examples[2] and some 3rd party extensions[3].

I'll point out that if you want permission prompts for certain behavior, you have to add that yourself. There's at least one example.

Edit: Just noticed the article's author is using a fork of Pi.

[1]: https://shittycodingagent.ai/

[2]: https://github.com/badlogic/pi-mono/tree/main/packages/codin...

[3]: https://github.com/nicobailon

Post reply on HN