Live data from Hacker News

Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

blog.can.ac

31–40 of 318 posts

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#31
post #13

What was the point of Claude code or Gemini banning the OP? Why would they care about how IDEs use the underlying API?

>What was the point of Claude code or Gemini banning the OP? Why would they care about how IDEs use the underlying API?

Underscores the importance of sovereign models you can run on the edge, finetune yourself, and run offline. At State of Utopia, we're working on it!

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#32
post #23
post #15

Earlier quoted context omitted.

I assume he was using Gemini the same way as he was Claude when I make the following statement. I don’t believe it’s exceptionally unique or new that companies will revoke access if you are using an unpublished API that the apps use. I don’t see anything wrong with it myself. If you want, pay for normal token use on the published APIs. There is no expectation that you can use APIs for an application, even if you are…

Indeed, that's why Anthropic, OpenAI and other LLM providers are known to adhere to published APIs to gather the world's data, obeying licensing and ROBOTS.txt. It's truly disgusting.

I was under the impression that they do obey robots.txt now? There are clearly a lot of dumb agents that don’t, but didn’t think it was the major AI labs.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#33
Yep this has been my experience with browser agents as well. One little change in the harness/agentic loop and the model suddenly becomes a whole lot smarter at navigating the web. I was also able to build a better browser agent than ‘claude —chrome’ in just a few afternoons just by tweaking the harness.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#34
post #20

Earlier quoted context omitted.

Also another place where having it change out from underneath you can drastically alter the quality of your work in unexpected ways. Like most things - assume the "20/100/200" dollar deals that are great now are going to go down the enshitification route very rapidly. Even if the "limits" on them stay generous, the product will start shifting to prioritize things the user doesn't want. Tool recommendations are my imm…

At this point subsidizing Chinese open-weights vendors by paying for them is just the right thing to do. Maybe they too might go closed-weights when they become SotA, but they're now pretty close and haven't done it.

I am wondering what kinds of harness are best for GLM, Deepseek, Qwen, Kimi.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#35
post #6

The harness matters far more than most people think. This post about the CORE benchmark where Opus’ score almost doubled when they switched to Claude Code from their own harness. https://x.com/sayashk/status/1996334941832089732

Which, IMHO, should be why we should be able to change them freely or make our own. Being locked into a specific harness because you pay 20 bucks per month vs. pay-per-use ... is kinda dumb.

The reason Anthropic is pushing on the closed harness is that they're not confident with their ability to win on model quality long term, so they're trying to build lock-in. They can capture some additional telemetry owning the harness as well, but given the amount of data the agent loop already transmits, that borders on unethical spyware (which might be part of the reason they're afraid to open source).

Ultimately the market is going to force them to open up and let people flex their subs.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#36
Great post. A few choice quotes:

> Often the model isn’t flaky at understanding the task. It’s flaky at expressing itself. You’re blaming the pilot for the landing gear.

> The model is the moat. The harness is the bridge. Burning bridges just means fewer people bother to cross. Treating harnesses as solved, or even inconsequential, is very short-sighted.

> The gap between “cool demo” and “reliable tool” isn’t model magic. It’s careful, rather boring, empirical engineering at the tool boundary.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#37
post #23

Earlier quoted context omitted.

Indeed, that's why Anthropic, OpenAI and other LLM providers are known to adhere to published APIs to gather the world's data, obeying licensing and ROBOTS.txt. It's truly disgusting.

I was under the impression that they do obey robots.txt now? There are clearly a lot of dumb agents that don’t, but didn’t think it was the major AI labs.

After 3 years of pirating and scraping the entire world by doing the above, I guess they have everything that they now need or want.

So then it's better to start obeying ROBOTS.txt as a ladder pull through a "nicely behaved" image advantage.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#38
post #17

Great article, recommend reading all of it. > Why bother, you ask? Opus may be a great model, but Claude Code to this day leaks raw JSONL from sub-agent outputs, wasting hundreds of thousands of tokens. I get to say, “fuck it, subagents output structured data now”. This is why I find the banning of using Claude subscriptions in other harnesses is so heinous. Their harness that they're forcing onto everyone has tons o…

I mean they want to make money right? CC is a cool tool, but obviously they want you to use the api eventually if you’re even remotely a power user, 200/month for all you can eat tokens (well, until some arbitrary limit of the day kicks in) just doesn’t make sense when compared to api prices. In other words, CC should be seen as a software subscription.

The token limit is the same whether used in CC or in other harnesses.
Post reply on HN