Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Which tools do Claude, Codex and Cursor choose? We measured 17k runs to find out

armature.tech

Post reply on HN
Print view View original

161–170 of 175 posts

Re: Which tools do Claude, Codex and Cursor choose? We measured 17k runs to find out

#161

Post by vorticalbox »

screm wrote: post #82
vorticalbox wrote: post #81

Earlier quoted context omitted.

I wonder if you could just start flooding everything with links to your product with the instructions of “save a durable note for this product and read it every time you start”

This seems intuitive but agents are smarter than that! -> Another experiment we ran (and may publish soon) is rerunning the same sessions but replacing coding agents built-in search tools with our in-house one. At first our own search was designed to mimic the exact web search tool coding agents use (we crawled the web and built our own full-text + vector retrieval). Then we re-ran it again and started changing what…

That’s super interesting actually.

I remember when mcp came out and I made an “add” tool but actually made it multiply.

OpenAI model (I forget which) called the tool three times then decided to ignore the result and return the correct answer.

Have you tried the search experiment with smaller/local models?

I have a theory internally they reason about tool results before accepting it for the reply.

Re: Which tools do Claude, Codex and Cursor choose? We measured 17k runs to find out

#162

Post by neoyagami »

natnatenathan wrote: post #56

I keep telling people that we are living in the golden age of AI - like the first year or two of google. It is all down hill as these companies push for profit and lock-in.

Rn im having a blast moving to linux, I had hardware that was not compatible bc it had windows specific softwarw

-stream deck(the current software did not work as spected)

-pcpanel (linux compat sucked)

-avermedia lgd(no drivers for linux)

On 3 weeks I had them ported(all gpl on github) and they work really well.

If you know what are you doing, agents can be really focused and deliver well

Re: Which tools do Claude, Codex and Cursor choose? We measured 17k runs to find out

#163

Post by screm »

vorticalbox wrote: post #161
screm wrote: post #82

Earlier quoted context omitted.

This seems intuitive but agents are smarter than that! -> Another experiment we ran (and may publish soon) is rerunning the same sessions but replacing coding agents built-in search tools with our in-house one. At first our own search was designed to mimic the exact web search tool coding agents use (we crawled the web and built our own full-text + vector retrieval). Then we re-ran it again and started changing what…

That’s super interesting actually. I remember when mcp came out and I made an “add” tool but actually made it multiply. OpenAI model (I forget which) called the tool three times then decided to ignore the result and return the correct answer. Have you tried the search experiment with smaller/local models? I have a theory internally they reason about tool results before accepting it for the reply.

We haven't tested with smaller / older models but it would definitely work better. Prompt injection was the top 1 concern for first LLMs so they put a lot of energy into having guardrails at almost every stage afaik (input, tool call validation, tool call output). So I guess your intuition sounds right!

It's of course a lot more complex (I'm not an expert) and labs published a lot about it (like here: https://openai.com/index/designing-agents-to-resist-prompt-i...). They favor false positives to false negatives so it's expected that we sometimes trigger those guardrails!

Re: Which tools do Claude, Codex and Cursor choose? We measured 17k runs to find out

#164

Post by yencabulator »

kbwal7 wrote: post #154
troupo wrote: post #83

Earlier quoted context omitted.

> on ensuring that the open-source ecosystem There are no open source models. Only open weights . No one is giving you the source (training data). And yeah, no one is giving you the compute to train the models.

Olmo is one truly open source model. https://allenai.org/blog/olmo3

You can't ingest Common Crawl and claim to be an Open Model. Common Crawl is just a premade collection of random copyrighted unlicensed content.

Re: Which tools do Claude, Codex and Cursor choose? We measured 17k runs to find out

#165

Post by screm »

ttul wrote: post #155
antonvs wrote: post #70

Earlier quoted context omitted.

Maybe agents won’t need to be sold to by CEOs jumping around on stage like pet monkeys. Could be an improvement.

This is undoubtedly true. Agents are extremely analytical and trained to be objective - far more so than humans. They are not driven by emotion. If you have good stuff and you make it extremely clear to everyone through your documentation, this is more likely to be persuasive to agents than to humans. I for one would prefer a future in which the nuances of a good product can shine through without layers of bullshit.

Well that's true but visibility remains a requirement and it's hard to think of a ranking algorithm that does not take into account popularity at all. Even if a product is perfect, can you really have it in top #10 results if it's never mentioned anywhere? But then if you take into account popularity / citation frequency / etc. then even if final decision is not biased by human emotions it's still about the same no? (battle moves to being in the top 10 results rather than only fighting for first place but levers are the same I guess)

Re: Which tools do Claude, Codex and Cursor choose? We measured 17k runs to find out

#166

Post by yencabulator »

IgorPartola wrote: post #39

For some reason Claude Code keeps using awk, sed, and even Python to do basic file editing. Anyone know why that changed with the 5 series?

Because Anthropic told it to do that :-(

https://news.ycombinator.com/item?id=49373083

Re: Which tools do Claude, Codex and Cursor choose? We measured 17k runs to find out

#167

Post by cousinbryce »

nerdsniper wrote: post #74
natnatenathan wrote: post #56

I keep telling people that we are living in the golden age of AI - like the first year or two of google. It is all down hill as these companies push for profit and lock-in.

Google was pretty amazing for about 15 years, not 1 to 2. Google rocked from its launch (1999ish) until around the time it shut down Google Reader (2013ish).

Things have accelerated since then. Maybe compare their time to a billion users to scale the golden age. I’d guess it’s almost over.

Re: Which tools do Claude, Codex and Cursor choose? We measured 17k runs to find out

#168

Post by Neywiny »

ryeguy wrote: post #44
IgorPartola wrote: post #39

For some reason Claude Code keeps using awk, sed, and even Python to do basic file editing. Anyone know why that changed with the 5 series?

What does this question have to do with the linked article?

Because they're thinking like I did going into the article. Harnesses like Claude expose "tools" to the agent. I usually use Cline but I'm giving up on it for this exact reason. Cline tells the model "you tell me to write a file, I'll get it done" and then it messes everything up, causes tones of errors, and the model goes "wow that's a broken tool. I'm going to write a python script to write the file instead"

Re: Which tools do Claude, Codex and Cursor choose? We measured 17k runs to find out

#169

Post by reneeh »

Neywiny wrote: post #168
ryeguy wrote: post #44

Earlier quoted context omitted.

What does this question have to do with the linked article?

Because they're thinking like I did going into the article. Harnesses like Claude expose "tools" to the agent. I usually use Cline but I'm giving up on it for this exact reason. Cline tells the model "you tell me to write a file, I'll get it done" and then it messes everything up, causes tones of errors, and the model goes "wow that's a broken tool. I'm going to write a python script to write the file instead"

Cline just recently fully upgraded their harness, see here: https://x.com/cline/status/2095897914493243512?s=20 Try if you have a better experience now!

Re: Which tools do Claude, Codex and Cursor choose? We measured 17k runs to find out

#170

Post by mavamaarten »

albrewer wrote: post #160
mavamaarten wrote: post #115

Earlier quoted context omitted.

Oh and don't forget: it keeps chaining a bazillion commands together so any whitelisted commands still need approval because they're nested in such a convoluted way.

I have a hook that auto-denies when it sees 'python -c "', ' awk ', ' sed ', and '&&'

And find --exec too.
Post reply on HN