Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Which tools do Claude, Codex and Cursor choose? We measured 17k runs to find out

armature.tech

Post reply on HN
Print view View original

51–60 of 175 posts

Re: Which tools do Claude, Codex and Cursor choose? We measured 17k runs to find out

#51

Post by pjm331 »

rcfox wrote: post #42
IgorPartola wrote: post #39

For some reason Claude Code keeps using awk, sed, and even Python to do basic file editing. Anyone know why that changed with the 5 series?

I've noticed that too. Maybe the normal Write tool has to output the entire file and this is an attempt to reduce token usage?

The Edit tool has been notoriously tricky to get right - it seems they have maybe branched out but I think morphllm started specifically with the pitch that they trained a small model to be good at editing files - most of their testimonials are about that

But I think it’s mostly a solved problem in frontier models and the bash tool usage is more likely an attempt to be more token efficient - I’ve noticed it used for making mechanical bulk edits that would be numerous “edit” tool uses otherwise

Re: Which tools do Claude, Codex and Cursor choose? We measured 17k runs to find out

#53

Post by hexapus »

ttul wrote: post #25

I built this for my own company. Armature is on to something. You start by analyzing the choices agents would make for various use cases and then glean what, if anything, you might do to start tilting the agents in the direction of your own product and away from the competitor. Selling to agents is similar to selling to humans. You dump money into marketing to make sure agents find your solution around every corner f…

Well that's a horrifying thought. Thanks, I hate it.

Re: Which tools do Claude, Codex and Cursor choose? We measured 17k runs to find out

#54

Post by appplication »

tiffanyh wrote: post #50
ttul wrote: post #25

I built this for my own company. Armature is on to something. You start by analyzing the choices agents would make for various use cases and then glean what, if anything, you might do to start tilting the agents in the direction of your own product and away from the competitor. Selling to agents is similar to selling to humans. You dump money into marketing to make sure agents find your solution around every corner f…

Doesn’t this ignore that the future of ads will probably just be some type of affiliate revenue going back to the agent for any product they help recommend.

Maybe but that future isn’t now and there’s real money to be made today with the above strategy.

Re: Which tools do Claude, Codex and Cursor choose? We measured 17k runs to find out

#55

Post by arcanemachiner »

hbarka wrote: post #20
screm wrote: post #2

Hey! Disclaimer: I am a Co-Founder of Armature (YC P26) which sells growth services to dev tools. This study is part of our broader work on how to influence coding agents choices and get products picked. To understand how agents pick tools we measured close to 17k sessions on an environment where agents run exactly like in the real world, on various repositories, talking to different personas (vibe-coder, junior or s…

> Claude Code rarely searches the web while Codex almost always does I’m trying to understand why they are opposite. I think it is true, I find myself giving a secondary prompt to Claude to “research this” and only then will it fetch. Codex is bang on fetching already.

Must be system prompts and tool instructions guiding the agents differently.

Re: Which tools do Claude, Codex and Cursor choose? We measured 17k runs to find out

#58

Post by nullbio »

natnatenathan wrote: post #56

I keep telling people that we are living in the golden age of AI - like the first year or two of google. It is all down hill as these companies push for profit and lock-in.

Exactly why everyone needs to be hyper-focused on ensuring that the open-source ecosystem is healthy and that we don't let them shut that down.

Re: Which tools do Claude, Codex and Cursor choose? We measured 17k runs to find out

#59

Post by nullbio »

nijave wrote: post #48
IgorPartola wrote: post #39

For some reason Claude Code keeps using awk, sed, and even Python to do basic file editing. Anyone know why that changed with the 5 series?

Actually, yes. Probably this https://github.com/anthropics/claude-code/issues/88041#issue...

They love adding flags to fix issues that users have without telling their users about the flags. It very much feels like: As long as our staff can have a good user experience, we're happy. We don't care about anyone else.

Re: Which tools do Claude, Codex and Cursor choose? We measured 17k runs to find out

#60

Post by rf15 »

So we build an LLM...

that grabs new tokens based on stochastics...

train it on all the programming teaching material and projects available on the internet...

and then analyse the output...

...for the distribution of content of the source material?

what?

You learn nothing.

Post reply on HN