Which tools do Claude, Codex and Cursor choose? We measured 17k runs to find out
1–10 of 175 posts
Re: Which tools do Claude, Codex and Cursor choose? We measured 17k runs to find out
#2Hey!
Disclaimer: I am a Co-Founder of Armature (YC P26) which sells growth services to dev tools. This study is part of our broader work on how to influence coding agents choices and get products picked.
To understand how agents pick tools we measured close to 17k sessions on an environment where agents run exactly like in the real world, on various repositories, talking to different personas (vibe-coder, junior or senior engineers) in different sizes of companies.
All the results are now public and we'd love to know what findings surprise you the most, here are a few we found interesting: - Claude Code rarely searches the web while Codex almost always does it and Cursor sits in the middle. - Coding agents disagree more frequently than they agree. - Some players (LangChain, Supabase, Netlify, Paypal, Adyen) are almost always mentioned in their categories but never chosen. - Modifying repository context can change the pick entirely.
If you feel like digging, all the traces are there and we probably missed interesting learnings so let us know what you find!
Re: Which tools do Claude, Codex and Cursor choose? We measured 17k runs to find out
#3Hey! Disclaimer: I am a Co-Founder of Armature (YC P26) which sells growth services to dev tools. This study is part of our broader work on how to influence coding agents choices and get products picked. To understand how agents pick tools we measured close to 17k sessions on an environment where agents run exactly like in the real world, on various repositories, talking to different personas (vibe-coder, junior or s…
Is there a way to force the usage of a tool for certain tasks? Example: alawys use cli "foobar" to retrieve weather starus.
By tool I mean mcp server, cli, etc.
Re: Which tools do Claude, Codex and Cursor choose? We measured 17k runs to find out
#4Looking at this, maybe in the future, the tools that AI prefers will become the mainstream. Even now, the tools that AI gives the highest priority to are the ones people already choose. There might be a concentration effect toward the tools that AI selects
Re: Which tools do Claude, Codex and Cursor choose? We measured 17k runs to find out
#5Hey! Disclaimer: I am a Co-Founder of Armature (YC P26) which sells growth services to dev tools. This study is part of our broader work on how to influence coding agents choices and get products picked. To understand how agents pick tools we measured close to 17k sessions on an environment where agents run exactly like in the real world, on various repositories, talking to different personas (vibe-coder, junior or s…
The data was cool. Then I tried to tap on one of the other tabs. “This content is easier to read while full screen!” - Ok I’m game. “Hey this is what makes armature special!” - I don’t care, I’m here to look at data, not onboard onto some random platform. It took ages to find the tiny “skip tour” button, hiding in black on black text. Then it gave me another popup, which I dismissed without reading. Then the full screen modal was visible but it was horizontally misaligned - the left edge was cut off and the right of my phone screen was all white. I closed the tab with great prejudice. (Safari on iOS if you wanna try reproducing it)
I’m sure you - or Claude - built something you’re proud of. But I left your website frustrated.
Re: Which tools do Claude, Codex and Cursor choose? We measured 17k runs to find out
#6Hey! Disclaimer: I am a Co-Founder of Armature (YC P26) which sells growth services to dev tools. This study is part of our broader work on how to influence coding agents choices and get products picked. To understand how agents pick tools we measured close to 17k sessions on an environment where agents run exactly like in the real world, on various repositories, talking to different personas (vibe-coder, junior or s…
Is there a way to force the usage of a tool for certain tasks? Example: alawys use cli "foobar" to retrieve weather starus. By tool I mean mcp server, cli, etc.
Not sure I got your question right but if you are wondering for your own coding agent then I guess the answer would be a skill?
Here what I meant by "how to influence coding agents choices and get products picked" is from a vendor PoV, making sure any developer x codebase in the world asking for a tool in your category gets your tool recommended and implemented by the coding agent.
Re: Which tools do Claude, Codex and Cursor choose? We measured 17k runs to find out
#7Hey! Disclaimer: I am a Co-Founder of Armature (YC P26) which sells growth services to dev tools. This study is part of our broader work on how to influence coding agents choices and get products picked. To understand how agents pick tools we measured close to 17k sessions on an environment where agents run exactly like in the real world, on various repositories, talking to different personas (vibe-coder, junior or s…
Is there a way to force the usage of a tool for certain tasks? Example: alawys use cli "foobar" to retrieve weather starus. By tool I mean mcp server, cli, etc.
Obviously you can ask the agent to use a specific tool, but the point of this article is about what they choose when the human on the other end has no opinion/taste/clue.
Re: Which tools do Claude, Codex and Cursor choose? We measured 17k runs to find out
#8Looking at this, maybe in the future, the tools that AI prefers will become the mainstream. Even now, the tools that AI gives the highest priority to are the ones people already choose. There might be a concentration effect toward the tools that AI selects
Definitely! But about concentration I'm not so sure, there are ways to counter this effect so in the end it will be a fight like SEO is today. What is certain though is that getting recommended by coding agents will be a top prio for all dev tools.
Re: Which tools do Claude, Codex and Cursor choose? We measured 17k runs to find out
#9Hey! Disclaimer: I am a Co-Founder of Armature (YC P26) which sells growth services to dev tools. This study is part of our broader work on how to influence coding agents choices and get products picked. To understand how agents pick tools we measured close to 17k sessions on an environment where agents run exactly like in the real world, on various repositories, talking to different personas (vibe-coder, junior or s…
The data was cool. Then I tried to tap on one of the other tabs. “This content is easier to read while full screen!” - Ok I’m game. “Hey this is what makes armature special!” - I don’t care, I’m here to look at data, not onboard onto some random platform. It took ages to find the tiny “skip tour” button, hiding in black on black text. Then it gave me another popup, which I dismissed without reading. Then the full scr…
Hey, thanks for the feedback, the leaderboards aren't displaying well on mobile indeed, we are currently shipping a fix that should help with that. Thanks anyway!
Re: Which tools do Claude, Codex and Cursor choose? We measured 17k runs to find out
#10Earlier quoted context omitted.
Is there a way to force the usage of a tool for certain tasks? Example: alawys use cli "foobar" to retrieve weather starus. By tool I mean mcp server, cli, etc.
Obviously you can ask the agent to use a specific tool, but the point of this article is about what they choose when the human on the other end has no opinion/taste/clue.
Exactly!