Live data from Hacker News

Which tools do Claude, Codex and Cursor choose? We measured 17k runs to find out

armature.tech

61–70 of 175 posts

Re: Which tools do Claude, Codex and Cursor choose? We measured 17k runs to find out

#61
post #58

I keep telling people that we are living in the golden age of AI - like the first year or two of google. It is all down hill as these companies push for profit and lock-in.

Exactly why everyone needs to be hyper-focused on ensuring that the open-source ecosystem is healthy and that we don't let them shut that down.

how does one do that from their comfy chair?

Re: Which tools do Claude, Codex and Cursor choose? We measured 17k runs to find out

#62
post #60

So we build an LLM... that grabs new tokens based on stochastics... train it on all the programming teaching material and projects available on the internet... and then analyse the output... ...for the distribution of content of the source material? what? You learn nothing.

Because of weightings around other things you aren't going to get a 1:1 50% of input code used node so it uses node 50% of the time. There's enough randomness and other stolen content to (in theory) bias it towards weird outputs/choices

Re: Which tools do Claude, Codex and Cursor choose? We measured 17k runs to find out

#63
post #58

Earlier quoted context omitted.

Exactly why everyone needs to be hyper-focused on ensuring that the open-source ecosystem is healthy and that we don't let them shut that down.

how does one do that from their comfy chair?

I for one am running what I can on my aging 1080Ti(!!), namely a Qwen2.5 14B (4-bit quantized). It’s not great, and the only other card I have is a 3070Ti but I need that for gaming. :(

What a terrible card that 3070Ti is. Mad regrets buying it because I wanted to save $400 compared to a 3080Ti.

Re: Which tools do Claude, Codex and Cursor choose? We measured 17k runs to find out

#64
post #2

Hey! Disclaimer: I am a Co-Founder of Armature (YC P26) which sells growth services to dev tools. This study is part of our broader work on how to influence coding agents choices and get products picked. To understand how agents pick tools we measured close to 17k sessions on an environment where agents run exactly like in the real world, on various repositories, talking to different personas (vibe-coder, junior or s…

Hi. I appreciate that you need to make rent, but if your business is basically "we do growth hacking and SEO tricks on models and get them to use products that aren't actually best for the job", you are scum. You are perpetuating shitty practices that have hurt developers for years now. Part of the reason people use AI is because of how useless search is due to the previous generation doing the same kind of thing you…

First time?

Re: Which tools do Claude, Codex and Cursor choose? We measured 17k runs to find out

#66
post #58

Earlier quoted context omitted.

Exactly why everyone needs to be hyper-focused on ensuring that the open-source ecosystem is healthy and that we don't let them shut that down.

how does one do that from their comfy chair?

By sending some cash (and maybe some non-sensitive training data too) the way of labs creating open models? At work we have expensive Anthropic and OpenAI subscriptions but also have some in house workflows plugged into DeepSeek and GLM APIs.

Re: Which tools do Claude, Codex and Cursor choose? We measured 17k runs to find out

#67
post #58

Earlier quoted context omitted.

Exactly why everyone needs to be hyper-focused on ensuring that the open-source ecosystem is healthy and that we don't let them shut that down.

how does one do that from their comfy chair?

By not harassing projects with AI generated pull requests.

Re: Which tools do Claude, Codex and Cursor choose? We measured 17k runs to find out

#68
post #59
post #48

Earlier quoted context omitted.

Actually, yes. Probably this https://github.com/anthropics/claude-code/issues/88041#issue...

They love adding flags to fix issues that users have without telling their users about the flags. It very much feels like: As long as our staff can have a good user experience, we're happy. We don't care about anyone else.

You are attributing too much agency. They just vibe code the thing and hope for the best probably.

Re: Which tools do Claude, Codex and Cursor choose? We measured 17k runs to find out

#69
post #26
post #14

Earlier quoted context omitted.

Yeah sounds kind of like the equivalent of SEA for AI agents (AEA?) except that it’s sneakier since agents can act without you noticing.. anyway this is in the hands of the labs

Is SEA Search Engine Advertising? So AEA is Agent Engine Advertising?

Yep

Re: Which tools do Claude, Codex and Cursor choose? We measured 17k runs to find out

#70
post #53
post #25

I built this for my own company. Armature is on to something. You start by analyzing the choices agents would make for various use cases and then glean what, if anything, you might do to start tilting the agents in the direction of your own product and away from the competitor. Selling to agents is similar to selling to humans. You dump money into marketing to make sure agents find your solution around every corner f…

Well that's a horrifying thought. Thanks, I hate it.

Maybe agents won’t need to be sold to by CEOs jumping around on stage like pet monkeys. Could be an improvement.
Post reply on HN