Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Which tools do Claude, Codex and Cursor choose? We measured 17k runs to find out

armature.tech

Post reply on HN
Print view View original

171–180 of 183 posts

Re: Which tools do Claude, Codex and Cursor choose? We measured 17k runs to find out

#171

Post by Neywiny »

reneeh wrote: post #169
Neywiny wrote: post #168

Earlier quoted context omitted.

Because they're thinking like I did going into the article. Harnesses like Claude expose "tools" to the agent. I usually use Cline but I'm giving up on it for this exact reason. Cline tells the model "you tell me to write a file, I'll get it done" and then it messes everything up, causes tones of errors, and the model goes "wow that's a broken tool. I'm going to write a python script to write the file instead"

Cline just recently fully upgraded their harness, see here: https://x.com/cline/status/2095897914493243512?s=20 Try if you have a better experience now!

They said as of aug 23rd it's 100% rollout. I've noticed 0 improvement. If anything a downgrade. The agents often refuse to use the tools after 1 try because the results are so trash. An open ticket that shows my biggest gripe with the write tool https://github.com/cline/cline/issues/13276

I guess the mistake count is down though. But that's because it's bypassing all the tools and just running commands

Re: Which tools do Claude, Codex and Cursor choose? We measured 17k runs to find out

#172

Post by pixelatedindex »

pixelatedindex wrote: post #63
Roland303 wrote: post #61

Earlier quoted context omitted.

how does one do that from their comfy chair?

I for one am running what I can on my aging 1080Ti(!!), namely a Qwen2.5 14B (4-bit quantized). It’s not great, and the only other card I have is a 3070Ti but I need that for gaming. :( What a terrible card that 3070Ti is. Mad regrets buying it because I wanted to save $400 compared to a 3080Ti.

I really don’t understand why this is being downvoted? Did I say something wrong / controversial? Or is it because I’m not using my resources the “optimal” way? Is it because my models aren’t open weight? I was only trying to contribute my experience :(

Re: Which tools do Claude, Codex and Cursor choose? We measured 17k runs to find out

#173

Post by lucas_t_a »

ttul wrote: post #25

I built this for my own company. Armature is on to something. You start by analyzing the choices agents would make for various use cases and then glean what, if anything, you might do to start tilting the agents in the direction of your own product and away from the competitor. Selling to agents is similar to selling to humans. You dump money into marketing to make sure agents find your solution around every corner f…

that's just making a good and complete product then

Re: Which tools do Claude, Codex and Cursor choose? We measured 17k runs to find out

#174

Post by kyleaa »

nullbio wrote: post #58
natnatenathan wrote: post #56

I keep telling people that we are living in the golden age of AI - like the first year or two of google. It is all down hill as these companies push for profit and lock-in.

Exactly why everyone needs to be hyper-focused on ensuring that the open-source ecosystem is healthy and that we don't let them shut that down.

Does Nvidia's acquisition put the open-source ecosystem at risk? I remember what Oracle purchased Java. IMO, it was a mess for many years after that. It is a strategic move for Nvidia, but I see it harming open-source AI options.

Re: Which tools do Claude, Codex and Cursor choose? We measured 17k runs to find out

#175

Post by distances »

sajithdilshan wrote: post #120
Pavilion2095 wrote: post #114

Earlier quoted context omitted.

In many countries Meta platforms—Facebook, Instagram, WhatsApps—are primary means of communication. If you want to find interesting events happening this weekend, you need to be on Facebook. In this regard, Google and Apple are much easier to leave because they offer personalised services, and you can always move your email and photos elsewhere. But you cannot move other people from Facebook or Instagram.

It’s not that easy to leave Google or Apple. As an example I wanted to leave iCloud and stop paying storage tax for Apple. But they have made it almost impossible to migrate the photos. There is no API to download the photos programmatically and the only way is to sync all the photos to MacBook and then move it. However, the photos are close to 1TB of storage and my MacBook doesn’t have enough storage to sync all tho…

> I have used my Gmail address for countless services and it’s almost part of my official identity.

This has been a topic for over a decade now, I think. People have been recommending getting your own domain. Best time to do that was ten years ago, and the second best is today.

Re: Which tools do Claude, Codex and Cursor choose? We measured 17k runs to find out

#176

Post by dgroshev »

elzbardico wrote: post #122
screm wrote: post #100

Earlier quoted context omitted.

We don't need to reproduce the same errors! Anyway I do think it is going to be different this time because generating content is becoming so easy today that the entire web would just become 99.99% slop very quickly if things don't change. That being said the solution isn't that trivial, curious if you have thoughts on it?

Youtube at least, is already 99% for new content, outside the channels you already subscribed. The amount of channels that are basically and AI animation with a voice-over of a curious wikipedia article is staggering, but those are the good ones. The worse part is a deluge of stupid boomer fanfic (The ones about ungrateful kids, ungrateful employers that fired someone who secretly was a load-bearing (ha!) element for…

You're talking to the CEO of the want-to-be-SEO company [1] that did that. It is very much content marketing for their services.

[1]: https://news.ycombinator.com/item?id=49557233

Re: Which tools do Claude, Codex and Cursor choose? We measured 17k runs to find out

#177

Post by tedmartynov »

xnorswap wrote: post #98

They didn't measure which programming language because we all know it's python, which it tries to use, every session, despite repeated memory files to not use python. Recently it's even taken to installing python to get jobs done.

[flagged]
Post reply on HN