Live data from Hacker News

How well do coding agents use your library?

stackbench.ai

21–30 of 62 posts

Re: How well do coding agents use your library?

#21
post #18

What do coding agents need my library for? Don't they know how to write their own code? Isn't that a coding agent's entire purpose in life? There must be conflicting definitions out there. What does "coding agent" mean in this context?

Why did my engineering team handle payments through Stripe instead of building a custom payment processor? Aren’t they supposed to be engineering things?

Re: How well do coding agents use your library?

#22

It's a neat idea. But if, as we're told, LLMs will get better and better, something like this, in theory, will be increasingly unnecessary. I feel like most of the problems with AI using a library is how we mix code and implementation. C, C++ got it right (even if by accident) with separating specification from implementation. Instead of lamenting the design trend of not maintaining this split, for my own code I wrot…

> It's a neat idea. But if, as we're told, LLMs will get better and better, something like this, in theory, will be increasingly unnecessary.

I don't think so. I think understanding the context of a project will always produce superior results. I think instead we'll just make it a lot easier to add to the training corpus the LLM pulls from.

Re: How well do coding agents use your library?

#23
post #18

What do coding agents need my library for? Don't they know how to write their own code? Isn't that a coding agent's entire purpose in life? There must be conflicting definitions out there. What does "coding agent" mean in this context?

Why did my engineering team handle payments through Stripe instead of building a custom payment processor? Aren’t they supposed to be engineering things?

Coding agents presumably don't know how to deal with non-coding things. Stripe's real value isn't in its technology, but it sorting out the complex human problems associated with payment processing. Sending a number over a network is not any great feat. Getting humans to agree that number has meaning is another matter.

Re: How well do coding agents use your library?

#24
In Elixir land, the Ash Framework created a package called usage_rules[0] as an experimental attempt to solve this problem a few months ago. The latest version of the Phoenix Framework (1.8) includes it in their `mix phx.new` generator and in their own hex packages[1]. Library owners would need to add their own usage rules, but it seems to help even for just the core packages Phoenix includes.

[0] https://hexdocs.pm/usage_rules/readme.html

[1] https://github.com/phoenixframework/phoenix/tree/main/usage-...

Re: How well do coding agents use your library?

#25
post #4

The skip-to-the-end answer: Context7 MCP is so good it seems like magic, even to many well-informed, highly capable hackers. Simply wildly good for libraries and SDKs. All it takes to start using it is to add the MCP provider to your agent config and save your arms, "Use Context7 for this". https://context7.com/

I'm confused a bit by this. For instance, Gemini was struggling to write proper Java code for using Firebase Admin SDK. It would write Java code using methods that only exist in the JavaScript SDK. And when I would correct it, it would give other options that also were only in the JavaScript SDK or were invalid. So I thought this is where context7 would be useful, but I'm confused what I'm looking at in the detail pa…

You're looking at a summary for chunks of code that are relevant to the given library. If you type what specifically you need documentation for and adjust output token count, it will give LLM relevant fragments.

It lets you emulate RAG.

Re: How well do coding agents use your library?

#26

If coding agents are the new entry point to your library, how sure are you that they’re using it well? I asked this question to about 50 library maintainers and dev tool builders, and the majority didn't really know. Existing code generation benchmarks focus mainly on self-contained code snippets and compare models not agents. Almost none focus on library-specific generation. So we built a simple app to test how well…

Why do we need to log in?

we send out an email when the tests are finished (takes about 30 mins)

Re: How well do coding agents use your library?

#28
post #25

Earlier quoted context omitted.

I'm confused a bit by this. For instance, Gemini was struggling to write proper Java code for using Firebase Admin SDK. It would write Java code using methods that only exist in the JavaScript SDK. And when I would correct it, it would give other options that also were only in the JavaScript SDK or were invalid. So I thought this is where context7 would be useful, but I'm confused what I'm looking at in the detail pa…

You're looking at a summary for chunks of code that are relevant to the given library. If you type what specifically you need documentation for and adjust output token count, it will give LLM relevant fragments. It lets you emulate RAG.

I think the main problem is that the source GP is using, https://github.com/firebase/firebase-admin-java, contained almost nothing that context7 extracted as "docs".

It looks like https://firebase.google.com/docs/ is being refreshed as I type this, I imagine that using that as a source and including "Java" in the topic filter might give more results (or maybe the https://github.com/firebase/firebase-docs has the same content).

Re: How well do coding agents use your library?

#30

If coding agents are the new entry point to your library, how sure are you that they’re using it well? I asked this question to about 50 library maintainers and dev tool builders, and the majority didn't really know. Existing code generation benchmarks focus mainly on self-contained code snippets and compare models not agents. Almost none focus on library-specific generation. So we built a simple app to test how well…

If making dev tooling is selling shovels to the miners, then this is like selling sheet metal to the shovel makers.

Yeah. Feels like a data mining operation for training data.

I could be wrong.

Post reply on HN