How well do coding agents use your library?
stackbench.ai
How well do coding agents use your library?
1–10 of 62 posts
Re: How well do coding agents use your library?
#2If coding agents are the new entry point to your library, how sure are you that they’re using it well?
I asked this question to about 50 library maintainers and dev tool builders, and the majority didn't really know.
Existing code generation benchmarks focus mainly on self-contained code snippets and compare models not agents. Almost none focus on library-specific generation.
So we built a simple app to test how well coding agents interact with libraries: • Takes your library’s docs • Automatically extracts usage examples • Tasks AI agents (like Claude Code) with generating those examples from scratch • Logs mistakes and analyzes performance
We’re testing libraries now, but it’s early days. If you're interested: Input your library, see what breaks, spot patterns, and share the results below.
We plan to expand to more coding agents, more library-specific tasks, and new metrics. Let us know what we should prioritize next.
Re: How well do coding agents use your library?
#3If coding agents are the new entry point to your library, how sure are you that they’re using it well? I asked this question to about 50 library maintainers and dev tool builders, and the majority didn't really know. Existing code generation benchmarks focus mainly on self-contained code snippets and compare models not agents. Almost none focus on library-specific generation. So we built a simple app to test how well…
Note that this comment is not hijacking. The author of this comment is also the author of the post.
Re: How well do coding agents use your library?
#4The skip-to-the-end answer: Context7 MCP is so good it seems like magic, even to many well-informed, highly capable hackers. Simply wildly good for libraries and SDKs. All it takes to start using it is to add the MCP provider to your agent config and save your arms, "Use Context7 for this".
Re: How well do coding agents use your library?
#5If coding agents are the new entry point to your library, how sure are you that they’re using it well? I asked this question to about 50 library maintainers and dev tool builders, and the majority didn't really know. Existing code generation benchmarks focus mainly on self-contained code snippets and compare models not agents. Almost none focus on library-specific generation. So we built a simple app to test how well…
If making dev tooling is selling shovels to the miners, then this is like selling sheet metal to the shovel makers.
Re: How well do coding agents use your library?
#6I've done a lot of work recently to make my library more "LLM Friendly", but I'm not willing at this time to sign up to a service which I don't know I'd ever use again just to run a test on your behalf. If you want to run the test on my library then its GitHub can be found here: https://github.com/KaliedaRik/Scrawl-canvas
Re: How well do coding agents use your library?
#7The skip-to-the-end answer: Context7 MCP is so good it seems like magic, even to many well-informed, highly capable hackers. Simply wildly good for libraries and SDKs. All it takes to start using it is to add the MCP provider to your agent config and save your arms, "Use Context7 for this". https://context7.com/
What is the best approach to have something like context7 for internal tools and libraries?
Re: How well do coding agents use your library?
#8If coding agents are the new entry point to your library, how sure are you that they’re using it well? I asked this question to about 50 library maintainers and dev tool builders, and the majority didn't really know. Existing code generation benchmarks focus mainly on self-contained code snippets and compare models not agents. Almost none focus on library-specific generation. So we built a simple app to test how well…
> If coding agents are the new entry point to your library, how sure are you that they’re using it well?
> I asked this question to about 50 library maintainers and dev tool builders, and the majority didn't really know.
Why should they even bother to answer such a loaded and hypothetical question?
Re: How well do coding agents use your library?
#9If coding agents are the new entry point to your library, how sure are you that they’re using it well? I asked this question to about 50 library maintainers and dev tool builders, and the majority didn't really know. Existing code generation benchmarks focus mainly on self-contained code snippets and compare models not agents. Almost none focus on library-specific generation. So we built a simple app to test how well…
Why do we need to log in?
Re: How well do coding agents use your library?
#10I'd use this if this was an open source tool.