Live data from Hacker News

Benchmarking coding agents on Databricks' multi-million line codebase

databricks.com

71–78 of 78 posts

Re: Benchmarking coding agents on Databricks' multi-million line codebase

#71
post #61

Earlier quoted context omitted.

of course it is; these entire systems are about priming the LLM to perform X task; if it's a coding harness, giving it very little priming for a task is going to make it grasp at straws.

Can anybody share a tested system prompt they use for general coding tasks in pi?

clone opencode and ask your ai yo find it. i did that then just wrote my own primary open code agent.

Re: Benchmarking coding agents on Databricks' multi-million line codebase

#72
post #63
post #59

Earlier quoted context omitted.

Pi has a subscription flow for Claude Code. Are they goading people to get banned or did they figure out a way to work with the subscription?

It seems like they are actually using the subscription providers' respective cli tools and managing context for them. In which case I believe it is not against the ToS any more than invoking codex cli from a custom python script would be? I personally cannot find an openai-hosted clarification, just on hn and x.

Awesome - that's what I do in my own harness for agy. Hopefully someone yells if they get banned.

Re: Benchmarking coding agents on Databricks' multi-million line codebase

#73

Doesnt this prove that there is really no moat in proprietary models for coding usecases, or the gaps is narrowing ? Also, since GLM5.2 can be run equally on amd and nvda, I guess there is no hw moat either. Further, switching costs for users is minimal not only in agents but models too. So there is really no stickiness or user preference involved. For this usecase I think it is a race to the bottom for costs in a go…

Yes exactly.

Re: Benchmarking coding agents on Databricks' multi-million line codebase

#74

We have an internal proxy (that I've been meaning to open source for ages) that routes all llm usage at our company, which allows us to see data in realtime. Its been fascinating how rapidly Pi has been adopted. Moreover since its pretty hackable, we've been able to automatically aggregate context from pi sessions, which has resulted in Pi efficacy being higher as more people use it, putting in place a interesting vi…

I keep hesitating to use Pi because I primarily use Claude and I worry it will be much more expensive to pay for API usage rather than the bundled subscription usage. But I haven't actually evaluated the cost tradeoff. Should I get over this hesitance and dive into Pi?

There are ways to do subscription plan on Pi through extensions.

Re: Benchmarking coding agents on Databricks' multi-million line codebase

#75
post #48

How is Pi so efficient? You'd think agent harness made by model makers would perform better.

Claude code's system prompt is filled with irrelevant stuff about how CC works, so that the agent can help the user set it up. And there's no way to disable all the extra stuff, AFAIK.

There's https://github.com/skrabe/lobotomized-claude-code , which strips many of those, but I'm not sure if it is "legal" to use.

Re: Benchmarking coding agents on Databricks' multi-million line codebase

#77
post #61

Earlier quoted context omitted.

Can anybody share a tested system prompt they use for general coding tasks in pi?

clone opencode and ask your ai yo find it. i did that then just wrote my own primary open code agent.

A bit late, but I followed up on this useful tip and found a gist that breaks down the opencode methodology:

https://gist.github.com/rmk40/cde7a98c1c90614a27478216cc0155...

The gist led me to the opencode session / prompt control folder:

https://github.com/anomalyco/opencode/tree/dev/packages/open...

Re: Benchmarking coding agents on Databricks' multi-million line codebase

#78
post #4

I wish they'd do a follow-on post drilling into the impact of the programming language on cost-per-task, specifically looking at cost to complete tasks in mainstream strongly typed languages (eg. C#, TypeScript) vs dynamic languages (eg. Python, JavaScript). Does the additional verbosity of the language help or hurt cost per task?

for that you would need to compare the same task implemented in two different languages - C# and Python for example, no?

The advantage of their dataset is it's large enough that you have a statistical number of PR's, allowing you to treat the average cost of Python PR's vs the average cost of C# costs (or etc.) as statistically meaningful.
Post reply on HN