Earlier quoted context omitted.
of course it is; these entire systems are about priming the LLM to perform X task; if it's a coding harness, giving it very little priming for a task is going to make it grasp at straws.
Can anybody share a tested system prompt they use for general coding tasks in pi?
Benchmarking coding agents on Databricks' multi-million line codebase
71–78 of 78 posts
Re: Benchmarking coding agents on Databricks' multi-million line codebase
#72Earlier quoted context omitted.
Pi has a subscription flow for Claude Code. Are they goading people to get banned or did they figure out a way to work with the subscription?
It seems like they are actually using the subscription providers' respective cli tools and managing context for them. In which case I believe it is not against the ToS any more than invoking codex cli from a custom python script would be? I personally cannot find an openai-hosted clarification, just on hn and x.
Re: Benchmarking coding agents on Databricks' multi-million line codebase
#73Doesnt this prove that there is really no moat in proprietary models for coding usecases, or the gaps is narrowing ? Also, since GLM5.2 can be run equally on amd and nvda, I guess there is no hw moat either. Further, switching costs for users is minimal not only in agents but models too. So there is really no stickiness or user preference involved. For this usecase I think it is a race to the bottom for costs in a go…
Re: Benchmarking coding agents on Databricks' multi-million line codebase
#74We have an internal proxy (that I've been meaning to open source for ages) that routes all llm usage at our company, which allows us to see data in realtime. Its been fascinating how rapidly Pi has been adopted. Moreover since its pretty hackable, we've been able to automatically aggregate context from pi sessions, which has resulted in Pi efficacy being higher as more people use it, putting in place a interesting vi…
I keep hesitating to use Pi because I primarily use Claude and I worry it will be much more expensive to pay for API usage rather than the bundled subscription usage. But I haven't actually evaluated the cost tradeoff. Should I get over this hesitance and dive into Pi?
Re: Benchmarking coding agents on Databricks' multi-million line codebase
#75How is Pi so efficient? You'd think agent harness made by model makers would perform better.
There's https://github.com/skrabe/lobotomized-claude-code , which strips many of those, but I'm not sure if it is "legal" to use.
Re: Benchmarking coding agents on Databricks' multi-million line codebase
#76Re: Benchmarking coding agents on Databricks' multi-million line codebase
#77Earlier quoted context omitted.
Can anybody share a tested system prompt they use for general coding tasks in pi?
clone opencode and ask your ai yo find it. i did that then just wrote my own primary open code agent.
https://gist.github.com/rmk40/cde7a98c1c90614a27478216cc0155...
The gist led me to the opencode session / prompt control folder:
https://github.com/anomalyco/opencode/tree/dev/packages/open...
Re: Benchmarking coding agents on Databricks' multi-million line codebase
#78I wish they'd do a follow-on post drilling into the impact of the programming language on cost-per-task, specifically looking at cost to complete tasks in mainstream strongly typed languages (eg. C#, TypeScript) vs dynamic languages (eg. Python, JavaScript). Does the additional verbosity of the language help or hurt cost per task?
for that you would need to compare the same task implemented in two different languages - C# and Python for example, no?