very interesting results!
GLM performed extremely well. we need GLM-6!
Benchmarking coding agents on Databricks' multi-million line codebase
51–60 of 78 posts
Re: Benchmarking coding agents on Databricks' multi-million line codebase
#52Earlier quoted context omitted.
Yes, this is a known issue. A significant amount of Edit tool calls fails in Pi witg newer models. https://lucumr.pocoo.org/2026/7/4/better-models-worse-tools/
https://blog.can.ac/2026/02/12/the-harness-problem/ It's worse than I expected.
I don't understand peoples expectation. If our language skills were as non-ambiguous as a coding language, we'd have solved world hunger by now.
So why would we expect all these bizarre bizantine language models to all conform to how a request is both made, expected and massaged.
For awhile, I was getting bizarre opencode tool errors where the only problem was the model was passing in a "1.0" or "0.0" where the harness dutifully wanted an integer. Of course 0.0 is the same as an integer in practical operations.
Re: Benchmarking coding agents on Databricks' multi-million line codebase
#53Earlier quoted context omitted.
Aside 2: Anecdotally we found that Pi performs more or less on par with native harnesses at lower cost on decently specified prompts. It is also phenomenal at context cacheing especially on Deepseek models (its hard to precisely attribute credit here are my understanding is this is a DS speciality). But it fails much worse on poorly drafted prompts. I'm generalising but native harnesses seem to be better kind of flai…
Is it because Pi’s default system prompt is so simple?
of course it is; these entire systems are about priming the LLM to perform X task; if it's a coding harness, giving it very little priming for a task is going to make it grasp at straws.
Re: Benchmarking coding agents on Databricks' multi-million line codebase
#54How is Pi so efficient? You'd think agent harness made by model makers would perform better.
they spent very little time validating what they're doing, and it works by not doing much of anything. If you spent a month figuring out a specific model+harness, you'd be way more efficient.
Other hanresses are doing overkill so they can work with any model.
Re: Benchmarking coding agents on Databricks' multi-million line codebase
#55Wow! It's great to see a large-scale real-world benchmark from a user of these tools, as opposed to the the benchmaxxed results from the vendors themselves. Also great to see different harnesses being tested, with considerably different results. Definitely a few surprises here: 1) GLM 5.2 using Pi performs identically in terms of pass rate (~87.5%) to Opus 4.8 high using Claude Code, but significantly cheaper ($1.25…
The catch-22 here is that you still save money using Claude Code directly here if you are on one of their subscription plan due to how heavily they are subsidizing that. Using Pi means you are using the API which is both a more accurate pricing model but also a more expensive one.
True, but OpenAI are OK with using their subscription plans with Pi, so GPT 5.6 with Pi may be a good combination.
There seems to be a lot of good buzz about GPT 5.6 on Twitter - people (incl. OpenCode team) preferring it to Fable 5.
Re: Benchmarking coding agents on Databricks' multi-million line codebase
#56Earlier quoted context omitted.
https://blog.can.ac/2026/02/12/the-harness-problem/ It's worse than I expected.
I don't understand peoples expectation. If our language skills were as non-ambiguous as a coding language, we'd have solved world hunger by now. So why would we expect all these bizarre bizantine language models to all conform to how a request is both made, expected and massaged. For awhile, I was getting bizarre opencode tool errors where the only problem was the model was passing in a "1.0" or "0.0" where the harne…
[dead]
Re: Benchmarking coding agents on Databricks' multi-million line codebase
#57It seems that pass rate decreases with effort increase, on GPT5.5? This is highly counter-intuitive and I don't see any explanation, any idea why they'd get this result?
Re: Benchmarking coding agents on Databricks' multi-million line codebase
#58How is Pi so efficient? You'd think agent harness made by model makers would perform better.
Different incentives. Claude Code makes more money for Anthropic by generating larger contexts. Anthropic also recently changed their tokenizer so the exact same code input creates 30% more tokens, so there's a pattern there.
Re: Benchmarking coding agents on Databricks' multi-million line codebase
#59Wow! It's great to see a large-scale real-world benchmark from a user of these tools, as opposed to the the benchmaxxed results from the vendors themselves. Also great to see different harnesses being tested, with considerably different results. Definitely a few surprises here: 1) GLM 5.2 using Pi performs identically in terms of pass rate (~87.5%) to Opus 4.8 high using Claude Code, but significantly cheaper ($1.25…
The catch-22 here is that you still save money using Claude Code directly here if you are on one of their subscription plan due to how heavily they are subsidizing that. Using Pi means you are using the API which is both a more accurate pricing model but also a more expensive one.
Pi has a subscription flow for Claude Code. Are they goading people to get banned or did they figure out a way to work with the subscription?
Re: Benchmarking coding agents on Databricks' multi-million line codebase
#60Earlier quoted context omitted.
https://blog.can.ac/2026/02/12/the-harness-problem/ It's worse than I expected.
I don't understand peoples expectation. If our language skills were as non-ambiguous as a coding language, we'd have solved world hunger by now. So why would we expect all these bizarre bizantine language models to all conform to how a request is both made, expected and massaged. For awhile, I was getting bizarre opencode tool errors where the only problem was the model was passing in a "1.0" or "0.0" where the harne…
I have one core complex task where there are a number of simple errors like this. The easiest thing for me was to just have a post-processing script that performs: lint > mark known fail-early results > fix common errors (all formatting or type errors right now) > relint > final pass / fail > pass to next step (retry with tweaks / user in the loop since I don't have the token budget to just blindly keep trying the same thing).