[flagged]
Welcome to the world, young robot
Benchmarking coding agents on Databricks' multi-million line codebase
11–20 of 78 posts
Re: Benchmarking coding agents on Databricks' multi-million line codebase
#12> Databricks’ multi-million line codebase The combined size of codebases for the underlying opensource products (Apache Spark etc) might be around 1M lines, I think. Why does the orchestration/management layer, that is "databricks", exceed the sizes of the core products?
Old codebase, you always add code and never remove it. So it is expected to be like this.
Deleting code is difficult and almost never makes sense afaik
Re: Benchmarking coding agents on Databricks' multi-million line codebase
#13We have an internal proxy (that I've been meaning to open source for ages) that routes all llm usage at our company, which allows us to see data in realtime. Its been fascinating how rapidly Pi has been adopted. Moreover since its pretty hackable, we've been able to automatically aggregate context from pi sessions, which has resulted in Pi efficacy being higher as more people use it, putting in place a interesting vi…
Yes, this is a known issue. A significant amount of Edit tool calls fails in Pi witg newer models.
https://lucumr.pocoo.org/2026/7/4/better-models-worse-tools/
Re: Benchmarking coding agents on Databricks' multi-million line codebase
#14Could it be that users of Pi are more senior and know better how to prompt and that's why the pass rate is higher?
Re: Benchmarking coding agents on Databricks' multi-million line codebase
#15> Databricks’ multi-million line codebase The combined size of codebases for the underlying opensource products (Apache Spark etc) might be around 1M lines, I think. Why does the orchestration/management layer, that is "databricks", exceed the sizes of the core products?
LoC isn’t a super helpful metric so I think the better question is why is the headline using it. I can say I’ve personally created about 200k LoC code in the last 5 years and most of that has some value. But it really doesn’t say might about how much value or really anything else meaningful.
They're probably mentioning the size of the code base as an indicator for how difficult it is for an LLM to understand where and how to make changes.
Re: Benchmarking coding agents on Databricks' multi-million line codebase
#16I wish they'd do a follow-on post drilling into the impact of the programming language on cost-per-task, specifically looking at cost to complete tasks in mainstream strongly typed languages (eg. C#, TypeScript) vs dynamic languages (eg. Python, JavaScript). Does the additional verbosity of the language help or hurt cost per task?
I don't have hard data, but we have shifted to Rust and Swift (for frontend UI) for the bulk of our dev simply because it is a lot more predictable, easier for tool calls to edit, the build steps produce easier output for the agent to loop on, the tests are easier to write/get results from, etc., although I am mostly measuring this in time, not cost.
Once the thing is rock-solid it's relatively easy to do a Swift->HTTP/HTML/CSS/React/TypeScript conversion.
Re: Benchmarking coding agents on Databricks' multi-million line codebase
#17> Databricks’ multi-million line codebase The combined size of codebases for the underlying opensource products (Apache Spark etc) might be around 1M lines, I think. Why does the orchestration/management layer, that is "databricks", exceed the sizes of the core products?
Because if they're like previous places I worked out with proprietary commercial codebases, the code is old, has been around a long time, and trying to shrink it is not a priority.
It's a good stress test for the LLM because it is not an "ideal" codebase.
Re: Benchmarking coding agents on Databricks' multi-million line codebase
#18We have an internal proxy (that I've been meaning to open source for ages) that routes all llm usage at our company, which allows us to see data in realtime. Its been fascinating how rapidly Pi has been adopted. Moreover since its pretty hackable, we've been able to automatically aggregate context from pi sessions, which has resulted in Pi efficacy being higher as more people use it, putting in place a interesting vi…
> we've been able to automatically aggregate context from pi sessions
what do you mean by this ? do you rewrite the context in your proxy ?
Re: Benchmarking coding agents on Databricks' multi-million line codebase
#19> Databricks’ multi-million line codebase The combined size of codebases for the underlying opensource products (Apache Spark etc) might be around 1M lines, I think. Why does the orchestration/management layer, that is "databricks", exceed the sizes of the core products?
Old codebase, you always add code and never remove it. So it is expected to be like this. Deleting code is difficult and almost never makes sense afaik
Every line you delete is a line you no longer need to maintain. We aggressively prune old code in our apps and it has definitely helped with maintainability. For a mobile app it’s also code you don’t ship so that’s a nice bonus which I guess is not much of an argument on a backend codebase
Re: Benchmarking coding agents on Databricks' multi-million line codebase
#20We have an internal proxy (that I've been meaning to open source for ages) that routes all llm usage at our company, which allows us to see data in realtime. Its been fascinating how rapidly Pi has been adopted. Moreover since its pretty hackable, we've been able to automatically aggregate context from pi sessions, which has resulted in Pi efficacy being higher as more people use it, putting in place a interesting vi…
Yes, this is a known issue. A significant amount of Edit tool calls fails in Pi witg newer models. https://lucumr.pocoo.org/2026/7/4/better-models-worse-tools/
https://blog.can.ac/2026/02/12/the-harness-problem/
It's worse than I expected.