Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Benchmarking coding agents on Databricks' multi-million line codebase

databricks.com

Post reply on HN
Print view View original

21–30 of 78 posts

Re: Benchmarking coding agents on Databricks' multi-million line codebase

#21

Post by N_Lens »

Sphax wrote: post #18
redmalang wrote: post #9

We have an internal proxy (that I've been meaning to open source for ages) that routes all llm usage at our company, which allows us to see data in realtime. Its been fascinating how rapidly Pi has been adopted. Moreover since its pretty hackable, we've been able to automatically aggregate context from pi sessions, which has resulted in Pi efficacy being higher as more people use it, putting in place a interesting vi…

> we've been able to automatically aggregate context from pi sessions what do you mean by this ? do you rewrite the context in your proxy ?

I imagine they're capturing all the tokens+context from pi sessions from every developer.

Re: Benchmarking coding agents on Databricks' multi-million line codebase

#22

Post by wwind123 »

ozgrakkurt wrote: post #12
zkmon wrote: post #8

> Databricks’ multi-million line codebase The combined size of codebases for the underlying opensource products (Apache Spark etc) might be around 1M lines, I think. Why does the orchestration/management layer, that is "databricks", exceed the sizes of the core products?

Old codebase, you always add code and never remove it. So it is expected to be like this. Deleting code is difficult and almost never makes sense afaik

On one hand, I understand that some old code is hard to delete because it's hard to detangle a lot of the legacy dependency. On the other hand, too much useless old code existing in the code base by itself could become a big maintenance burden for both humans and AI. In some cases at some point it might become more economical to just invest a bunch of resource to detangle the dependencies to be able to remove the old code.

Re: Benchmarking coding agents on Databricks' multi-million line codebase

#23

Post by [deleted] »

Sphax wrote: post #18
redmalang wrote: post #9

We have an internal proxy (that I've been meaning to open source for ages) that routes all llm usage at our company, which allows us to see data in realtime. Its been fascinating how rapidly Pi has been adopted. Moreover since its pretty hackable, we've been able to automatically aggregate context from pi sessions, which has resulted in Pi efficacy being higher as more people use it, putting in place a interesting vi…

> we've been able to automatically aggregate context from pi sessions what do you mean by this ? do you rewrite the context in your proxy ?

[deleted]

Re: Benchmarking coding agents on Databricks' multi-million line codebase

#24

Post by redmalang »

Sphax wrote: post #18
redmalang wrote: post #9

We have an internal proxy (that I've been meaning to open source for ages) that routes all llm usage at our company, which allows us to see data in realtime. Its been fascinating how rapidly Pi has been adopted. Moreover since its pretty hackable, we've been able to automatically aggregate context from pi sessions, which has resulted in Pi efficacy being higher as more people use it, putting in place a interesting vi…

> we've been able to automatically aggregate context from pi sessions what do you mean by this ? do you rewrite the context in your proxy ?

Yeah.

Re: Benchmarking coding agents on Databricks' multi-million line codebase

#25

Post by fmind-dev »

falaki wrote: post #3
falaki wrote: post #2

1) Many models are now competitive at the top tier, including open source. 2) GLM 5.2 in particular was a major step forward in open source coding agent performance, 3) Harnesses make a huge difference in cost-performance. 4) Cheaper per-token does not imply cheaper per-task.

Also they suggest every company should build their own benchmark and repeat these tests with new models instead of relying on the SWE bench.

It takes time and effort to build such benchmark. It works at Databricks scale, I'm not sure smaller companies are ready to invest on internal benchmarks.

But they are more vendor neutral, now they don't sell their own model. It's interesting from a benchmark point of view.

Re: Benchmarking coding agents on Databricks' multi-million line codebase

#26

Post by alexott »

zkmon wrote: post #8

> Databricks’ multi-million line codebase The combined size of codebases for the underlying opensource products (Apache Spark etc) might be around 1M lines, I think. Why does the orchestration/management layer, that is "databricks", exceed the sizes of the core products?

:facepalm: "orchestration/management layer" that just runs dozens of millions of VMs per day, across the three clouds, with things like model serving, AI gateway, OLTP (Lakebase/Neon), Unity Catalog, etc.

Forget Databricks == Apache Spark...

Re: Benchmarking coding agents on Databricks' multi-million line codebase

#27

Post by twalla »

est wrote: post #20
lukax wrote: post #13

Earlier quoted context omitted.

Yes, this is a known issue. A significant amount of Edit tool calls fails in Pi witg newer models. https://lucumr.pocoo.org/2026/7/4/better-models-worse-tools/

https://blog.can.ac/2026/02/12/the-harness-problem/ It's worse than I expected.

[dead]
Post reply on HN