Seems like for a hobby project $1 or $2 per task would add up a bit, depending on how many tasks you need to do. I mean it makes sense for a software company
Benchmarking coding agents on Databricks' multi-million line codebase
31–40 of 78 posts
Re: Benchmarking coding agents on Databricks' multi-million line codebase
#32I wish they'd do a follow-on post drilling into the impact of the programming language on cost-per-task, specifically looking at cost to complete tasks in mainstream strongly typed languages (eg. C#, TypeScript) vs dynamic languages (eg. Python, JavaScript). Does the additional verbosity of the language help or hurt cost per task?
for that you would need to compare the same task implemented in two different languages - C# and Python for example, no?
Re: Benchmarking coding agents on Databricks' multi-million line codebase
#33Is there any technical analysis of why contex grows slower in Pi compared to codex and CC?
different system prompts, codex will have system prompt telling the model to gather a lot more context before starting work
Re: Benchmarking coding agents on Databricks' multi-million line codebase
#34Would be great to see time spent per task per model. Especially since article references 390+ tokens per second for GLM5.2.
Re: Benchmarking coding agents on Databricks' multi-million line codebase
#35> the results showed clear clustering of the models and harnesses into 3 capability tiers
pretty sure the only thing making that 'clear' is the coloured stripes, if you took that away it'd look like two tiers
good result for GLM 5.2 though
and Sonnet 5 seems like a waste of time
Re: Benchmarking coding agents on Databricks' multi-million line codebase
#36> as we aggressively adopt AI for engineering
Why do we need to aggressively adopt things rather than thoughtfully adopt things?
It sounds like they are probably punching AI and engineers in the process
Re: Benchmarking coding agents on Databricks' multi-million line codebase
#37We have an internal proxy (that I've been meaning to open source for ages) that routes all llm usage at our company, which allows us to see data in realtime. Its been fascinating how rapidly Pi has been adopted. Moreover since its pretty hackable, we've been able to automatically aggregate context from pi sessions, which has resulted in Pi efficacy being higher as more people use it, putting in place a interesting vi…
Aside 2:
Anecdotally we found that Pi performs more or less on par with native harnesses at lower cost on decently specified prompts. It is also phenomenal at context cacheing especially on Deepseek models (its hard to precisely attribute credit here are my understanding is this is a DS speciality).
But it fails much worse on poorly drafted prompts. I'm generalising but native harnesses seem to be better kind of flailing along on those.
Re: Benchmarking coding agents on Databricks' multi-million line codebase
#38> as we aggressively adopt AI for engineering Why do we need to aggressively adopt things rather than thoughtfully adopt things? It sounds like they are probably punching AI and engineers in the process
What if other competing companies adopt it aggressively and push out features faster and get more market share?
What if pushing features faster brings more money because users like the features?
Re: Benchmarking coding agents on Databricks' multi-million line codebase
#39Earlier quoted context omitted.
Old codebase, you always add code and never remove it. So it is expected to be like this. Deleting code is difficult and almost never makes sense afaik
On one hand, I understand that some old code is hard to delete because it's hard to detangle a lot of the legacy dependency. On the other hand, too much useless old code existing in the code base by itself could become a big maintenance burden for both humans and AI. In some cases at some point it might become more economical to just invest a bunch of resource to detangle the dependencies to be able to remove the old…
At huge companies, it's hard to prioritize this because it's hard to pin dollar values to removing legacy code, while it's easier to show how building feature X will earn the company $Y amount of revenue. And because of that, there is also no incentive to do it, you don't get promoted by deleting old code, you get promoted by showing how your effort helped contribute to company revenue. At my previous company (100+ engineers, hundreds of microservices), teams that regularly clean legacy codebase tend to be platform teams (cost centers), while teams that struggle to get these prioritized are product vertical teams (revenue centers).
Re: Benchmarking coding agents on Databricks' multi-million line codebase
#40Earlier quoted context omitted.
> we've been able to automatically aggregate context from pi sessions what do you mean by this ? do you rewrite the context in your proxy ?
Yeah.
Can you say more? This sounds like it could be more useful than trying to build a knowledge base (say qmd) and then get the models to search it for the right context each time.