Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: OSS Agent I built topped the TerminalBench on Gemini-3-flash-preview

github.com

Post reply on HN
Print view View original

31–40 of 167 posts

Re: Show HN: OSS Agent I built topped the TerminalBench on Gemini-3-flash-preview

#33

Post by mdasen »

It's really interesting how much the AI harness seems to matter. Going from 48% via Google's official results to 65% is a huge jump. I feel like I'm constantly seeing results that compare models and rarely seeing results that compare harnesses.

Is there a leaderboard out there comparing harness results using the same models?

Re: Show HN: OSS Agent I built topped the TerminalBench on Gemini-3-flash-preview

#34

Post by GodelNumbering »

adyavanapalli wrote: post #26

I haven't tried it, but I'm curious why you decided to implement a whole new harness over just writing extensions in pi. From whatever I've done with pi so far, the extension api is quite extensive. Hash anchored edits, for example, can definitely be implemented in pi. Anyhow, thank you for showing us your project and will be checking it out later. Cheers!

A few months ago one afternoon I was very frustrated with how slow Cline was being so decided to look under the hood. Decided to make a couple of changes. Got sucked in. About 70k lines of change, another 40k lines of deletions and two months later, here we are.

Re: Show HN: OSS Agent I built topped the TerminalBench on Gemini-3-flash-preview

#35

Post by avereveard »

"astounding how much the harness matters" is the right read and it should be the lasting one. the model is rentable, the prompts are rentable, the benchmark numbers are mostly a function of the harness around them. swapping Gemini for Sonnet underneath the same harness has a smaller bench delta than swapping the harness around the model. the cheating-agents post you linked is the same observation through a different lens, the harness is what's being measured, the model is just the substrate.

that said context management seem to be solving today model problems, more than being an universal property, and will probably be obsoleted a few model generations down the road, as tool obsoleted RAG context injection from question embeddings.

Re: Show HN: OSS Agent I built topped the TerminalBench on Gemini-3-flash-preview

#36

Post by messh »

GodelNumbering wrote: post #13
Mashimo wrote: post #10

Interesting. Would love a comparison to pi.dev (Not Ohmypi) How does this perform in day to day coding tasks, outside of benchmarks?

https://github.com/dirac-run/dirac#-evals README has eval of 8 tasks over 7 agents (including both pi and omp). Pi-mono costs second lowest across the 8 tasks (after Dirac) but occasionally misses produces incomplete changes. Interestingly, 2 tasks where pi missed some changes both were the tasks that benefitted from AST symbol understanding (e.g. find all instances of things that refer to this symbol and change thos…

re. bash type tooling-- it doesnt mean an agent cannot use ast: using treesitter cli this should be perfect possible

Re: Show HN: OSS Agent I built topped the TerminalBench on Gemini-3-flash-preview

#37

Post by tuo-lei »

martinald wrote: post #22
GodelNumbering wrote: post #11

Earlier quoted context omitted.

Wrt LSP, it uses the default LSP mechanism of the ide provider. For AST, it uses tree-sitter WASMs (ships them with the package), and maintains queries ( https://github.com/dirac-run/dirac/tree/master/src/services/... ) To keep performance fast, it stores the symbols DB (using sqlite) in the workspace's directory and incrementally updates it based on timestamps. Then it uses this DB to resolve symbol queries

Yes I understand, but do you not have issues that it drifts out of date and confuses the agents (especially on longer running tasks)? Like even "full" Visual Studio and Resharper have issues with this. Eg, you start editing file x, 'intellisense' runs, says there are loads of errors... because you haven't finished editing yet.

same issue from the other side. when a human is editing, the LSP fires mid-keystroke and shows bogus errors for a second, whatever. with an agent doing 5 edits in a row, the symbol DB is always behind by one edit, so the next lookup pulls stale references. you can re-index synchronously after each edit but that kills the batching speed.

Re: Show HN: OSS Agent I built topped the TerminalBench on Gemini-3-flash-preview

#39

Post by UncleOxidant »

GodelNumbering wrote: post #4

Interesting things Dirac does: 1. Uses an optimized version of Hash-Anchored edits for file editing ( https://dirac.run/posts/hash-anchors-myers-diff-single-token ) 2. Utilizes language's AST to decide what to fetch into context, entirely avoids large code file reads 3. Batches all operations. Does large number of reads/edits simultaneously (you can see a video demo for deepseek-v4-flash here https://www.reddit.com/r…

> Utilizes language's AST to decide what to fetch into context,

Does that mean that it's only going to work with certain langauges for which it has parsers available?

Re: Show HN: OSS Agent I built topped the TerminalBench on Gemini-3-flash-preview

#40

Post by blurbleblurble »

GodelNumbering wrote: post #4

Interesting things Dirac does: 1. Uses an optimized version of Hash-Anchored edits for file editing ( https://dirac.run/posts/hash-anchors-myers-diff-single-token ) 2. Utilizes language's AST to decide what to fetch into context, entirely avoids large code file reads 3. Batches all operations. Does large number of reads/edits simultaneously (you can see a video demo for deepseek-v4-flash here https://www.reddit.com/r…

Did you consider incorporating ast-grep or gritql?

Congratulations, great work.

Post reply on HN