Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: OSS Agent I built topped the TerminalBench on Gemini-3-flash-preview

github.com

Post reply on HN
Print view View original

81–90 of 167 posts

Re: Show HN: OSS Agent I built topped the TerminalBench on Gemini-3-flash-preview

#81

Post by drakythe »

GodelNumbering wrote: post #4

Interesting things Dirac does: 1. Uses an optimized version of Hash-Anchored edits for file editing ( https://dirac.run/posts/hash-anchors-myers-diff-single-token ) 2. Utilizes language's AST to decide what to fetch into context, entirely avoids large code file reads 3. Batches all operations. Does large number of reads/edits simultaneously (you can see a video demo for deepseek-v4-flash here https://www.reddit.com/r…

How are the two token anchors chosen when the initial 1700 single token anchors run out? I'm assuming just a 2 word combination from the 1700.

Re: Show HN: OSS Agent I built topped the TerminalBench on Gemini-3-flash-preview

#83

Post by culi »

mdasen wrote: post #33

It's really interesting how much the AI harness seems to matter. Going from 48% via Google's official results to 65% is a huge jump. I feel like I'm constantly seeing results that compare models and rarely seeing results that compare harnesses. Is there a leaderboard out there comparing harness results using the same models?

Maybe the future isn't a human-like centralized intelligence but an octopus-like decentralized intelligence where more focus is placed on making the harness itself "smart"

Re: Show HN: OSS Agent I built topped the TerminalBench on Gemini-3-flash-preview

#84

Post by amunozo »

kha1n3vol3 wrote: post #44

I am using dirac with Kimi 2.6 for refactoring a rust codebase. I have a Clean Architecture design which is being reinforced. The scope of work is laid out in a Beads epic with sub-issues. The planning was done with gpt5.5, and gpt5.5 is checking the work is complete. I have found that dirac is more productive on large codebase refactoring than OpenCode which actually trashed the .rs file and had to revert the code.

For gpt-5.5, do you use Dirac, too?

Re: Show HN: OSS Agent I built topped the TerminalBench on Gemini-3-flash-preview

#85

Post by vova_hn2 »

himata4113 wrote: post #45
avereveard wrote: post #35

"astounding how much the harness matters" is the right read and it should be the lasting one. the model is rentable, the prompts are rentable, the benchmark numbers are mostly a function of the harness around them. swapping Gemini for Sonnet underneath the same harness has a smaller bench delta than swapping the harness around the model. the cheating-agents post you linked is the same observation through a different…

That's why ARC-AGI-3 doesn't allow the use of a harnesses. The model has to create the harness instead.

The model is not allowed to create a harness either, I think.

Re: Show HN: OSS Agent I built topped the TerminalBench on Gemini-3-flash-preview

#86

Post by dominotw »

culi wrote: post #83
mdasen wrote: post #33

It's really interesting how much the AI harness seems to matter. Going from 48% via Google's official results to 65% is a huge jump. I feel like I'm constantly seeing results that compare models and rarely seeing results that compare harnesses. Is there a leaderboard out there comparing harness results using the same models?

Maybe the future isn't a human-like centralized intelligence but an octopus-like decentralized intelligence where more focus is placed on making the harness itself "smart"

That would be counter to AI company goals. They want harness to be dumb and models to be smart so they can sell models.

Re: Show HN: OSS Agent I built topped the TerminalBench on Gemini-3-flash-preview

#87

Post by GodelNumbering »

martinald wrote: post #22
GodelNumbering wrote: post #11

Earlier quoted context omitted.

Wrt LSP, it uses the default LSP mechanism of the ide provider. For AST, it uses tree-sitter WASMs (ships them with the package), and maintains queries ( https://github.com/dirac-run/dirac/tree/master/src/services/... ) To keep performance fast, it stores the symbols DB (using sqlite) in the workspace's directory and incrementally updates it based on timestamps. Then it uses this DB to resolve symbol queries

Yes I understand, but do you not have issues that it drifts out of date and confuses the agents (especially on longer running tasks)? Like even "full" Visual Studio and Resharper have issues with this. Eg, you start editing file x, 'intellisense' runs, says there are loads of errors... because you haven't finished editing yet.

It does a before/after comparison. Fetch the LSP error state, apply all edits, fetch it again, diff

Re: Show HN: OSS Agent I built topped the TerminalBench on Gemini-3-flash-preview

#88

Post by anandkrshnn »

Really impressive results. The point about the harness mattering more than the model is spot on — we've seen similar patterns in our own work.

One thing that stood out to me is your use of hash-anchored edits + AST-based context selection. We're building something in a similar direction with the Sovereign AI Stack, but with a stronger focus on governance and verification.

Curious — did you run into issues with context drift when using AST queries on very large codebases? We found that combining it with incremental symbol DB updates helped a lot.

Congrats on the results!

Re: Show HN: OSS Agent I built topped the TerminalBench on Gemini-3-flash-preview

#89

Post by hedgehog »

messh wrote: post #64
GodelNumbering wrote: post #4

Interesting things Dirac does: 1. Uses an optimized version of Hash-Anchored edits for file editing ( https://dirac.run/posts/hash-anchors-myers-diff-single-token ) 2. Utilizes language's AST to decide what to fetch into context, entirely avoids large code file reads 3. Batches all operations. Does large number of reads/edits simultaneously (you can see a video demo for deepseek-v4-flash here https://www.reddit.com/r…

Anchor based editing requires injecting new anchors to the context, and dirac does so via a diff. So how is this more efficient (token-wise) than search and replace?? Even at a single token per hash. Also, code is read more than written so these just add up. I experimented once with stable anchors, albeit longer than a single token, and found it a downgrade. My conclusion is that the efficiency dirac sees comes mainl…

I'm not sure one way or another but I've been using a related tool called Tilth by another poster here. It doesn't do anchor-based editing, but it does do syntax-aware search and will e.g. report the line range for function definitions, provide file outlines with line numbers on a file name match, etc.

https://github.com/jahala/tilth

Re: Show HN: OSS Agent I built topped the TerminalBench on Gemini-3-flash-preview

#90

Post by grzracz »

himata4113 wrote: post #45
avereveard wrote: post #35

"astounding how much the harness matters" is the right read and it should be the lasting one. the model is rentable, the prompts are rentable, the benchmark numbers are mostly a function of the harness around them. swapping Gemini for Sonnet underneath the same harness has a smaller bench delta than swapping the harness around the model. the cheating-agents post you linked is the same observation through a different…

That's why ARC-AGI-3 doesn't allow the use of a harnesses. The model has to create the harness instead.

Seems completely backwards to me. This is like judging Formula 1 just by the raw power of the engine. The rest of the car has just as much engineering, if not more.
Post reply on HN