Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: OSS Agent I built topped the TerminalBench on Gemini-3-flash-preview

github.com

Post reply on HN
Print view View original

131–140 of 167 posts

Re: Show HN: OSS Agent I built topped the TerminalBench on Gemini-3-flash-preview

#131

Post by anandkrshnn »

all2 wrote: post #111
anandkrshnn wrote: post #88

Really impressive results. The point about the harness mattering more than the model is spot on — we've seen similar patterns in our own work. One thing that stood out to me is your use of hash-anchored edits + AST-based context selection. We're building something in a similar direction with the Sovereign AI Stack, but with a stronger focus on governance and verification. Curious — did you run into issues with contex…

I'd be curious to hear more about your work on the 'Sovereign AI Stack'. I'm also working on a project that prioritizes governance and verification and I'd love to compare notes.

[dead]

Re: Show HN: OSS Agent I built topped the TerminalBench on Gemini-3-flash-preview

#133

Post by faangguyindia »

GodelNumbering wrote: post #4

Interesting things Dirac does: 1. Uses an optimized version of Hash-Anchored edits for file editing ( https://dirac.run/posts/hash-anchors-myers-diff-single-token ) 2. Utilizes language's AST to decide what to fetch into context, entirely avoids large code file reads 3. Batches all operations. Does large number of reads/edits simultaneously (you can see a video demo for deepseek-v4-flash here https://www.reddit.com/r…

Instead of burning tokens on SOTA models, why not use a dirt-cheap specialised model for file editing?

Where the SOTA model just makes a cheaper model to make edits, and it does so.

Re: Show HN: OSS Agent I built topped the TerminalBench on Gemini-3-flash-preview

#134

Post by gertlabs »

This is fascinating, and aligned with my experience working on harnesses. I bet there is still significant upside left on the table with that same model.

There was a narrative last year by Anthropic that each new model release had them making the harness closer to a simple while loop with tools, but now it seems to be going in the other direction. There's just so much to explore with harnesses. Rolling context windows (instead of compaction) have been very powerful in my work with agentic harnesses, while keeping a persistent high level summary and a detailed automated feedback pipeline (granted, this is easier said than done if you don't have specific, consistent goals for your agent like we do).

Re: Show HN: OSS Agent I built topped the TerminalBench on Gemini-3-flash-preview

#135

Post by Clueed »

deaux wrote: post #55

1. Would be good to benchmark at least one other model from a different family to see if it indeed generalizes. Minimax 2.7 seems a good candidate to keep it affordable. Until then we can't really tell if it's just overfit on Gemini 3 Flash. 2. Until then your landing page needs to mention all the numbers are just from running on Gemini 3 Flash. Currently there's no mention at all of Gemini. 3. Assuming that cheaper…

I tried it with minimax 2.7 and it really didn’t like the editing tool; collapsing rather quickly to using sed to edit files.

I guess it makes sense that models don’t generalize perfectly to arbitrary tools but are biased to those in its training data, especially for a common operation like editing files.

The Gemini family might be a good pick here since it generally underperforms in agentic tasks (due to lack of training data or other reasons) and thus might not have this inherent bias towards specific tools.

Re: Show HN: OSS Agent I built topped the TerminalBench on Gemini-3-flash-preview

#137

Post by alfiedotwtf »

mdasen wrote: post #33

It's really interesting how much the AI harness seems to matter. Going from 48% via Google's official results to 65% is a huge jump. I feel like I'm constantly seeing results that compare models and rarely seeing results that compare harnesses. Is there a leaderboard out there comparing harness results using the same models?

For my local tests the past few months on the same local model, I’ve found Claude Code to be way better than OpenCode, and OpenCode to be better than Codex.

Re: Show HN: OSS Agent I built topped the TerminalBench on Gemini-3-flash-preview

#138

Post by gbalduzzi »

faangguyindia wrote: post #133
GodelNumbering wrote: post #4

Interesting things Dirac does: 1. Uses an optimized version of Hash-Anchored edits for file editing ( https://dirac.run/posts/hash-anchors-myers-diff-single-token ) 2. Utilizes language's AST to decide what to fetch into context, entirely avoids large code file reads 3. Batches all operations. Does large number of reads/edits simultaneously (you can see a video demo for deepseek-v4-flash here https://www.reddit.com/r…

Instead of burning tokens on SOTA models, why not use a dirt-cheap specialised model for file editing? Where the SOTA model just makes a cheaper model to make edits, and it does so.

Yeah I also believe that there are plenty of efficiency gains available by using different models for different tasks. Reasoning models such as opus should only be used for the main planning and decision flows, but sub operations (exploring, applying edits etc etc) could be delegated to smaller and cheaper models. You also end up with a much smaller context for the main big model

Re: Show HN: OSS Agent I built topped the TerminalBench on Gemini-3-flash-preview

#139

Post by [deleted] »

satvikpendem wrote: post #116
adyavanapalli wrote: post #26

I haven't tried it, but I'm curious why you decided to implement a whole new harness over just writing extensions in pi. From whatever I've done with pi so far, the extension api is quite extensive. Hash anchored edits, for example, can definitely be implemented in pi. Anyhow, thank you for showing us your project and will be checking it out later. Cheers!

I've been looking into local LLMs and new harnesses recently, how good is Pi compared to OpenCode, I'm seeing that it's a lot better? What are the best models and customizations for it to fully utilize it?

[deleted]

Re: Show HN: OSS Agent I built topped the TerminalBench on Gemini-3-flash-preview

#140

Post by vintagedave »

lukeundtrug wrote: post #105
deskamess wrote: post #12

Earlier quoted context omitted.

I always wondered why AST's were not more of a part in both editing and scoping of changes/parsing code. I thought I read an article where they said 'grep' was just as effective. It kinda made sense for the case they were talking about.

Happened to have written both a tool and a blog post about the topic. It’s more about the different technical approaches you have in solving the problem but it might still interest you :) https://www.context-master.dev/blog/deterministic-semantic-c... Let me know, what you think

This is interesting - I have been working on the same thing, building contextual data, LSP-style.

I saw the tools page where if I understand right, `get-symbol-context` is actually the main useful tool for what you provide? The others seem more metadata it's easy to get already (?) but that tool provides the extra info.

I had been working on exposing mine as more high-level, ie multiple APIs to query different kinds of metadata about symbols, types, etc. But I am still not sure of the best approach, where my thinking was about not overloading the AI with too many different tools. They accumulate quickly.

Post reply on HN