Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: OSS Agent I built topped the TerminalBench on Gemini-3-flash-preview

github.com

Post reply on HN
Print view View original

51–60 of 167 posts

Re: Show HN: OSS Agent I built topped the TerminalBench on Gemini-3-flash-preview

#51

Post by GodelNumbering »

UncleOxidant wrote: post #39
GodelNumbering wrote: post #4

Interesting things Dirac does: 1. Uses an optimized version of Hash-Anchored edits for file editing ( https://dirac.run/posts/hash-anchors-myers-diff-single-token ) 2. Utilizes language's AST to decide what to fetch into context, entirely avoids large code file reads 3. Batches all operations. Does large number of reads/edits simultaneously (you can see a video demo for deepseek-v4-flash here https://www.reddit.com/r…

> Utilizes language's AST to decide what to fetch into context, Does that mean that it's only going to work with certain langauges for which it has parsers available?

It uses tree-sitter wasms. Currently, 14 languages are available (https://github.com/dirac-run/dirac/tree/master/src/services/...)

The agent would work even without a language parser, just that the AST-based functionalities won't work

Re: Show HN: OSS Agent I built topped the TerminalBench on Gemini-3-flash-preview

#52

Post by redrove »

GodelNumbering wrote: post #27
redrove wrote: post #25

I keep trying to use dirac-cli with codex and it won't work: Error: Codex API error: Codex API request failed: 400. Any ideas?

Assuming you logged in with OAuth, I am guessing you are trying to use gpt-5.5? In my tests, it worked using gpt-5.4 for me and I assumed gpt-5.5 is not available to me because I am on the free plan Do you have the subscription that allows 5.5? If so, I can look into what changed in API. Sorry I rarely use openAI so it is a bit of an untrodden path

Yes I'm on ChatGPT Pro (OAuth) and I'm trying to use gpt-5.5-xhigh.

That was the issue, 5.4 works just fine.

Support for service: priority (GPT /fast mode) would also be cool!

Re: Show HN: OSS Agent I built topped the TerminalBench on Gemini-3-flash-preview

#53

Post by scoopdewoop »

The Hash-anchor edit guy! Sincerely great idea, I used it in my own toy harness to good effect. I just checked this out, never tried it before, and its great! Clearly a well-iterated design with good choices made.

It is so refreshing to see real FOSS and not a grift. Simple openrouter api key, and I'm going.

This is what I'm using from now on. You are doing the best work in this space.

Re: Show HN: OSS Agent I built topped the TerminalBench on Gemini-3-flash-preview

#54

Post by sally_glance »

blurbleblurble wrote: post #40
GodelNumbering wrote: post #4

Interesting things Dirac does: 1. Uses an optimized version of Hash-Anchored edits for file editing ( https://dirac.run/posts/hash-anchors-myers-diff-single-token ) 2. Utilizes language's AST to decide what to fetch into context, entirely avoids large code file reads 3. Batches all operations. Does large number of reads/edits simultaneously (you can see a video demo for deepseek-v4-flash here https://www.reddit.com/r…

Did you consider incorporating ast-grep or gritql? Congratulations, great work.

Can't speak for OP but I tried providing ast-grep in the execution context of an execute_bash tool, but even with pretty aggressive steering most models just don't seem to use it a lot. More expensive/SOTA models or higher reasoning increases the chances but lowers speed and raises cost. Maybe due to training bias for exploration tasks?

Re: Show HN: OSS Agent I built topped the TerminalBench on Gemini-3-flash-preview

#55

Post by deaux »

1. Would be good to benchmark at least one other model from a different family to see if it indeed generalizes. Minimax 2.7 seems a good candidate to keep it affordable. Until then we can't really tell if it's just overfit on Gemini 3 Flash.

2. Until then your landing page needs to mention all the numbers are just from running on Gemini 3 Flash. Currently there's no mention at all of Gemini.

3. Assuming that cheaper also means faster in this case where model is equal? If so, then why not add this to the benchmarks to highlight another advantage - time until completion of the tasks. If it's the opposite and it takes longer (seems unlikely), then it would be transparent to note this.

4. Would be good to note if it does or does not support skills, (nested) AGENTS.md, MCP and so on for people considering migrating.

Re: Show HN: OSS Agent I built topped the TerminalBench on Gemini-3-flash-preview

#56

Post by blurbleblurble »

sally_glance wrote: post #54
blurbleblurble wrote: post #40

Earlier quoted context omitted.

Did you consider incorporating ast-grep or gritql? Congratulations, great work.

Can't speak for OP but I tried providing ast-grep in the execution context of an execute_bash tool, but even with pretty aggressive steering most models just don't seem to use it a lot. More expensive/SOTA models or higher reasoning increases the chances but lowers speed and raises cost. Maybe due to training bias for exploration tasks?

Yes, I've tried this passive approach too and didn't dig much further after that. I thought maybe they'd figured out something more intentional in the prompting to enable these kinds of approaches.

Re: Show HN: OSS Agent I built topped the TerminalBench on Gemini-3-flash-preview

#57

Post by npodbielski »

Ha! I had an idea to do something like that myself over the weekend after trying Junie and Mistral to write some test for my personal project, that took literally hours! because Qwen 3.5 I am using locally can run 10k prompt for 10mins. Which should not be the case if agent would ask really simple questions like:

- what tool you need?

- what would be parameters for the tool

- what method you want to read?

instead of sending few kilobytes of build output and waiting for response. Oh well.. Good thing someone already did that!

Re: Show HN: OSS Agent I built topped the TerminalBench on Gemini-3-flash-preview

#58

Post by gobdovan »

Very interesting, especially the harness point, how much of performance is in the wrapper tools (when I almost run out of credits, I change my model to a smaller one and try to give it more structured prompts; very often gpt-5.4-mini with structure works better than gpt-5.4 with vibes)

This inspired me to start a "skill distillery" [0] where I take good agent workflow ideas and turning them into small, inspectable/installable skills.

The first one is dirac-workflow, based on Dirac's structural code workflow. It's not a Dirac clone tho, it has no runtime, persistent AST index, hash-anchor editing engine, or benchmark harness. Just a small AST helper and the workflow discipline as a portable skill.

I also dogfooded it on the Dirac repo itself and included a short report.

Would appreciate feedback from the original author, if the prompts and tools [1] are representative.

[0] https://github.com/ouatu-ro/skill-distillery

[1] https://github.com/ouatu-ro/skill-distillery/blob/main/skill...

Re: Show HN: OSS Agent I built topped the TerminalBench on Gemini-3-flash-preview

#59

Post by jbellis »

GodelNumbering wrote: post #4

Interesting things Dirac does: 1. Uses an optimized version of Hash-Anchored edits for file editing ( https://dirac.run/posts/hash-anchors-myers-diff-single-token ) 2. Utilizes language's AST to decide what to fetch into context, entirely avoids large code file reads 3. Batches all operations. Does large number of reads/edits simultaneously (you can see a video demo for deepseek-v4-flash here https://www.reddit.com/r…

> Batches all operations. Does large number of reads/edits simultaneously...

I wasn't sure what this meant, so I looked at the source. It seems to be referring to tool APIs being designed around taking multiple targets as a list parameter, instead of hoping the model makes appropriately parallel tool calls. (This matches my experience btw, models are reluctant to make a large number of parallel calls simultaneously, and this seems more pronounced with weaker models.)

Re: Show HN: OSS Agent I built topped the TerminalBench on Gemini-3-flash-preview

#60

Post by manx »

mdasen wrote: post #33

It's really interesting how much the AI harness seems to matter. Going from 48% via Google's official results to 65% is a huge jump. I feel like I'm constantly seeing results that compare models and rarely seeing results that compare harnesses. Is there a leaderboard out there comparing harness results using the same models?

We probably want to compare the cartesian product of model+harness.
Post reply on HN