Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: OSS Agent I built topped the TerminalBench on Gemini-3-flash-preview

github.com

Post reply on HN
Print view View original

21–30 of 167 posts

Re: Show HN: OSS Agent I built topped the TerminalBench on Gemini-3-flash-preview

#21

Post by howdareme »

GodelNumbering wrote: post #13
Mashimo wrote: post #10

Interesting. Would love a comparison to pi.dev (Not Ohmypi) How does this perform in day to day coding tasks, outside of benchmarks?

https://github.com/dirac-run/dirac#-evals README has eval of 8 tasks over 7 agents (including both pi and omp). Pi-mono costs second lowest across the 8 tasks (after Dirac) but occasionally misses produces incomplete changes. Interestingly, 2 tasks where pi missed some changes both were the tasks that benefitted from AST symbol understanding (e.g. find all instances of things that refer to this symbol and change thos…

Going to assume you didnt capture the data but could you add time taken to completion for each if you have it?

Re: Show HN: OSS Agent I built topped the TerminalBench on Gemini-3-flash-preview

#22

Post by martinald »

GodelNumbering wrote: post #11
martinald wrote: post #9

Very interesting! I've often thought static analysis could really help agents (I wrote this last summer: https://martinalderson.com/posts/claude-code-static-analysis... ), but despite being hyped for LSPs in Claude Code it turned out to be very underwhelming (for many of the reasons that they can be annoying in a "real" IDE, ie static analysis starts firing mid edit and complaining and cached analysis getting stuck).…

Wrt LSP, it uses the default LSP mechanism of the ide provider. For AST, it uses tree-sitter WASMs (ships them with the package), and maintains queries ( https://github.com/dirac-run/dirac/tree/master/src/services/... ) To keep performance fast, it stores the symbols DB (using sqlite) in the workspace's directory and incrementally updates it based on timestamps. Then it uses this DB to resolve symbol queries

Yes I understand, but do you not have issues that it drifts out of date and confuses the agents (especially on longer running tasks)?

Like even "full" Visual Studio and Resharper have issues with this. Eg, you start editing file x, 'intellisense' runs, says there are loads of errors... because you haven't finished editing yet.

Re: Show HN: OSS Agent I built topped the TerminalBench on Gemini-3-flash-preview

#23

Post by blueTiger33 »

Stared it. will try it later. one question though, to make it simpler for me, in what tasks does this model shine, how do you improve the score? I already use some skills to cut down CC costs, like caveman, rtk cli and a few others. just want to understand

Re: Show HN: OSS Agent I built topped the TerminalBench on Gemini-3-flash-preview

#26

Post by adyavanapalli »

I haven't tried it, but I'm curious why you decided to implement a whole new harness over just writing extensions in pi. From whatever I've done with pi so far, the extension api is quite extensive. Hash anchored edits, for example, can definitely be implemented in pi. Anyhow, thank you for showing us your project and will be checking it out later. Cheers!

Re: Show HN: OSS Agent I built topped the TerminalBench on Gemini-3-flash-preview

#27

Post by GodelNumbering »

redrove wrote: post #25

I keep trying to use dirac-cli with codex and it won't work: Error: Codex API error: Codex API request failed: 400. Any ideas?

Assuming you logged in with OAuth, I am guessing you are trying to use gpt-5.5?

In my tests, it worked using gpt-5.4 for me and I assumed gpt-5.5 is not available to me because I am on the free plan

Do you have the subscription that allows 5.5? If so, I can look into what changed in API. Sorry I rarely use openAI so it is a bit of an untrodden path

Re: Show HN: OSS Agent I built topped the TerminalBench on Gemini-3-flash-preview

#28

Post by GodelNumbering »

blueTiger33 wrote: post #23

Stared it. will try it later. one question though, to make it simpler for me, in what tasks does this model shine, how do you improve the score? I already use some skills to cut down CC costs, like caveman, rtk cli and a few others. just want to understand

I did limited testing using Sonnet on CC vs Sonnet on Dirac. I could not confirm the costs however

Re: Show HN: OSS Agent I built topped the TerminalBench on Gemini-3-flash-preview

#30

Post by mdasen »

nthypes wrote: post #7

Can't OpenCode reach the same just developing this as a feature or plug-in? Like anchored edit?

Sure. Dirac is just a fork of the Cline harness and obviously OpenCode could take the same techniques and implement them. I don't know how difficult it would be to implement them in OpenCode, but given that Dirac and OpenCode are both open source, a future version of OpenCode could always be a re-branded Dirac (I'm sure there are ways to implement Dirac's techniques without having to completely replace OpenCode's underlying code base, but this illustrates that at the extreme, they could clearly just take Dirac in its entirety to get the same results).
Post reply on HN