Really impressive results. The point about the harness mattering more than the model is spot on — we've seen similar patterns in our own work. One thing that stood out to me is your use of hash-anchored edits + AST-based context selection. We're building something in a similar direction with the Sovereign AI Stack, but with a stronger focus on governance and verification. Curious — did you run into issues with contex…
Show HN: OSS Agent I built topped the TerminalBench on Gemini-3-flash-preview
111–120 of 167 posts
Re: Show HN: OSS Agent I built topped the TerminalBench on Gemini-3-flash-preview
#112How do I connect it to a local llama.cpp instance?
OPENAI_COMPATIBLE_CUSTOM_KEY="xxx" dirac -y --provider "https://localhost/v1" --model "hi..."
Re: Show HN: OSS Agent I built topped the TerminalBench on Gemini-3-flash-preview
#113It's really interesting how much the AI harness seems to matter. Going from 48% via Google's official results to 65% is a huge jump. I feel like I'm constantly seeing results that compare models and rarely seeing results that compare harnesses. Is there a leaderboard out there comparing harness results using the same models?
Re: Show HN: OSS Agent I built topped the TerminalBench on Gemini-3-flash-preview
#114Earlier quoted context omitted.
https://github.com/dirac-run/dirac#-evals README has eval of 8 tasks over 7 agents (including both pi and omp). Pi-mono costs second lowest across the 8 tasks (after Dirac) but occasionally misses produces incomplete changes. Interestingly, 2 tasks where pi missed some changes both were the tasks that benefitted from AST symbol understanding (e.g. find all instances of things that refer to this symbol and change thos…
re. bash type tooling-- it doesnt mean an agent cannot use ast: using treesitter cli this should be perfect possible
Re: Show HN: OSS Agent I built topped the TerminalBench on Gemini-3-flash-preview
#115Earlier quoted context omitted.
Maybe the future isn't a human-like centralized intelligence but an octopus-like decentralized intelligence where more focus is placed on making the harness itself "smart"
That would be counter to AI company goals. They want harness to be dumb and models to be smart so they can sell models.
Re: Show HN: OSS Agent I built topped the TerminalBench on Gemini-3-flash-preview
#116I haven't tried it, but I'm curious why you decided to implement a whole new harness over just writing extensions in pi. From whatever I've done with pi so far, the extension api is quite extensive. Hash anchored edits, for example, can definitely be implemented in pi. Anyhow, thank you for showing us your project and will be checking it out later. Cheers!
Re: Show HN: OSS Agent I built topped the TerminalBench on Gemini-3-flash-preview
#117Earlier quoted context omitted.
I always wondered why AST's were not more of a part in both editing and scoping of changes/parsing code. I thought I read an article where they said 'grep' was just as effective. It kinda made sense for the case they were talking about.
Has anybody thought about encoding AST tokens as LLM tokens, similar to how different words can have different meanings and that's reflected in their embedding?
Immediate downside is that mapping variable name to token and back would probably require indexing the whole codebase. You’d need a 1:1 mapping for every name that was in scope, and probably need to be clever about disambiguating names that come in and out of scope.
Re: Show HN: OSS Agent I built topped the TerminalBench on Gemini-3-flash-preview
#118Interesting things Dirac does: 1. Uses an optimized version of Hash-Anchored edits for file editing ( https://dirac.run/posts/hash-anchors-myers-diff-single-token ) 2. Utilizes language's AST to decide what to fetch into context, entirely avoids large code file reads 3. Batches all operations. Does large number of reads/edits simultaneously (you can see a video demo for deepseek-v4-flash here https://www.reddit.com/r…
I always wondered why AST's were not more of a part in both editing and scoping of changes/parsing code. I thought I read an article where they said 'grep' was just as effective. It kinda made sense for the case they were talking about.
Changes that are primarily code refactorings, like breaking up a large module into a bunch of smaller ones, or renaming a commonly-used class, are extremely tedious to review, both in LLM generated diffs and human-written PRs. You still have to do it; LLMs have a habit of mangling comments when moving code across files, while for a human, an unassuming "rename FooAPIClient to LegacyFooAPIClient" PR is the best place to leave a backdoor when taking over a developer's account. Nevertheless, many developers just LGTM changes like this because of the tedium involved in reviewing them.
If one could express such changes as a simple AST-wrangling script in a domain-specific language, which would then be executed in a trusted environment after being reviewed, that would decrease the review burden considerably.
I believe that with agentic development, the most important constraint we have is human time. Making the LLM better and faster won't help us much if the human still needs to spend a majority of their time reading code. We should do what we can to give us less code to read, without losing confidence in the changes that the LLM makes.
Re: Show HN: OSS Agent I built topped the TerminalBench on Gemini-3-flash-preview
#119It's really interesting how much the AI harness seems to matter. Going from 48% via Google's official results to 65% is a huge jump. I feel like I'm constantly seeing results that compare models and rarely seeing results that compare harnesses. Is there a leaderboard out there comparing harness results using the same models?
somewhat remarkably, claude code ranks last for Opus 4.6 - which may say something about cc, or say something about the benchmark