Live data from Hacker News

Show HN: OSS Agent I built topped the TerminalBench on Gemini-3-flash-preview

github.com

111–120 of 167 posts

Re: Show HN: OSS Agent I built topped the TerminalBench on Gemini-3-flash-preview

#111

Really impressive results. The point about the harness mattering more than the model is spot on — we've seen similar patterns in our own work. One thing that stood out to me is your use of hash-anchored edits + AST-based context selection. We're building something in a similar direction with the Sovereign AI Stack, but with a stronger focus on governance and verification. Curious — did you run into issues with contex…

I'd be curious to hear more about your work on the 'Sovereign AI Stack'. I'm also working on a project that prioritizes governance and verification and I'd love to compare notes.

Re: Show HN: OSS Agent I built topped the TerminalBench on Gemini-3-flash-preview

#112

How do I connect it to a local llama.cpp instance?

It supports LMStudio or you can start a local endpoint, then run

OPENAI_COMPATIBLE_CUSTOM_KEY="xxx" dirac -y --provider "https://localhost/v1" --model "hi..."

Re: Show HN: OSS Agent I built topped the TerminalBench on Gemini-3-flash-preview

#113
post #33

It's really interesting how much the AI harness seems to matter. Going from 48% via Google's official results to 65% is a huge jump. I feel like I'm constantly seeing results that compare models and rarely seeing results that compare harnesses. Is there a leaderboard out there comparing harness results using the same models?

Isn't that what terminal-bench does?

Re: Show HN: OSS Agent I built topped the TerminalBench on Gemini-3-flash-preview

#114
post #36

Earlier quoted context omitted.

https://github.com/dirac-run/dirac#-evals README has eval of 8 tasks over 7 agents (including both pi and omp). Pi-mono costs second lowest across the 8 tasks (after Dirac) but occasionally misses produces incomplete changes. Interestingly, 2 tasks where pi missed some changes both were the tasks that benefitted from AST symbol understanding (e.g. find all instances of things that refer to this symbol and change thos…

re. bash type tooling-- it doesnt mean an agent cannot use ast: using treesitter cli this should be perfect possible

I assume that this benchmarks where done without any modifications to the default open-sourced harness. treesitter CLI would be an extra plugin for pi-mono, put I'd be equally curious about whether it would accomplish the task.

Re: Show HN: OSS Agent I built topped the TerminalBench on Gemini-3-flash-preview

#115
post #83

Earlier quoted context omitted.

Maybe the future isn't a human-like centralized intelligence but an octopus-like decentralized intelligence where more focus is placed on making the harness itself "smart"

That would be counter to AI company goals. They want harness to be dumb and models to be smart so they can sell models.

Not really. Anthropic for example sells both the harness and the models as a unified kit via Claude Code, it is in their best interest to make sure both parts work as well as possible, via reinforcement learning of previous usage as well for new model performance increases.

Re: Show HN: OSS Agent I built topped the TerminalBench on Gemini-3-flash-preview

#116

I haven't tried it, but I'm curious why you decided to implement a whole new harness over just writing extensions in pi. From whatever I've done with pi so far, the extension api is quite extensive. Hash anchored edits, for example, can definitely be implemented in pi. Anyhow, thank you for showing us your project and will be checking it out later. Cheers!

I've been looking into local LLMs and new harnesses recently, how good is Pi compared to OpenCode, I'm seeing that it's a lot better? What are the best models and customizations for it to fully utilize it?

Re: Show HN: OSS Agent I built topped the TerminalBench on Gemini-3-flash-preview

#117
post #106

Earlier quoted context omitted.

I always wondered why AST's were not more of a part in both editing and scoping of changes/parsing code. I thought I read an article where they said 'grep' was just as effective. It kinda made sense for the case they were talking about.

Has anybody thought about encoding AST tokens as LLM tokens, similar to how different words can have different meanings and that's reflected in their embedding?

Language keywords are almost definitely individual tokens. But I think you mean more than that. Basically replacing identifiers with special tokens as well. It’s worth a shot but there’s some practical problems.

Immediate downside is that mapping variable name to token and back would probably require indexing the whole codebase. You’d need a 1:1 mapping for every name that was in scope, and probably need to be clever about disambiguating names that come in and out of scope.

Re: Show HN: OSS Agent I built topped the TerminalBench on Gemini-3-flash-preview

#118

Interesting things Dirac does: 1. Uses an optimized version of Hash-Anchored edits for file editing ( https://dirac.run/posts/hash-anchors-myers-diff-single-token ) 2. Utilizes language's AST to decide what to fetch into context, entirely avoids large code file reads 3. Batches all operations. Does large number of reads/edits simultaneously (you can see a video demo for deepseek-v4-flash here https://www.reddit.com/r…

I always wondered why AST's were not more of a part in both editing and scoping of changes/parsing code. I thought I read an article where they said 'grep' was just as effective. It kinda made sense for the case they were talking about.

I think we should use ASTS more, not for performance, but for easier code review.

Changes that are primarily code refactorings, like breaking up a large module into a bunch of smaller ones, or renaming a commonly-used class, are extremely tedious to review, both in LLM generated diffs and human-written PRs. You still have to do it; LLMs have a habit of mangling comments when moving code across files, while for a human, an unassuming "rename FooAPIClient to LegacyFooAPIClient" PR is the best place to leave a backdoor when taking over a developer's account. Nevertheless, many developers just LGTM changes like this because of the tedium involved in reviewing them.

If one could express such changes as a simple AST-wrangling script in a domain-specific language, which would then be executed in a trusted environment after being reviewed, that would decrease the review burden considerably.

I believe that with agentic development, the most important constraint we have is human time. Making the LLM better and faster won't help us much if the human still needs to spend a majority of their time reading code. We should do what we can to give us less code to read, without losing confidence in the changes that the LLM makes.

Re: Show HN: OSS Agent I built topped the TerminalBench on Gemini-3-flash-preview

#119
post #33

It's really interesting how much the AI harness seems to matter. Going from 48% via Google's official results to 65% is a huge jump. I feel like I'm constantly seeing results that compare models and rarely seeing results that compare harnesses. Is there a leaderboard out there comparing harness results using the same models?

the most cited is terminal bench 2.0, but its also plagued by cheating accusations and benchmaxxing.

somewhat remarkably, claude code ranks last for Opus 4.6 - which may say something about cc, or say something about the benchmark

[0] https://www.tbench.ai/leaderboard/terminal-bench/2.0

Post reply on HN