Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Graft – Claude Code hooks that cut grep tokens by 42%

github.com

Post reply on HN
Print view View original

31–40 of 49 posts

Re: Show HN: Graft – Claude Code hooks that cut grep tokens by 42%

#31

Post by gabosarmiento »

shrishdwi wrote: post #24
gabosarmiento wrote: post #12

What's the benchmark against graphify?

In our tests, graft found the right code about twice as often as graphify (MRR 0.73 vs 0.38, recall@10 54% vs 20%). The difference is simple: graft searches inside the actual code, while graphify only looks at names and file paths. You feel it while working too. graft hands the agent the exact file and line it needs for each question, so Claude keeps using graft. Same is true for any other cli tools, claude never act…

Thanks, that makes sense. The “Claude actually uses it” part is especially interesting. I actually find it better this approach don’t add another skill Claude might ignore, but make better retrieval the default.

Re: Show HN: Graft – Claude Code hooks that cut grep tokens by 42%

#33

Post by shrishdwi »

icodestuff wrote: post #28
shrishdwi wrote: post #20

Earlier quoted context omitted.

For the staleness specifically we are using hooks on claude code and codex, where on every turn, or post edit or pre-tool use, we sync the graph. so that graph never goes stale. We have run tests on DeepSWE as well which are long running tasks, we got 20% better accuracy on the tasks where sonnet 5 failed. didn't want to post that numbers yet as I think we can do better on DeepSWE and on a cheaper model like gpt-5.6-…

The graph is pre-filling the context that does the sync, right? Doesn't that create a bias towards what's already present? If anything hallucinated does make it in at some point, won't the sync just reinforce that hallucination if some new component touches the bad info, just like a long session rabbit-holing? New sessions, in contrast, have no contextual assumptions and often catch those problems.

[flagged]

Re: Show HN: Graft – Claude Code hooks that cut grep tokens by 42%

#35

Post by seizethecheese »

shrishdwi wrote: post #30
seizethecheese wrote: post #26

This looks cool and the mechanism looks plausible. I found the experience of trying to understand whether the claims here are legit to be aggravating. First of all, the whole readme section about benchmarks appears to be Claude/Codex written. What a slog to read this. Second, they claim success on SWE-Bench Verified, but it's only on 50 tasks, not making clear how these tasks are chosen. I know from experience that y…

we are running them on a recurring basis, so will keep on updating that 50 number. and hence as it's currently running that section gets updated by claude only.

Wow, this is really weird. I looked at the github and see you published benchmark updates in these sizes of N:

9 → 20 → 36 → 50

These are super arbitrary numbers of benchmarks to run (9, 11, 16, 14). I see a few explanations for this:

> Claude dropped "pathological trials" for you before publishing. This would make your results quite dishonest.

> You chose different sized task groups (almost certainly not the case).

> You excluded timed out trials, or something (also would be dishonest).

Hopefully there's a better explanation here!

Re: Show HN: Graft – Claude Code hooks that cut grep tokens by 42%

#36

Post by shrishdwi »

seizethecheese wrote: post #35
shrishdwi wrote: post #30

Earlier quoted context omitted.

we are running them on a recurring basis, so will keep on updating that 50 number. and hence as it's currently running that section gets updated by claude only.

Wow, this is really weird. I looked at the github and see you published benchmark updates in these sizes of N: 9 → 20 → 36 → 50 These are super arbitrary numbers of benchmarks to run (9, 11, 16, 14). I see a few explanations for this: > Claude dropped "pathological trials" for you before publishing. This would make your results quite dishonest. > You chose different sized task groups (almost certainly not the case).…

that's just when our claude limit's were about to exhaust :/ as we have to spin up a whole claude session to test it.

but you could 100% be Sherlock Holmes!

Re: Show HN: Graft – Claude Code hooks that cut grep tokens by 42%

#37

Post by seizethecheese »

shrishdwi wrote: post #36
seizethecheese wrote: post #35

Earlier quoted context omitted.

Wow, this is really weird. I looked at the github and see you published benchmark updates in these sizes of N: 9 → 20 → 36 → 50 These are super arbitrary numbers of benchmarks to run (9, 11, 16, 14). I see a few explanations for this: > Claude dropped "pathological trials" for you before publishing. This would make your results quite dishonest. > You chose different sized task groups (almost certainly not the case).…

that's just when our claude limit's were about to exhaust :/ as we have to spin up a whole claude session to test it. but you could 100% be Sherlock Holmes!

HN commenters shouldn't have to be Sherlock Holmes!

Extraordinary claims should be supported by evidence. You could have easily posted the seed, the 50 questions, and the results for claude and graft on each, but didn't.

I also see Graft got 16/16 on one update and 2/14 on another. Further evidence that each group of questions was selected somehow, not random.

Post reply on HN