Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Grepathy – Claude made a decision nobody approved

github.com

Post reply on HN
Print view View original

31–40 of 49 posts

Re: Show HN: Grepathy – Claude made a decision nobody approved

#31

Post by wakawaka28 »

triyambakam wrote: post #23

Couldn't this just be in git commits and give the a skill to look there?

Not really. That's like saying a design document should be pulled out of commit messages. There are a few reasons I can think of why this would be a bad idea. Maybe the biggest is that commit messages should be quickly readable and only explain the changes in the commit. To truly explain the reasons a path was chosen by a LLM you need at least a summary of what was asked for, and related discussions or "don't do" instructions. I've heard of people wanting to commit the entire chat logs as files before, but that is noisy and might only get you halfway there as far as explanations go. I certainly don't want this data bloat, snooping, etc. I would rather for important details to be curated. I can see why maximum details could be helpful sometimes but I don't want to bloat the repo with that.

Re: Show HN: Grepathy – Claude made a decision nobody approved

#33

Post by evansjp »

embedding-shape wrote: post #5

> Decisions only, never your messages For me this is basically the opposite of how it would need to be, the decisions are all in my messages to the agent, almost never directly in the agent's replies, sometimes indirectly though I suppose. Interesting to see that apparently some people use these tools and are the ones listening and doing what the LLM decide and says, rather than the opposite. > Ran a blind eval befor…

- the distiller reads your messages, it just doesn't quote them. Decisions you directed get captured and labeled `directed`.

- eval is public here: https://github.com/evansjp/grepathy/blob/main/docs/REPORT.md

- fair point on the last one too. For the original Clerk incident the transcript is deleted, so I can't prove nobody approved it. Maybe I even waved it through. That's kind of the point though. Right now there's no record either way. I'd rather have some sort of receipt.

Re: Show HN: Grepathy – Claude made a decision nobody approved

#34

Post by evansjp »

SpicyLemonZest wrote: post #8

Can we get the transcript of the session where you had Claude write this post? I want to understand the "why" behind this project, and whether a human being was at any point involved in the "blind, pre-registered evaluation of Grepathy against an honest baseline".

lol yes, agents ran the mechanics. I set the bars before any run and audited the keys. And the "why" behind the project is literally in the repo's .ai/why/ where you grep it if you would like.

Re: Show HN: Grepathy – Claude made a decision nobody approved

#35

Post by evansjp »

trjordan wrote: post #30
skinfaxi wrote: post #28

Earlier quoted context omitted.

How does your solution differ from OPs? edit: To me it looks like yours has the following negative qualities: - closed source - requires signup for some service - says code doesnt leave my machine but this connects to your servers - requires Goose AI as a dependency

So, we tried feeding the logs back to the LLM, and it mostly produced slop. Lots of decisions nobody cared about. The biggest things that moved the needle were: - Baseline it. We mine previous logs, github comments, etc. for "what you care about." That helps pull out decisions that you actually care to read. - Anchor to code. "The code enshrines this decision" is more interesting than "the agent self-talked this." Ag…

hey that's awesome! yeah the eval showed first pass was only ~65% real decisions. The fix that stuck was an entry has to name a real file it touches or it gets dropped. A code decision names code.

I agree agents don't always self-talk decisions, that's why we distill the whole transcript after the fact instead of asking them to log anything. Your baselining idea is good!

Re: Show HN: Grepathy – Claude made a decision nobody approved

#36

Post by NateEag »

pfannkuchen wrote: post #29
NateEag wrote: post #24

Earlier quoted context omitted.

Whether or not he's serious, as someone who's grudgingly using Claude at work due to mandates, the productivity gains do shrink massively (and sometimes go negative) if you actually attempt to gain the level of understanding you'd have of the system had you written it yourself. Does it matter if the developers understand the system they maintain? I guess that depends whether the genAI maximalists turn out to be right…

What happens when there’s an outage though? Just hope the LLM can fix it? Who is on the hook if it can’t? Do you have people frantically trying to reverse engineer a basically unfamiliar-to-them code base at 2am while bleeding cash?

As far as I can tell, yes, the workflow is "assume the LLM can fix it."

I've certainly seen people on this site defend that idea vociferously.

I won't tell you it's impossible - Claude does do legitimately amazing things.

It does not, however, seem to me to actually have deep understanding of what it's doing. "Amazing" does not mean "admirable" or "trustworthy."

Re: Show HN: Grepathy – Claude made a decision nobody approved

#37

Post by NateEag »

pavel_lishin wrote: post #27
NateEag wrote: post #24

Earlier quoted context omitted.

Whether or not he's serious, as someone who's grudgingly using Claude at work due to mandates, the productivity gains do shrink massively (and sometimes go negative) if you actually attempt to gain the level of understanding you'd have of the system had you written it yourself. Does it matter if the developers understand the system they maintain? I guess that depends whether the genAI maximalists turn out to be right…

> Does it matter if the developers understand the system they maintain? Are they maintaining it if all they do is type prompts into Claude?

Yes, they are.

They may be doing a poor job of it - time will tell.

But, at present, they're still the ones merging PRs and deploying changes to production.

Re: Show HN: Grepathy – Claude made a decision nobody approved

#38

Post by albert_e »

weird-eye-issue wrote: post #12
albert_e wrote: post #10

> The reasoning was in a transcript on my laptop. Claude Code deletes those after 30 days by default. Two of my projects lost their whole history that way. Wait what? Claude Code (and Codex) transcripts are auto-deleted?? That's a treasure trove of information for all my projects and I had built some tooling and workflows tahat deal with extracting insights from them -- withut ever realizing I mightbe losing old chat…

Yes but it's just a setting you can change I don't think you really need an archival step

Yes done this now. Thanks.

I will still try and include archival -- storage is cheap. A reinstall of Claude Code or buggy upgrade might still wipe history. I also ocassionally use multiple PCs -- so there is some small value in consolidating them all for analytics and insights.

(When I have a weekly quota unused and about to expire -- i task Opus with churning through these archives and surfacing learining, best practices, what's not working etc)

Re: Show HN: Grepathy – Claude made a decision nobody approved

#39

Post by albert_e »

trjordan wrote: post #14
albert_e wrote: post #10

> The reasoning was in a transcript on my laptop. Claude Code deletes those after 30 days by default. Two of my projects lost their whole history that way. Wait what? Claude Code (and Codex) transcripts are auto-deleted?? That's a treasure trove of information for all my projects and I had built some tooling and workflows tahat deal with extracting insights from them -- withut ever realizing I mightbe losing old chat…

> "cleanupPeriodDays": 99999 Throw that in ~/.claude/settings.json

Thanks. Updated.

Codex apparently does not do auto delete presently.

Need to watch.

Re: Show HN: Grepathy – Claude made a decision nobody approved

#40

Post by pavel_lishin »

NateEag wrote: post #37
pavel_lishin wrote: post #27

Earlier quoted context omitted.

> Does it matter if the developers understand the system they maintain? Are they maintaining it if all they do is type prompts into Claude?

Yes, they are. They may be doing a poor job of it - time will tell. But, at present, they're still the ones merging PRs and deploying changes to production.

It brings to mind the Chinese Room argument: https://en.wikipedia.org/wiki/Chinese_room

If a person in a Chinese Room doesn't actually understand Chinese, though the system as a whole behaves as if it does - can we say that the engineer merging PRs and deploying to production understands the system or maintains it?

Post reply on HN