Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

If AI writes code, should the session be part of the commit?

github.com

Post reply on HN
Print view View original

171–180 of 410 posts

Re: If AI writes code, should the session be part of the commit?

#171

Post by D-Machine »

onion2k wrote: post #160
globular-toast wrote: post #155

Earlier quoted context omitted.

This only works if the software is still crafted by a human and merely using AI as a tool. In that case the use of AI is similar to using editor macros or test-driven development. I don't need to see that process playing out in real time. It's less clear to me if the software isn't crafted by a human at all, though. In that case I would prefer to see the prompt.

I agree that fully agentic development will change things, but I don't know how. I'm still very much in the human-in-the-loop phase of AI where I want to understand and verify that it's not done anything silly. I care far more about the code that I'm deploying than the prompt that got me there and probably will for a long time. So will my prodsec team.

Appreciate this very sane take. The actual code always is more important than the intentions, and this is basically tautological.

When dealing with a particularly subtle / nuanced issue, knowing the intentions is still invaluable, but this is usually rare. How often AI code runs you into these issues is currently unclear, and constantly changing (and how often such issue are actually crucial depends heavily on the domain).

Re: If AI writes code, should the session be part of the commit?

#172

Post by jumploops »

I've been experimenting with a few ways to keep the "historical context" of the codebase relevant to future agent sessions.

First, I tried using simple inline comments, but the agents happily (and silently) removed them, even when prompted not to.

The next attempt was to have a parallel markdown file for every code file. This worked OK, but suffered from a few issues:

1. Understanding context beyond the current session

2. Tracking related files/invocations

3. Cold start problem on an existing codebases

To solve 1 and 3, I built a simple "doc agent" that does a poor man's tree traversal of the codebase, noting any unknowns/TODOs, and running until "done."

To solve 2, I explored using the AST directly, but this made the human aspect of the codebase even less pronounced (not to mention a variety of complex edge-cases), and I found the "doc agent" approach good enough for outlining related files/uses.

To improve the "doc agent" cold start flow, I also added a folder level spec/markdown file, which in retrospect seems obvious.

The main benefit of this system, is that when the agent is working, it not only has to change the source code, but it has to reckon with the explanation/rationale behind said source code. I haven't done any rigorous testing, but in my anecdotal experience, the models make fewer mistakes and cause less regressions overall.

I'm currently toying around with a more formal way to mark something as a human decision vs. an agent decision (i.e. this is very important vs. this was just the path of least resistance), however the current approach seems to work well enough.

If anyone is curious what this looks like, I ran the cold start on OpenAI's Codex repo[0].

[0]https://github.com/jumploops/codex/blob/file-specs/codex-rs/...

Re: If AI writes code, should the session be part of the commit?

#173

Post by jedberg »

The way I write code with AI is that I start with a project.md file, where I describe what I want done. I then ask it to make a plan.md file from that project.md to describe the changes it will make (or what it will create if Greenfield).

I then iterate on that plan.md with the AI until it's what I want. I then ask it to make a detailed todo list from the plan.md and attach it to the end of plan.md.

Once I'm fully satisfied, I tell it to execute the todo list at the end of the plan.md, and don't do anything else, don't ask me any questions, and work until it's complete.

I then commit the project.md and plan.md along with the code.

So my back and forth on getting the plan.md correct isn't in the logs, but that is much like intermediate commits before a merge/squash. The plan.md is basically the artifact an AI or another engineer can use to figure out what happened and repeat the process.

The main reason I do this is so that when the models get a lot better in a year, I can go back and ask them to modify plan.md based on project.md and the existing code, on the assumption it might find it's own mistakes.

Re: If AI writes code, should the session be part of the commit?

#174

Post by stackghost »

jedberg wrote: post #173

The way I write code with AI is that I start with a project.md file, where I describe what I want done. I then ask it to make a plan.md file from that project.md to describe the changes it will make (or what it will create if Greenfield). I then iterate on that plan.md with the AI until it's what I want. I then ask it to make a detailed todo list from the plan.md and attach it to the end of plan.md. Once I'm fully sa…

>I then iterate on that plan.md with the AI until it's what I want.

Which tools/interface are you using for this? Opencode/claude code? Gas town?

Re: If AI writes code, should the session be part of the commit?

#175

Post by StrangeSound »

stackghost wrote: post #174
jedberg wrote: post #173

The way I write code with AI is that I start with a project.md file, where I describe what I want done. I then ask it to make a plan.md file from that project.md to describe the changes it will make (or what it will create if Greenfield). I then iterate on that plan.md with the AI until it's what I want. I then ask it to make a detailed todo list from the plan.md and attach it to the end of plan.md. Once I'm fully sa…

>I then iterate on that plan.md with the AI until it's what I want. Which tools/interface are you using for this? Opencode/claude code? Gas town?

I find that Antigravity is really good for this. You can comment on the plan documents in-line.

Re: If AI writes code, should the session be part of the commit?

#178

Post by onion2k »

mikepurvis wrote: post #116
D-Machine wrote: post #108

Earlier quoted context omitted.

I think this is the right analogy, contrary to some other very poor ones in this thread. Yes, it is rare to really look at commit messages, but it can be invaluable in some cases. With vibe-coding, you risk having no documentation at all for the reasoning (AI comments and tests can be degenerate / useless), but the prompts, at bare minimum, reveal something about the reasoning / motivation. Whether this needs to be i…

Depending on the size it might make sense as a kind of commit metadata reference to external, like the signed-off-by field. Chat-Session-Ref: claude://gjhgdvbnjuteshjoiyew Perhaps that could also link out to other kinds of meeting transcripts or something too.

That wouldn't be very portable. A benefit of committing to your history is that it lives with the code no matter where the code or the AI service you use goes.

Re: If AI writes code, should the session be part of the commit?

#179

Post by devmor »

majormajor wrote: post #145
rzerowan wrote: post #122

Earlier quoted context omitted.

This is a central problem that weve already seen proliferate wildly in Scientific research , and currently if the same is allowed to be embedded in foundational code. The future outlook would be grim. Replication crisis[1]. Given initial conditions and even accounting for 'noise' would a LLm arrive at the same output.It should , for the same reason math problems require one to show their working. Scientific papers re…

You can leave commit messages or comments without spamming your history with every "now I'm inspecting this file..." or "oops, that actually works differently than I expected" transcript. In fact, I'd wager that all that excess noise would make it harder to discern meaningful things in the future than simply distilling the meaningful parts of the session into comments and commit messages.

Those messages are part of the linguistic context used to generate the code, though. Don’t confuse them for when humans (or human written programs) display progress messages.

If they aren’t important for your specific purposes, you can summarize them with an LLM.

Re: If AI writes code, should the session be part of the commit?

#180

Post by panarky »

eru wrote: post #111
827a wrote: post #94

IMO: This might be a contrarian opinion, but I don't think so. Its much the same problem as asking, for example, if every single line you write, or every function, becomes a commit. The answer to this granularity is, much like anything, you have to think of the audience: Who is served by persisting these sessions? I would suspect that there is little reason why future engineers, or future LLMs, would need access to t…

> Its much the same problem as asking, for example, if every single line you write, or every function, becomes a commit. Hmm, I think that's the wrong comparison? The more useful comparison might be: should all your notes you made and dead ends you tried become part of the commit?

When a human writes the code should all their slack messages about the project be committed into the repo?
Post reply on HN