Live data from Hacker News

How version control will evolve for the agent boom

entire.io

61–70 of 72 posts

Re: How version control will evolve for the agent boom

#61

Earlier quoted context omitted.

Disagree completely. The spec is vital so that future changes continue to conform to it. Specs absolutely need to live in the VCS, because they continue to be needed to keep the code conformant. They essentially are a form of code now. And code goes in the VCS.

At scale, specs can only be vital to the degree to which their conformance testing is automated. Good specs should use a formal, runnable verification language. Otherwise you'll accumulate specs that are right when they ship, wrong in subtle ways 3 months in, and wrong in glaring ways 6 months in. AI doesn't change this dynamic, it amplifies it.

Not true.

This is about ensuring that when AI's make future changes, they aren't just looking at the existing code and making assumptions about intent. They should always be pulling the specs to ensure that changes maintain compatibility with the specified intents.

> Otherwise you'll accumulate specs that are right when they ship, wrong in subtle ways 3 months in, and wrong in glaring ways 6 months in. AI doesn't change this dynamic, it amplifies it.

Not if the changes you're making are always to the specs, as opposed to the code. The whole point here is that you don't change the code, you change the specs, then approve the code that the LLM changes as a result. This way the spec should never diverge from the code.

AI absolutely changes the dynamic so that code doesn't converge from the spec. That's the whole point, and the whole point of committing the specs like code.

Automating testing is great too, of course, but that's not the full picture. It ensures formal compliance, but doesn't encapsulate anything about the spirit or purpose of why the design in a certain way. Good specs do. The purpose/motivation sections and engineering guidelines are some of the most important for an LLM. Which is what helps the LLM figure out how to then best modify the existing code when features need to be changed or added.

Re: How version control will evolve for the agent boom

#62

Earlier quoted context omitted.

But conformance testing is where agents really excel at if you set things up right: * Black-box testing: the agent writing the tests cannot see implementation and the agent writing implementation cannot see tests, they only agree on a spec and an interface (the minimum needed to write tests). * Evaluate test coverage using code coverage, but when gaps are found communicate those gaps in terms of the specification. Go…

> Good specs should be grounded (complete and not ambiguous), they don't need to be formal IMHO, this is a mistake. I guess we play with it because there’s isn’t anything better nowadays. Writing and maintaining “specs” in plain english is painful.

> Writing and maintaining “specs” in plain english is painful.

The subject of this entire post is development with agents. Writing specs in English is how you do that. If you don't like it, then this is probably not the right article for you to be commenting on.

Re: How version control will evolve for the agent boom

#63

Earlier quoted context omitted.

At scale, specs can only be vital to the degree to which their conformance testing is automated. Good specs should use a formal, runnable verification language. Otherwise you'll accumulate specs that are right when they ship, wrong in subtle ways 3 months in, and wrong in glaring ways 6 months in. AI doesn't change this dynamic, it amplifies it.

Not true. This is about ensuring that when AI's make future changes, they aren't just looking at the existing code and making assumptions about intent. They should always be pulling the specs to ensure that changes maintain compatibility with the specified intents. > Otherwise you'll accumulate specs that are right when they ship, wrong in subtle ways 3 months in, and wrong in glaring ways 6 months in. AI doesn't cha…

> The whole point here is that you don't change the code, you change the specs,

> but doesn't encapsulate anything about the spirit or purpose of why the design in a certain way. Good specs do.

Specs are meta solutions. They describe the general shape of the solutions by refusing to make any technical decision that would leads to incidental problems and thus only needs to focus on the essential ones. So they're always simplistic, because they ignore the cascading effect of implementation decisions.

Generating code with AI is rolling the dice every time said generation is done. Proper implementation happens because with making decisions and going down a path, backtracking if necessary when it's no longer working. Going with AI is breaking down that continuity because they restart from scratch everytime.

Re: How version control will evolve for the agent boom

#64

Earlier quoted context omitted.

Not true. This is about ensuring that when AI's make future changes, they aren't just looking at the existing code and making assumptions about intent. They should always be pulling the specs to ensure that changes maintain compatibility with the specified intents. > Otherwise you'll accumulate specs that are right when they ship, wrong in subtle ways 3 months in, and wrong in glaring ways 6 months in. AI doesn't cha…

> The whole point here is that you don't change the code, you change the specs, > but doesn't encapsulate anything about the spirit or purpose of why the design in a certain way. Good specs do. Specs are meta solutions. They describe the general shape of the solutions by refusing to make any technical decision that would leads to incidental problems and thus only needs to focus on the essential ones. So they're alway…

> Specs are meta solutions. They describe the general shape of the solutions by refusing to make any technical decision that would leads to incidental problems and thus only needs to focus on the essential ones.

Yes, this is a feature not a bug. Then, code review ensures that the actual code makes the requisite technical decisions in a desirable way. This is much faster.

> Generating code with AI is rolling the dice every time said generation is done. Proper implementation happens because with making decisions and going down a path, backtracking if necessary when it's no longer working. Going with AI is breaking down that continuity because they restart from scratch everytime.

False. You only do the initial code generation an initial time, and then update the spec to specify (at a high level, ideally) the constraints that are missing. But then you have the AI modify the existing code to meet the new spec, not start from scratch. AI does not "restart from scratch everytime", it takes an updated spec plus existing code and updates the code to match the updated spec.

It's still "making decisions and going down a path," which you call proper implementation. That's the whole point.

Re: How version control will evolve for the agent boom

#65

Earlier quoted context omitted.

> The whole point here is that you don't change the code, you change the specs, > but doesn't encapsulate anything about the spirit or purpose of why the design in a certain way. Good specs do. Specs are meta solutions. They describe the general shape of the solutions by refusing to make any technical decision that would leads to incidental problems and thus only needs to focus on the essential ones. So they're alway…

> Specs are meta solutions. They describe the general shape of the solutions by refusing to make any technical decision that would leads to incidental problems and thus only needs to focus on the essential ones. Yes, this is a feature not a bug. Then, code review ensures that the actual code makes the requisite technical decisions in a desirable way. This is much faster. > Generating code with AI is rolling the dice…

The reason people don't bother with specs is that they're extraordinarily verbose, ambiguous, and simplistic at the same time. Which is why we have invented formal notations, because they're short, have one definition, and enough to create the whole thing, if not to understand the why. The latter can still be done in prose.

What a spec is truly useful for is that it lets you manipulate the solution at the idea stage, without tainting it with real world concerns. But to do it requires a lot knowledge about those real world concerns, so that your assumptions are not straight lies.

This is good for iteration. But any implementation ossifies your specs, and while code are more malleable than most, there's a certain points when the specs should be discarded and the actual implementation being the source of truth for any further development. In any project, that is done with Change Request, which only outline the specific area of changes, assuming the rest as being immutable.

> AI does not "restart from scratch everytime", it takes an updated spec plus existing code and updates the code to match the updated spec.

Maybe you can share an example of that workflow and an actual argument that would make it sustainable.

I've written the formal version of a spec (a suite of tests) and if there's one thing that would make the above idea ridiculous is how much wiggle room there is to follow the specs and be at the same time unusable.

Re: How version control will evolve for the agent boom

#66
post #45

> Our hypothesis is simple: session logs are now the most important artifact in software development, and should be stored alongside the code itself in the repository. I don’t think this scales. We recently have been doing “spec driven development” and we are committing the specs and prompts to our repo, alongside the generated code. At the beginning it seems fine: you wanna change something, you update the spec and…

Exactly. The spec should go in a git comment, NOT vcs. Or, you keep the document in history for the duration of the PR and remove it post review or pre merge

The problem I see is that git commit meaaages are not as flexible at storing context information as files. Or easier to retrieve. We do have --grep to search them but it's definitely not as poweful as all the tools we have to handle files at the OS level. As verbose as storing specs in files is, I also prefer it

Re: How version control will evolve for the agent boom

#67

> Our hypothesis is simple: session logs are now the most important artifact in software development, and should be stored alongside the code itself in the repository. I don’t think this scales. We recently have been doing “spec driven development” and we are committing the specs and prompts to our repo, alongside the generated code. At the beginning it seems fine: you wanna change something, you update the spec and…

If only we had a way of describing exactly and in great detail to the machine what to do! Some sort of language, maybe, idk… /s

Re: How version control will evolve for the agent boom

#70

Anyone else finds the LLMs version control discipline lacking? The engineering practices in general are lacking. Not tests nor assumptions validation ever, unless explicitly asked.

Have you tried creating skills for your agents to follow your patterns?
Post reply on HN