A case study in testing with 100+ Claude agents in parallel

Are you seriously interested in the answer, or are you just mad?

I could give you some pointers, but will only type it out if there is a point

> The instinct should be to tweak the agent to do it right.

Ah, yes; must always remember to add "And don't make any mistakes" into the prompt /s

I think we are talking about different timespans. I am talking about change in the world after decades of something like that happening. How those Senior Engineers will know how good software looks like if they would never write it themeselves? Imagine looking at someone driving a car for 20 years. Will it be enough for you to drive a car yourself?

Thinking about that always makes me think about Foundation, The Merchant Princess. Mallow travels to the edge of the Empire to look how things are on one of those worlds. He learns that there is the cast of the tech priests and those people have absolutely no idea how those devices actually work.

He said:

> The machines work from generation to generation automatically, and the caretakers are a hereditary caste who would be helpless if a single D-tube in all that vast structure burned out

It was a sign of severe decline of the entire empire. People had no idea how devices work and they would not be able to reproduce it or even repair if one would broke.

It was recurring premise of civilisation decline in the series: no proper maintaince and people loosing interests and knowledge how things are done and how they work.

I just wondering if this is not the same thing starting to happining know with our civilisation.

And evolution? Evolution means mass extinction of species and its normal. I am not sure about you but I would rather avoid any mass extinction regarding humanity.

It is about tweaking inline documentation to make sure that

1. It is not ambiguous 2. It is as complete as possible.

I am surprised that I got down voted for proposing the improve a code base such that agents can run on it as a means to increased productivity.

(author here) I agree that it's super important to understand what software actually does--that's part of the whole reason we made mngr in the first place!

I believe we can use these types of tools to make software more understandable, and mngr is an example of how to do that.

In our case study, we're using AI to increase our test coverage, and if you look at it, I would argue that we are making it more understandable--now instead of just having 100's of tests, we simply have a document that describes how the software is supposed to work, and the tests are linked to that document, and checked to ensure that they conform.

That means that anyone--not just the author of the software--is now able to read through the high level tutorial description of how the commands work in order to understand what the program should do!

And as for the tests themselves, we've been able to make nice testing infrastructure--like the transcripts and recordings that were highlighted in the post--to make it even easier for us to verify the behavior of the software.

We also have an incredibly detailed style guide and set of tests and guidelines to ensure that the entire code base is consistent, and high quality. You can drop into any of the code and pretty quickly understand what is happening. And if not, claude will do an excellent job of describing how any given component works, and how it relates to the others.

Finally, mngr itself is designed to be fully transparent when it is running--you can literally attach to the coding agent you are running and see exactly what is happening, and the program makes extensive log outputs for everything it does (feel free to open a PR if you'd like to see more!)

It's not perfect formal verification, but it does feel like we're making meaningful progress on making it easier to understand software--not harder.