I use agents to ship small products and I behave to them like to real human. In real world, would you add the test of the codec to the same engineer who wrote it? No. Same here, just have one agent to write the code (engineer) and another one to test the code (quality assurance) They are alive creatures :)
Building an (almost) fully self-hosted, sandboxed, agentic software factory
61–70 of 75 posts
Re: Building an (almost) fully self-hosted, sandboxed, agentic software factory
#62In such systems, producing code is the easy part, verification is hard. Verification via test cases just feels like the same agent validating its own assumptions. Wondering what the actual verification loop looks like once you start taking these systems to production.
I've been enforcing TDD with Claude since the start and to be honest it actually works great. It doesn't write bad tests just to fit a bad implementation. And when running against a Pglite in-memory database it can iterate rapidly.
Re: Building an (almost) fully self-hosted, sandboxed, agentic software factory
#63In such systems, producing code is the easy part, verification is hard. Verification via test cases just feels like the same agent validating its own assumptions. Wondering what the actual verification loop looks like once you start taking these systems to production.
I've been enforcing TDD with Claude since the start and to be honest it actually works great. It doesn't write bad tests just to fit a bad implementation. And when running against a Pglite in-memory database it can iterate rapidly.
[flagged]
Re: Building an (almost) fully self-hosted, sandboxed, agentic software factory
#64Earlier quoted context omitted.
People have been making analogies like this since 2023, hardly worth a link to a random twitter thread.
why do people find comments like these necessary? What did you gain by making this comment?
In principle I’d agree usually, but I think this specific case, it’s reasonable to highlight that the “experts get value” argument doesn’t really add much to the discourse anymore.
We know this and you just need to look at what’s happening with LLMs in maths to see it’s true. It doesn’t add anything to the verification point however.
Re: Building an (almost) fully self-hosted, sandboxed, agentic software factory
#65So - apparently it's not fully self-hosted, since I don't see a GPU. I'm interested in hearing from folks who are hosting their own GPU to run coding models. So far my own results are... not great. Seems like frontier models are needed via the big providers?
My favorite real-world example that I worked on: I created a YouTube documentary series about corruption in a small town in Illinois. This required downloading thousands of hours of government meeting videos from YouTube, transcribing + chunking + embedding, summarizing meeting segments, running a couple passes of validation, then searching for interesting storylines. I also scraped thousands of public documents whic…
Do you have a writeup of how you did it, which tools you used? I have a similar project on a much more mundane topic, where I want to build a searchable index of Youtube videos, blog posts, etc. based on their content.
Re: Building an (almost) fully self-hosted, sandboxed, agentic software factory
#66[flagged]
Re: Building an (almost) fully self-hosted, sandboxed, agentic software factory
#67In such systems, producing code is the easy part, verification is hard. Verification via test cases just feels like the same agent validating its own assumptions. Wondering what the actual verification loop looks like once you start taking these systems to production.
I think self-verification can work in some ways. For example, I (well, an LLM) recently wrote some caching optimizations for a particular filesystem intensive workload on a cluster. This is an internal thing, not customer facing, for context.
Deploy to the staging cluster, have the LLM monitor with read access to the logs. Opus is… better than a human at looking at logs and comparing timings and checking for warnings and errors.
If it passes a small load test there, deploy to the live cluster and actively monitor. If it doesn’t fall over under load, is faster, is still correct, not throwing errors it works good enough.
Does much more matter? This produced a fairly large performance improvement. Verification in this case is the process of running it under load and double checking the code isn’t doing anything truly wacky.
User facing apps are harder to validate, sure, but you can easily have an LLM looping looking for errors, log issues, etc.
For home apps, this is more validation than you would have personally done
Re: Building an (almost) fully self-hosted, sandboxed, agentic software factory
#68In such systems, producing code is the easy part, verification is hard. Verification via test cases just feels like the same agent validating its own assumptions. Wondering what the actual verification loop looks like once you start taking these systems to production.
Agreed, we had agents writing standard integration/unit/e2e tests but turned out to be a ton of work reviewing just that and hard to wrap one’s head around as a developer. We ended up building our own test platform to reduce the human bottleneck in the verification loop. Takes the form of a hermetic test environment that run all our code and infra in a way that matches production. Our tests interact with it from the…
This is a good insight.
While doing local development where your environment is already setup and dependencies are installed doing something like this seems viable after the initial effort.
But how would this work in a software-factory setup, where coding agents run on fresh VMs or when using something like Claude Code on the web? How do you quickly recreate the required environment and dependencies there?
Re: Building an (almost) fully self-hosted, sandboxed, agentic software factory
#69Earlier quoted context omitted.
My favorite real-world example that I worked on: I created a YouTube documentary series about corruption in a small town in Illinois. This required downloading thousands of hours of government meeting videos from YouTube, transcribing + chunking + embedding, summarizing meeting segments, running a couple passes of validation, then searching for interesting storylines. I also scraped thousands of public documents whic…
Sounds amazing! What’s the name of the series if we wanted to find it on YouTube?
Honored by your interested :) It is called "Dolton Documentaries".
Re: Building an (almost) fully self-hosted, sandboxed, agentic software factory
#70Earlier quoted context omitted.
Sounds amazing! What’s the name of the series if we wanted to find it on YouTube?
I don't know the creator handle or series name, but it's got to be Dixon, IL. lol
Excellent guess, and the documentary All The Queen's Horses does a much better job explaining that story than I ever could!
This one is about Dolton, IL - the series is called "Dolton Documentaries" on YouTube.