Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building an (almost) fully self-hosted, sandboxed, agentic software factory

blog.jakesaunders.dev

Post reply on HN
Print view View original

41–50 of 75 posts

Re: Building an (almost) fully self-hosted, sandboxed, agentic software factory

#41

Post by alexpotato »

ashu1461 wrote: post #27

In such systems, producing code is the easy part, verification is hard. Verification via test cases just feels like the same agent validating its own assumptions. Wondering what the actual verification loop looks like once you start taking these systems to production.

I just wrote a Twitter thread about how LLMs are like chainsaws [0]

In short:

- they are very powerful

- beginners feel like they are just as good as experts

- but they can cause a LOT of damage if you don't know what you are doing

- experts will get way more done and beginners may cut off a limb or drop a tree on their car

0 - https://x.com/alexpotato/status/2090872063464509912?s=20

Re: Building an (almost) fully self-hosted, sandboxed, agentic software factory

#42

Post by RajT88 »

iforgetti wrote: post #23
primitivesuave wrote: post #20

Earlier quoted context omitted.

My favorite real-world example that I worked on: I created a YouTube documentary series about corruption in a small town in Illinois. This required downloading thousands of hours of government meeting videos from YouTube, transcribing + chunking + embedding, summarizing meeting segments, running a couple passes of validation, then searching for interesting storylines. I also scraped thousands of public documents whic…

Sounds amazing! What’s the name of the series if we wanted to find it on YouTube?

I don't know the creator handle or series name, but it's got to be Dixon, IL. lol

Re: Building an (almost) fully self-hosted, sandboxed, agentic software factory

#43

Post by t0mas88 »

Kinrany wrote: post #39

Is there a name for the infrastructure stack that is designed to be operated by a human instead? Spawn agents each in a separate VM with the repo checked out and the tooling set up, allow them to spawn subagents in the same manner, but pull their changes from their branches (set up as remotes in your own dev environment) and merge into trunk yourself?

I've created a simple project for that, except that by default they open PRs on Forgejo/Github/Gitlab. It's open source and the workflows are config files, so you can configure it to allow you to pull from them instead of creating a PR. Source: https://github.com/smithy-ai/smithy-ai

One way I use it that is similar to what I think you want is to run a local Forgejo instance (just like OP) and have the Dockerised Claude instances work in that, then pull from there to my local git repo, review, test etc and push from my local repo to a remote like Github.

Re: Building an (almost) fully self-hosted, sandboxed, agentic software factory

#45

Post by 1123581321 »

RajT88 wrote: post #42
iforgetti wrote: post #23

Earlier quoted context omitted.

Sounds amazing! What’s the name of the series if we wanted to find it on YouTube?

I don't know the creator handle or series name, but it's got to be Dixon, IL. lol

Can’t be if they were having to scrub government meetings to put the puzzle together. Unless, of course, they made this before she was caught. :)

Completely coincidentally, I’m typing this from the Aldi at 88 and 26, waiting out a storm.

Re: Building an (almost) fully self-hosted, sandboxed, agentic software factory

#46

Post by NewsaHackO »

alexpotato wrote: post #41
ashu1461 wrote: post #27

In such systems, producing code is the easy part, verification is hard. Verification via test cases just feels like the same agent validating its own assumptions. Wondering what the actual verification loop looks like once you start taking these systems to production.

I just wrote a Twitter thread about how LLMs are like chainsaws [0] In short: - they are very powerful - beginners feel like they are just as good as experts - but they can cause a LOT of damage if you don't know what you are doing - experts will get way more done and beginners may cut off a limb or drop a tree on their car 0 - https://x.com/alexpotato/status/2090872063464509912?s=20

People have been making analogies like this since 2023, hardly worth a link to a random twitter thread.

Re: Building an (almost) fully self-hosted, sandboxed, agentic software factory

#47

Post by ReplaceMe »

ashu1461 wrote: post #27

In such systems, producing code is the easy part, verification is hard. Verification via test cases just feels like the same agent validating its own assumptions. Wondering what the actual verification loop looks like once you start taking these systems to production.

Maybe unpopular prediction:

I think vision models will come more into play for validating things. It’s the most like consciousness, and less like - as you put it an LLM validating its own assumptions.

It’s at least an independent way of analyzing the work (as glyphs and images).

Re: Building an (almost) fully self-hosted, sandboxed, agentic software factory

#48

Post by fabianlindfors »

ashu1461 wrote: post #27

In such systems, producing code is the easy part, verification is hard. Verification via test cases just feels like the same agent validating its own assumptions. Wondering what the actual verification loop looks like once you start taking these systems to production.

Agreed, we had agents writing standard integration/unit/e2e tests but turned out to be a ton of work reviewing just that and hard to wrap one’s head around as a developer.

We ended up building our own test platform to reduce the human bottleneck in the verification loop.

Takes the form of a hermetic test environment that run all our code and infra in a way that matches production. Our tests interact with it from the ”outside”, e.g. by driving a browser, a terminal or making API calls.

We still have coding agents writing those tests but the big difference is that we record the browser and the API calls and can review and verify through a UI without having to do any manual testing. It also serves as a regression test suite with E2E coverage, which is kind of a must to avoid coding agents making assumptions and breaking things.

Not a huge fan of the term ”software factory” but working like this I actually think it’s achievable.

Re: Building an (almost) fully self-hosted, sandboxed, agentic software factory

#49

Post by jagenabler2 »

tcdent wrote: post #24
copemaxxxing wrote: post #19

People who build software factory are in for a disappointing future. Spoken as someone who uses AI day to day for job and personal projects, I've came across a few super gnarly bugs already, ranging from frontend React apps (yes, believe it or not, frontend is far from solved) to embedded that Claude can't solve. Those bugs are feature breaker. My primary skillset is fullstack/frontend leaning, not embedded. Granted,…

What most people leave out of the conversation is the element of time. We are early. The technology continues to improve. Is it perfect now? Absolutely not. Will it continue to get better at a surprising pace? Absolutely yes. So, just adjust your framing from "we're there" to "we're getting there" and all of your concerns go away.

On the other end of it, people regularly leave out the "why" and "how". Why do software factories not work at scale right now? What are the pieces of building software that agents don't excel at? Is it possible that the 10% that it can't get right, it'll never get right? Is the technology actually improving in the direction that makes this a possible future or we at a local maxima?

I won't admit to knowing the answers, but more discussion needs to happen around the meta of software engineering before anyone can make conclusions about whether we're actually getting there.

My intuition tells me that there's too much nuanced complexity that goes unnoticed for automation to truly take place.

Re: Building an (almost) fully self-hosted, sandboxed, agentic software factory

#50

Post by copemaxxxing »

tcdent wrote: post #24
copemaxxxing wrote: post #19

People who build software factory are in for a disappointing future. Spoken as someone who uses AI day to day for job and personal projects, I've came across a few super gnarly bugs already, ranging from frontend React apps (yes, believe it or not, frontend is far from solved) to embedded that Claude can't solve. Those bugs are feature breaker. My primary skillset is fullstack/frontend leaning, not embedded. Granted,…

What most people leave out of the conversation is the element of time. We are early. The technology continues to improve. Is it perfect now? Absolutely not. Will it continue to get better at a surprising pace? Absolutely yes. So, just adjust your framing from "we're there" to "we're getting there" and all of your concerns go away.

Read "no silver bullet" article.
Post reply on HN