161–170 of 296 posts
I call it the Intent-Implement-Quality problem. Software factories can implement anything given a one-liner requirement. That one-liner requirement can be a complete app/product, epic, feature, bug, design change or refactoring. But these one liner requirements are requirements coming from a human who has an intent or requirement or direction for the product to evolve in mind. Can Software factories manufacture inten…
If you built a real factory, you’d basically never want it to be dark. You’d want a culture of getting wrenches out to inspect the cars being built. You’d want to care about small details. Not because we need to build cars by hand. But because looking at the real product (cars, code, etc) is the best way to make the factory better. It’s the best way to know what problems aren’t being measured, what processes need to…
> Xiaomi has an 860,000 square foot factory in Changping, Beijing capable of manufacturing 10 million smartphones a year across 11 fully automated production lines.
> ASE Group, a Taiwanese company that handles outsourced semiconductor packaging and testing have 56 lights out factories.
It has always been the goal to make factories go dark.
I call it the Intent-Implement-Quality problem. Software factories can implement anything given a one-liner requirement. That one-liner requirement can be a complete app/product, epic, feature, bug, design change or refactoring. But these one liner requirements are requirements coming from a human who has an intent or requirement or direction for the product to evolve in mind. Can Software factories manufacture inten…
If you’re working on an app or software that has few users, no revenue or minimal revenue, tolerance to bugs is higher and just another Claude prompt away, Software factories are a perfect fit. Most personal software or hobby software or 0-1 yet-to-find-PMF startups belong in this category. You can even take a stand that you’ll never look at the code and just ship. This is a perfect equilibrium for a Software factory…
I would rather claim that for my personal/hobby software, I apply a much higher quality standard than for the software that I write at work.
Believe me: I would love to apply the same high (or even higher) standard at work, too.
The problem rather is that it is a very common situation that bosses actively prohibit me from rocking the boat too much. If I make (I think typically good) suggestions, I often get very stern reactions.
At work, elegance is not celebrated - just implement the tasks of users who don't care about quality or elegance.
Earlier quoted context omitted.
Gating integration behind code review is futile. I (and many other engineers) already automated it. My agent responds to review requests and reviews as me. Company policies enforcing human code review are futile. I think all these platforms chasing code review are doomed. My LLM doesn't need any of this tooling. We should be reviewing the actual working software. Systems that make it easy and instant to demo any prop…
>I (and many other engineers) already automated it. My agent responds to review requests and reviews as me. Company policies enforcing human code review are futile. This is also known as being a terrible engineer. If a company enforces human review and someone deliberately tries to circumvent this with an LLM, I'd fire that person in an instant The reason for human code review is: 1. So *you* understand what's going…
SOX requires code review by a second party. If the reviews are automated, the development process isn't in compliance and that can get expensive in a hurry.
> Verifying quality is orders of magnitude harder than "did the tests pass"
Agree that agentic grading is the future here. Cognition's Frontier Code is probably the best large public benchmark at this. You attribute agent quality issues to RLVR's binary pass/fail, however I wouldn't be surprised if labs are already supplementing that with rubrics as rewards to train more 'tasteful' models like Fable.
What can a practitioner do? I think there's promise in turning the optimization machine to the harness itself - building out a representative dataset of tasks on your repo, grading agent quality on them across various configurations, and optimizing [AGENTS.md / SKILLS.md / workflow / model / harness / tools] on that signal. High quality grading is still very hard, but it's more tractable at smaller, repo-level scale, and you can afford slower, more expensive verification for each task. You only need it to be right about your codebase's standards.
> In fact, it's not hard to imagine that if a model could reliably tell good code from bad, it might have written the good version to begin with
Pushing back slightly - detecting slop and discriminating quality is easier than generating it (why code review is so effective), and why grading is viable at repo eval scale even if it's much harder at RL scale.
Everyone is flying blind. For example, I am genuinely interested in trying HumanLayer, but would likely want some harder evidence (beyond anecdotes) that it's actually making my agents more effective before rolling out to an enterprise team.
I'm building this harness optimization loop @ https://stet.sh if curious
Earlier quoted context omitted.
Did you even read further on? The section right after that addresses the step changes in current frontier models https://github.com/humanlayer/advanced-context-engineering-f...
As I mentioned explicitly in the comment you're replying to: yes, I did read that section. Did you not bother to read my whole comment before telling me to read more?
This part of your comment is confusing.
The article talks about agents past this period. Yet it seems like you’re saying this article isn’t relevant because it doesn’t?
Earlier quoted context omitted.
If you’re working on an app or software that has few users, no revenue or minimal revenue, tolerance to bugs is higher and just another Claude prompt away, Software factories are a perfect fit. Most personal software or hobby software or 0-1 yet-to-find-PMF startups belong in this category. You can even take a stand that you’ll never look at the code and just ship. This is a perfect equilibrium for a Software factory…
> If you’re working on an app or software that has few users, no revenue or minimal revenue, tolerance to bugs is higher and just another Claude prompt away, Software factories are a perfect fit. Most personal software or hobby software or 0-1 yet-to-find-PMF startups belong in this category. I would rather claim that for my personal/hobby software, I apply a much higher quality standard than for the software that I…
Now, if you've got a really good model you can get much further with dead reckoning compared to a worse model. But a model is not reality so no matter how good it is, so without feedback eventually you still run into this problem no matter what.
I imagine that's a lot like what goes on in these agentic loops.
Earlier quoted context omitted.
PR review sucks because engineers often don't optimise their code authoring for it. When they do it's a joy. PR review of agent generated code sucks because agents are, in my experience, very poor at authoring for review. This makes sense because the review process is not represented in the resulting code artifact which is what they're trained on. Agents are always producing much larger changes than should be in a si…
Agents are also controllable to a degree. How agents author PRs is now an engineering problem.
A good PR tells a story to the reader that leaves them confident in the result. That means knowing the reader and how they think, and that is something LLMs are very bad at.