[flagged]
Harness engineering: Leveraging Codex in an agent-first world
141–150 of 222 posts
Re: Harness engineering: Leveraging Codex in an agent-first world
#142Earlier quoted context omitted.
I don’t think it’s solvable. And I think Anthropic etc know it. LLMs can only reconstitute things in its training data and they are so hungry they can’t do a good job in long lived codebase full of complexity and novelty. There’s never going to be enough similar code on the open internet.
> LLMs can only reconstitute things in its training data Such as a 4D raytracing engine in Metal? Or integrating APIs for features first released months after their knowledge cut-off date? LLMs have shown an ability to transfer "knowledge" and capabilities across domains, languages, and use-cases outside their training data. Case in point: GPT-2 "learning" to translate English to French and vice versa despite non-Eng…
Was this in the GPT2 paper?
Re: Harness engineering: Leveraging Codex in an agent-first world
#143This mirrors exactly what I have been doing. - Give Claude/Codex a way to verify its own work (browser, smoke tests, e2e tests, high-fidelity local environment) - Keep all context (issue tracking, docs, ideas, plans, worklogs) in-repo ( https://github.com/shepherdjerred/monorepo/tree/main/package... ) - Give Claude/Codex access to observability (Grafana, Prometheus, Tempo, PagerDuty) - Have Claude/Codex follow good e…
I like the idea of saving the work done into files - helps to prevent the llm from redoing the same work. Maybe one day instead of code in a repo it will just be a list of prompts.
Re: Harness engineering: Leveraging Codex in an agent-first world
#144Given that we can code at 10x speed for at least half a year, one would expect to see at least some pieces of machine-created software with 5 years' worth of equivalent human engineering work.
Anyone know some?
Re: Harness engineering: Leveraging Codex in an agent-first world
#145Earlier quoted context omitted.
That's a big ask. This kind of harness usually contains plenty of proprietary insights about their business. And also, nowadays, a good harness is a major competitive advantage.
Good thing I wasn't asking you. Also, a skill is not a harness.
Your hostile tone is unfortunate, especially since my post was actually friendly. I was just trying to point why it is very likely the OP won't give you what you're asking so you're not left confused if he ends up ghosting you.
Many people use the term harness to refer to the agent coding software (eg. Opencode, Claude Code...), i use this term more broadly to refer to the environment (set of skills, system prompts, constraints, memory, hooks etc...). What the OP is referring to is not just one giant skill. It's usually a comprehensive ecosystem of skills, bespoke tools to make certain agent tasks deterministic (eg localization), and so on.
I've seen someone post Github repos in this thread, these can be very useful especially if you use the same tech stack, but you won't reach the level of productivity reported by successful teams unless you invest substantial time to build your own harness. But the way to do so is to do it progressively : start with something simple to address the need you have on day 1 . And then, turn recurring prompts into skills, turn recurring coding patterns and coding style recommendations into guidelines, turn repetivive tasks for which the LLM tends to build a python script that it occasionally gets wrong into a deterministic tool documented in a skill etc...
And after a couple of days, weeks, and months, you'll have a very dependable harness giving you optimal productivity, without needing to invest weeks of work upfront or take the fun out of agent-assisted coding.
Hope this helps.
Re: Harness engineering: Leveraging Codex in an agent-first world
#146What I still can't understand is why is massive amount of code generated is a flex? I don't feel that software has gotten a lot better in past 3 years, only sloppier. It's surprising to me that people who know about reward hacking choose a simple objective like lines of code generated as a signal for quality. I'd argue you have to optimize for less lines generated as possible while secondary optimization should be re…
Well, to be fair, the amount of goalpost shifting that is going on is quite intense. AI not being able to work in a "serious" project, and being limited to "toy projects" has been a long standing critique. But also, bigger projects need some amount of loc written and it's a bit silly to pretend that this is not the case or a bad thing. So the answer to the question is roughly: Establishing that an agent can work in a…
Should we not be counting function points rather than LOC’s.
Lines of Code is a meaningless measure. It should also be easy to count function points using AI.
Re: Harness engineering: Leveraging Codex in an agent-first world
#147I wish these breathless blog posts would actually try to be more didactic. For example, actually doing a walkthrough of how to set up these allegedly super powered workflows and concrete demonstrations. I’m not an AI skeptic. Rather I’d don’t want to miss out on any actual super powers.
I have an example of a side-project [1] where I think I naturally applied the best practices described in this article. My goal was to see if it's possible to code an entire project using a single agent (Claude).
To do this, I "simply" asked the agent, every time it encountered an issue, how to resolve it, using a validation tool or script. I also asked it to code these tools during audits. As a result, I now have over 30+ rules [2] for validating their commits. It's working pretty well now.
[1] https://github.com/gildas-lormeau/rebuild-and-ruin (let the timer expire to see the "demo" mode)
[2] https://github.com/gildas-lormeau/rebuild-and-ruin/blob/a4c3...
Re: Harness engineering: Leveraging Codex in an agent-first world
#148digression: It's interesting this was submitted to HN over 15 times since it was published in February: https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu... But this is the only submission that's had any traction. Since the content is nearly the same for all submissions, it highlights how getting to the front page can be a bit random. (Though this is the only one that capitalized 'Leveraged' so maybe that'…
time of day also matters
(OP here) The truth is much simpler. I mailed the mods and they bumped it up in the second chance pool [1] :)
Re: Harness engineering: Leveraging Codex in an agent-first world
#149I find it so interesting "Agent legibility is the goal" picks up James C. Scott term (without defining it, so I assume that's what they mean) which is _not a good thing_. Legibility is a governance effort to box in life.
Legibility has a time and place, for example, global health data around pandemics.
Re: Harness engineering: Leveraging Codex in an agent-first world
#150Earlier quoted context omitted.
Well, to be fair, the amount of goalpost shifting that is going on is quite intense. AI not being able to work in a "serious" project, and being limited to "toy projects" has been a long standing critique. But also, bigger projects need some amount of loc written and it's a bit silly to pretend that this is not the case or a bad thing. So the answer to the question is roughly: Establishing that an agent can work in a…
Should we not be counting function points rather than LOC’s. Lines of Code is a meaningless measure. It should also be easy to count function points using AI.
I'd argue LoC isn't actually a meaningless measure, but people use it the wrong way. The same program with the same features but less LoC is more likely to have a proper design and architecture, and is most likely easier to change and maintain in the future. Of course, only if it's less LoC because of proper design, not because you've folded everything to one line.
So if anything, we should find a way to aim for as little lines of code as possible. If you have two agents, and one can build exactly the same program as another, but with half the LoC, then most likely the first agent is better at software engineering and particularly software design.
Of course, as the author of an experiment that investigated exactly this, I'm slightly biased. Cursor's browser had millions lines of code which sounded weird to me based on the features and functionality it had. Meanwhile, I built the same thing but actually thinking about the design with the agent and ended up with ~20K lines of code instead.