Yep thats true pre grounding is very much worth it, if you just feed the agent a quick environment brief upront instead of making it spam tool calls to figure out where it is, you save a lot of tokens.
Harness engineering: Leveraging Codex in an agent-first world
161–170 of 222 posts
Re: Harness engineering: Leveraging Codex in an agent-first world
#162> We had weeks to ship what ended up being a million lines of code... Five months later, the repository contains on the order of a million lines of code across application logic, infrastructure, tooling, documentation, and internal developer utilities. Over that period, roughly 1,500 pull requests have been opened and merged with a small team of just three engineers driving Codex. This translates to an average throug…
[flagged]
Re: Harness engineering: Leveraging Codex in an agent-first world
#163> We tried this early on — used ChatGPT as "project manager" to set up the entire harness before writing any code. After a week it produced 140+ docs of rules, architecture, frameworks. Zero lines of code. When we finally brought in another tool to review, the verdict was: "a perfectly secure empty safe." The harness was immaculate. There was just nothing inside it.
>
> Harness matters, but if you're not shipping code alongside it, you're just writing fiction.
Re: Harness engineering: Leveraging Codex in an agent-first world
#164Earlier quoted context omitted.
>I suspect it's not seen as a problem by providers because more lines generated means more tokens used and hence more billing put out on customers. I have also grown skeptical of token usage in order to run up my bill! But since I feel like it takes me MORE effort to write LESS lines of code myself, I'd expect a quick and dirty AI-generated solution to be MORE lines of code and cost LESS to generate than a concise/el…
The latest frontier models will write code better than you and more elegant, with less lines of code, in 100th of the time, with full test coverage. Hand coding is like writing out assembly/machine code rather than using a compiler.
This hasn’t been my experience. State of the art models available to the public still do all sorts of bandaids and bad hacks. Putting code where it doesn’t belong. Stapling types onto variable (in TypeScript) when abstractions/types already exist to use. I use it to generate code, but still have to review every line and have corrections/steering basically every time.
Maybe you have access to some other model?
Re: Harness engineering: Leveraging Codex in an agent-first world
#165Earlier quoted context omitted.
> LLMs can only reconstitute things in its training data Such as a 4D raytracing engine in Metal? Or integrating APIs for features first released months after their knowledge cut-off date? LLMs have shown an ability to transfer "knowledge" and capabilities across domains, languages, and use-cases outside their training data. Case in point: GPT-2 "learning" to translate English to French and vice versa despite non-Eng…
Was this in the GPT2 paper?
In "Language Models are Unsupervised Multitask Learners"[0]. Not sure whether it’s "the" GPT-2 paper.
3.7 Translation
> Performance on this task was surprising to us, since we deliberately removed non-English webpages from WebText as a filtering step. In order to con- firm this, we ran a byte-level language detector2 on WebText which detected only 10MB of data in the French language […]
[0]: https://cdn.openai.com/better-language-models/language_model...
Re: Harness engineering: Leveraging Codex in an agent-first world
#166[deleted]
Re: Harness engineering: Leveraging Codex in an agent-first world
#167Earlier quoted context omitted.
>a large portion of your skillset is about to become completely worthless I'm not convinced of that. I watched a video of an architect using AI to create architectural drawings. It became very clear to me that he has a lot of skills and terminology that helped him produce something very specific, in a few minutes. I've been working on some home improvement stuff including a studio/shed and I've struggled to produce e…
How do you keep your skills if you no longer engage in the activity that keeps them sharp?
See, I just don't get that angle.
Just because I'm not typing "strcat(); strcpy(); sprintf()" doesn't mean I'm not thinking about problems. I'm still doing critical thinking all over my stack, and I don't see that going away. I'm just doing different thinking.
There are people who think, and AI just isn't going to change that. There are people who don't think, and they've existed long before AI. Back in the 90s when I worked at the phone company, man, I worked with some people who didn't do a lick of work (along with some really sharp people).
Re: Harness engineering: Leveraging Codex in an agent-first world
#168Earlier quoted context omitted.
>a large portion of your skillset is about to become completely worthless I'm not convinced of that. I watched a video of an architect using AI to create architectural drawings. It became very clear to me that he has a lot of skills and terminology that helped him produce something very specific, in a few minutes. I've been working on some home improvement stuff including a studio/shed and I've struggled to produce e…
This is the case now - I can explain to the AI that I want to re-factor a component to support different implementations using a strategy pattern, and I can get a similar outcome to what I would have written, just implemented a bit faster. My expertise brings value. But that's not what this specific article is describing. The world this article is describing is one where you describe the business requirements, and yo…
>where you describe the business requirements
That right there is what I'm talking about: that architect would write the requirements for a building way different than I would.
Re: Harness engineering: Leveraging Codex in an agent-first world
#169Earlier quoted context omitted.
Here's a fun one: firefox lists its current count at about 2.5M LOC, from roughly 1M commits during the years. You end up with about 3 lines added per commit, which is not ridiculous when you consider that most would be editions rather than full additions. Here, we have 1500 PRs and 1M LOC, which is about 650 added LOC per PR. Remember, not 650 lines total in the PR, but +650 balance after additions-removals. Fun que…
> - Do we have reasons to care about LOC in a world where we don't write code manually? What happens to token usage numbers when the codebase is significantly larger? Yes, at least to the extent that we care about context windows and tokens consumed by coding agents processing code that is ultimately irrelevant to their assigned task. Anecdotally, I've found keeping file sizes small has been important for agentic cod…
Seconded on smaller files. I feel like I tend to get better responses faster.
A notable flaw here is that I’ve not tried large vs small files in a large codebase. Most of my experimentation there has been on personal projects where even a small file contains a significant part of the project. I could see degradation when it has to load 5 files to figure out how something works.
Total LOC (tokens, really, literal lines probably don’t matter) is interesting as a factor. That might go some way towards explaining why LLMs are weirdly good at Clojure.
Eg last I checked Anthropics one-shot performance on Clojure was about the same as Python or Go despite almost certainly being less represented in training data. The combination of density and simple primitives might be easier for an LLM to wrangle, ameliorating the impact of a less popular language.
Re: Harness engineering: Leveraging Codex in an agent-first world
#170Hello! I’m one of the three engineers who write this piece. Happy to answer questions.