Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Harness engineering: Leveraging Codex in an agent-first world

openai.com

Post reply on HN
Print view View original

181–190 of 222 posts

Re: Harness engineering: Leveraging Codex in an agent-first world

#181

Post by torben-friis »

everforward wrote: post #169
stult wrote: post #84

Earlier quoted context omitted.

> - Do we have reasons to care about LOC in a world where we don't write code manually? What happens to token usage numbers when the codebase is significantly larger? Yes, at least to the extent that we care about context windows and tokens consumed by coding agents processing code that is ultimately irrelevant to their assigned task. Anecdotally, I've found keeping file sizes small has been important for agentic cod…

Seconded on smaller files. I feel like I tend to get better responses faster. A notable flaw here is that I’ve not tried large vs small files in a large codebase. Most of my experimentation there has been on personal projects where even a small file contains a significant part of the project. I could see degradation when it has to load 5 files to figure out how something works. Total LOC (tokens, really, literal line…

>Eg last I checked Anthropics one-shot performance on Clojure was about the same as Python or Go despite almost certainly being less represented in training data. The combination of density and simple primitives might be easier for an LLM to wrangle, ameliorating the impact of a less popular language.

There might be tons of confounding factors there. One that comes to mind is the quality of of data, it might perfectly be that the average clojure snippet is higher quality, due to the users demographics. Very few people start writing code with clojure, whether in college or during bootcamps.

Re: Harness engineering: Leveraging Codex in an agent-first world

#183

Post by Art9681 »

aleqs wrote: post #16
bko wrote: post #2

> We had weeks to ship what ended up being a million lines of code... Five months later, the repository contains on the order of a million lines of code across application logic, infrastructure, tooling, documentation, and internal developer utilities. Over that period, roughly 1,500 pull requests have been opened and merged with a small team of just three engineers driving Codex. This translates to an average throug…

> should expect maybe 5x faster cycle in major software apps To what end and what would that even look like though? Enshittifying everything at maximum speed? The apps/platforms I use regularly - GitHub, Spotify, Google maps (just to name a few), have gotten noticeably shittier in recent times.

Confirmation bias. The internet has complained about software updates decades before LLMs became ubiquitous. I made a career fixing human slop by domain experts.

We easily forget that the great majority of software engineering is fixing the mistakes of other highly capable software engineers.

It's just so easy to blame the machine instead of admitting no one here is an expert on anything and they count their hits and not misses. If they did, we would find the probability of making a mistake to be higher than a fronter coding agent.

It's a hard headed crowd and everyone, LLM pilled or not, suffers from the Dunning-Kruger. All of us.

Just look at the comments. Everyone is perfect when they do things themselves.

Re: Harness engineering: Leveraging Codex in an agent-first world

#184

Post by Eridrus »

murat124 wrote: post #17

The other day I came across to a video showing workers in a e-vape factory. They pick up a bunch of e-vapes from the conveyor belt (each has 6 e-vape think), stick in their mouth and vigorously vape all of them for about 5 seconds, then test the next bunch. Humans reviewing hundreds of lines of change in a PR written by AI is not very different.

You can do statistical testing of the e-vape line because you have a specific criteria and well defined tolerances that you can define on a per-sample basis and that the factory meets with some acceptable 9s of reliability.

PRs are not like this because a single bad PR can be catastrophic for your business in a way that a single bad e-vape cannot.

I would also argue that the current output from the AIs when sampled by software engineers regularly doesn't meet the bar of quality we want in our product, hence the need to review every PR and fix a substantial fraction.

If you can start to bound the impact of changes and the outputs begin to be generally acceptable unsupervised, such that all you're doing is double checking that nothing has regressed in the factory, then the sampling approach can work.

Re: Harness engineering: Leveraging Codex in an agent-first world

#185

Post by getnormality »

bko wrote: post #2

> We had weeks to ship what ended up being a million lines of code... Five months later, the repository contains on the order of a million lines of code across application logic, infrastructure, tooling, documentation, and internal developer utilities. Over that period, roughly 1,500 pull requests have been opened and merged with a small team of just three engineers driving Codex. This translates to an average throug…

We've known for decades that output metrics like LOC/day are very bad measures of real productivity in software. But they seem to be back in vogue in the age of AI, because AI is so good at maxing these useless metrics, and we need to show how impressive our AI is and how impressive our usage of AI is.

Re: Harness engineering: Leveraging Codex in an agent-first world

#186

Post by DenisM »

shepherdjerred wrote: post #46
para_parolu wrote: post #43

Earlier quoted context omitted.

Does it yield good results? I found that instead of docs it’s easier just to ask ai to read code. I feel like this is same as comments in code. Become outdated fast

I don't really use "docs" for documentation. I've prompted Claude/Codex to always write a "log" and save it in-repo to track what it did and why. I've found this to be really helpful, e.g. "you did this last week, and now some other thing is happening" or "you tried this approach before to solve alert X but it didn't work" -- except it can discover this itself. https://github.com/shepherdjerred/monorepo/tree/main/pac…

I find that preserving logs that contain errors will confuse future sessions even if the errors were corrected at the time. Do you have that problem?

Essentially preserving logs extends the context window with all related problems.

Re: Harness engineering: Leveraging Codex in an agent-first world

#187

Post by pramodbiligiri »

swyx wrote: post #108

we interviewed Ryan here: https://www.latent.space/p/harness-eng and he gave a talk version of it in london: https://www.youtube.com/watch?v=am_oeAoUhew

This is a great interview! I had to switch from the transcript to watching it though - the transcript had many mistakes.

Re: Harness engineering: Leveraging Codex in an agent-first world

#188

Post by shepherdjerred »

DenisM wrote: post #186
shepherdjerred wrote: post #46

Earlier quoted context omitted.

I don't really use "docs" for documentation. I've prompted Claude/Codex to always write a "log" and save it in-repo to track what it did and why. I've found this to be really helpful, e.g. "you did this last week, and now some other thing is happening" or "you tried this approach before to solve alert X but it didn't work" -- except it can discover this itself. https://github.com/shepherdjerred/monorepo/tree/main/pac…

I find that preserving logs that contain errors will confuse future sessions even if the errors were corrected at the time. Do you have that problem? Essentially preserving logs extends the context window with all related problems.

I haven’t actually noticed that, but I’m not sure why. Maybe because I specifically describe it to the agent as a work log rather than documentation? I’m not sure

Re: Harness engineering: Leveraging Codex in an agent-first world

#189

Post by shepherdjerred »

c0rruptbytes wrote: post #133
para_parolu wrote: post #43

Earlier quoted context omitted.

Does it yield good results? I found that instead of docs it’s easier just to ask ai to read code. I feel like this is same as comments in code. Become outdated fast

it does not result in great results left unattended, it’ll start creating slop or hardcoding solutions but overtime if you adjust your verification rubric, it’s not too bad, gets pretty good, if you do make it do TDD, it gets kinda crazy and you’ll have 2000-3000 tests after awhile, or on my common case, 6000-7000 lines of code in single files (i usually have a cron to audit files for decomposition and create tickets…

Static analysis can help here! Add CI checks for duplicated code or file length.

For test growth, maybe use a coverage tracker and remove redundant tests?

Re: Harness engineering: Leveraging Codex in an agent-first world

#190

Post by pramodbiligiri »

zbrock wrote: post #170

Hello! I’m one of the three engineers who write this piece. Happy to answer questions.

Have you been satisfied with the quality of code generated by the model? Or did you have to tweak some rule file or skill to improve it? Or is human-readable code not even a goal at this point?
Post reply on HN