Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Harness engineering: Leveraging Codex in an agent-first world

openai.com

Post reply on HN
Print view View original

201–210 of 222 posts

Re: Harness engineering: Leveraging Codex in an agent-first world

#201

Post by shepherdjerred »

geoffbp wrote: post #143
shepherdjerred wrote: post #34

This mirrors exactly what I have been doing. - Give Claude/Codex a way to verify its own work (browser, smoke tests, e2e tests, high-fidelity local environment) - Keep all context (issue tracking, docs, ideas, plans, worklogs) in-repo ( https://github.com/shepherdjerred/monorepo/tree/main/package... ) - Give Claude/Codex access to observability (Grafana, Prometheus, Tempo, PagerDuty) - Have Claude/Codex follow good e…

I like the idea of saving the work done into files - helps to prevent the llm from redoing the same work. Maybe one day instead of code in a repo it will just be a list of prompts.

Yes, this was a huge help for me. For example I would have a difficult bug that requires a few sessions/deployments to truly close out.

With the worklog, it can easily see "oh I've worked on something similar before"

Re: Harness engineering: Leveraging Codex in an agent-first world

#202

Post by zbrock »

DenisM wrote: post #192
zbrock wrote: post #170

Hello! I’m one of the three engineers who write this piece. Happy to answer questions.

Fantastic job! Can you share what type of project that was? On the spectrum from a database engine to cat picture sharing web site (very high demand for correctness vs very lax).

This was primarily an Electron app with some small hosted backend services

Re: Harness engineering: Leveraging Codex in an agent-first world

#203

Post by zbrock »

pramodbiligiri wrote: post #190
zbrock wrote: post #170

Hello! I’m one of the three engineers who write this piece. Happy to answer questions.

Have you been satisfied with the quality of code generated by the model? Or did you have to tweak some rule file or skill to improve it? Or is human-readable code not even a goal at this point?

We spent a lot of time tweaking skills, doc files, and prompts. I’d say that was our primary activity as engineers. Our job became tweaking the harness every time we got code or results we didn’t like. Eventually we were pretty happy with most agent runs, but we were always happy to just throw out ones that didn’t meet our standards. I think more than half didn’t.

Re: Harness engineering: Leveraging Codex in an agent-first world

#204

Post by zbrock »

getnormality wrote: post #191
zbrock wrote: post #172

Earlier quoted context omitted.

At the time we wrote the article we hadn’t released the product and weren’t ready to talk about it. It was an internal prototype that looked very much like the current Codex app.

So, did this internal prototype ultimately end up being used to create/influence a real product, e.g. Codex app?

Yep!

Re: Harness engineering: Leveraging Codex in an agent-first world

#205

Post by whateveracct »

andai wrote: post #28
bronny1989 wrote: post #24

why do you have “weeks” to ship what would take “months”?

> We intentionally chose this constraint so we would build what was necessary to increase engineering velocity by orders of magnitude.

holy wow. so they really are deluded and are believing fallacies from the mythical man month

Re: Harness engineering: Leveraging Codex in an agent-first world

#206

Post by whateveracct »

rfw300 wrote: post #9

I understand that the’ve written zero lines of code for this application, but would it kill them to write a few lines of the blog post by hand? Forcing readers to wade through an unceasing string of LLM clichés demonstrates the opposite of the point you’re trying to make—that the consumers of your work are worse off because you exercised no human judgment in creating it.

these AI companies are high in their own supply

and/or they are really trying to shift norms where AI is required to exist (which is good for their financials)

Re: Harness engineering: Leveraging Codex in an agent-first world

#207

Post by QuercusMax »

Aperocky wrote: post #5
bko wrote: post #2

> We had weeks to ship what ended up being a million lines of code... Five months later, the repository contains on the order of a million lines of code across application logic, infrastructure, tooling, documentation, and internal developer utilities. Over that period, roughly 1,500 pull requests have been opened and merged with a small team of just three engineers driving Codex. This translates to an average throug…

> ended up being a million lines of code This almost reeks of "I've never cleaned up our code base because there is too much code, and didn't even bother having agents/LLM cleaning them up". You almost never need a million lines of code - this includes your software, infra, testing and operational tools. You didn't ship the linux kernel in 3 weeks and you know it. The code is already speghetti and it achieve the basi…

I ported/rewrote a million-LOC medical imaging workstation app over the course of 2 years with a team of 6. We had a full feature matrix with an extensive manual testing plan from previous work.

I suspect there's insane amounts of duplication.

Re: Harness engineering: Leveraging Codex in an agent-first world

#208

Post by swyx »

pramodbiligiri wrote: post #148
swyx wrote: post #109

Earlier quoted context omitted.

time of day also matters

(OP here) The truth is much simpler. I mailed the mods and they bumped it up in the second chance pool [1] :) 1. https://news.ycombinator.com/item?id=26998309

ha ok but why do that for a post you didnt write?

Re: Harness engineering: Leveraging Codex in an agent-first world

#209

Post by swyx »

pramodbiligiri wrote: post #187
swyx wrote: post #108

we interviewed Ryan here: https://www.latent.space/p/harness-eng and he gave a talk version of it in london: https://www.youtube.com/watch?v=am_oeAoUhew

This is a great interview! I had to switch from the transcript to watching it though - the transcript had many mistakes.

ya sorry we upgraded our thing after this

Re: Harness engineering: Leveraging Codex in an agent-first world

#210

Post by dang »

varenc wrote: post #19

digression: It's interesting this was submitted to HN over 15 times since it was published in February: https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu... But this is the only submission that's had any traction. Since the content is nearly the same for all submissions, it highlights how getting to the front page can be a bit random. (Though this is the only one that capitalized 'Leveraged' so maybe that'…

Someone emailed us about it and we ended up putting it in the SCP (https://news.ycombinator.com/item?id=26998308).

I'd say this case is a failure mode for HN for sure. Falling through the cracks 15 times in a row just sucks. Not sure what the best fix would be.

Post reply on HN