Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Harness engineering: Leveraging Codex in an agent-first world

openai.com

Post reply on HN
Print view View original

191–200 of 222 posts

Re: Harness engineering: Leveraging Codex in an agent-first world

#191

Post by getnormality »

zbrock wrote: post #172
h4ny wrote: post #87

I'm not an AI skeptic but I'm skeptical of the intent of this article. It makes great claims about agent-first engineering and tries to make a real case based on a real product, with real users, and a real team that's been growing — all without even saying what was built or showing it, just like every other AI hype article.

At the time we wrote the article we hadn’t released the product and weren’t ready to talk about it. It was an internal prototype that looked very much like the current Codex app.

So, did this internal prototype ultimately end up being used to create/influence a real product, e.g. Codex app?

Re: Harness engineering: Leveraging Codex in an agent-first world

#192

Post by DenisM »

zbrock wrote: post #170

Hello! I’m one of the three engineers who write this piece. Happy to answer questions.

Fantastic job!

Can you share what type of project that was? On the spectrum from a database engine to cat picture sharing web site (very high demand for correctness vs very lax).

Re: Harness engineering: Leveraging Codex in an agent-first world

#194

Post by e12e »

zbrock wrote: post #170

Hello! I’m one of the three engineers who write this piece. Happy to answer questions.

Interesting write up!

Have you been able to extract libraries or tools from this project yet? If so how was that experience?

That is, do you see yourself releasing a metric harness, or sub-projects that are equivalent of ActiveRecord, zod, or similar open source tooling that frequently originate in a large in-house project - and then is exported out as a stand-alone toll, utility, library or framework?

Because while ai can reimplement minor tools, it's utility entirely depends on the existence of solid tools, libraries and frameworks.

Re: Harness engineering: Leveraging Codex in an agent-first world

#195

Post by kmetan »

drivebyhooting wrote: post #62

I wish these breathless blog posts would actually try to be more didactic. For example, actually doing a walkthrough of how to set up these allegedly super powered workflows and concrete demonstrations. I’m not an AI skeptic. Rather I’d don’t want to miss out on any actual super powers.

Instead of reading articles like this one end to end, I ask AI to read them in detail and prepare a new harness for me. The important part is not to do this in a single prompt, but to first create a detailed plan and let the model think deeply about each aspect. This approach lets me build the new harness without the missing didactic information you mentioned.

Basically, I am moving from “I build products without writing or reading the code” to “I build products without writing or reading the harness.”

Once the new implementation harness is prepared, I start it, but I keep the original session open. In that original session, “we” monitor the implementation harness from the outside: how effective it is, where the bottlenecks are, what breaks down, and what could be improved. From time to time, the monitoring session suggests changes to the implementation harness. We apply those changes, restart the harness, and monitor it again.

The overall approach is not to spend X hours understanding an article like this in detail, because another similar article will appear in 3 weeks. Instead, I take immediate action, learn on the fly, and replace the harness when a better pattern emerges. And yes, I still have to spend X hours on setting up, monitoring and fine tuning the new harness, but at the end I have the latest fancy "thing" working for me.

Re: Harness engineering: Leveraging Codex in an agent-first world

#197

Post by vividfrier »

para_parolu wrote: post #43
shepherdjerred wrote: post #34

This mirrors exactly what I have been doing. - Give Claude/Codex a way to verify its own work (browser, smoke tests, e2e tests, high-fidelity local environment) - Keep all context (issue tracking, docs, ideas, plans, worklogs) in-repo ( https://github.com/shepherdjerred/monorepo/tree/main/package... ) - Give Claude/Codex access to observability (Grafana, Prometheus, Tempo, PagerDuty) - Have Claude/Codex follow good e…

Does it yield good results? I found that instead of docs it’s easier just to ask ai to read code. I feel like this is same as comments in code. Become outdated fast

[dead]

Re: Harness engineering: Leveraging Codex in an agent-first world

#199

Post by YZF »

yurimo wrote: post #114
YZF wrote: post #73

Earlier quoted context omitted.

Lines of code has always been a terrible metric. But all else being equal it is a measure. If all else is not equal, which is usually the case, then it's not. A lot of the focus has been on AI recently. Three years ago we didn't have software where a non-software engineer can describe what they want in English and get working (-ish) software generated by other software? Is that not "software has gotten a lot better"?…

I personally don't view coding agents making software as "software gotten better" you are comparing a tool and the end result, these are two different things. Agent you use going down and your product going down mean two different things to you customers. I will not deny that we made incredible progress in coding and hell, even design over the past 3.5 years, this technology is here to stay. That being said while I a…

My point was "look at what computers(software) can do today vs. 3 years ago" - for everyone. You are saying that software that the arborist can have ChatGPT write to help it draw the garden isn't the same quality as a team of software engineers would write manually (I think). GitHub is (mostly?) software for software developers. Most software in the world is software for random people. Nobody(tm) cares about the quality of GitHub. The "has software (computers) gotten better" has to be measured from the perspective of the consumer, not the perspective of the software engineer, and nobody using AI is going to tell you "computers now are worse/can do less than they were 3 years ago". At least that's my thesis.

If AI today can make you more productive that's already progress. If it can't then maybe it makes other people more productive.

Re: Harness engineering: Leveraging Codex in an agent-first world

#200

Post by drivebyhooting »

kmetan wrote: post #195
drivebyhooting wrote: post #62

I wish these breathless blog posts would actually try to be more didactic. For example, actually doing a walkthrough of how to set up these allegedly super powered workflows and concrete demonstrations. I’m not an AI skeptic. Rather I’d don’t want to miss out on any actual super powers.

Instead of reading articles like this one end to end, I ask AI to read them in detail and prepare a new harness for me. The important part is not to do this in a single prompt, but to first create a detailed plan and let the model think deeply about each aspect. This approach lets me build the new harness without the missing didactic information you mentioned. Basically, I am moving from “I build products without wri…

I love this idea! Thanks
Post reply on HN