Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Towards a harness that can do anything

eardatasci.github.io

Post reply on HN
Print view View original

31–40 of 121 posts

Re: Towards a harness that can do anything

#31

Post by brainless »

ardatasci wrote: post #22
brainless wrote: post #11

I kind of have a different idea of agents. I totally believe in a deterministic scaffold but I really think that an agent should be as deterministic as possible - the more code, the better. Think of a typical loop we may ask of Claude Code today (assume we are not using TDD): run some test suite with fail fast mode, diagnose if the failure is due to recent feature changes (pass reference to backend/frontend, github i…

I don't think we're necessarily in disagreement here: I agree that determinism should be taken as far as possible, but once we zoom out of the software engineering world I think things tend to get a lot less easy to automate if that makes sense.

Not necessarily. I have been building client projects for the last few months only using coding agents. I use way more existing tools to handle a lot more of our digital footprint than I used to before: pdf, images, excel, ocr and many more.

As coding agents have accelerated my work, I just build tons of tooling around existing software. Or in rare cases build new ones. If we zoom out of software engineering, we will still be in the realm of files - text or binary. That does not change.

The question is - do we let agents run the tools or the "programs" call the LLMs. The OS is the new agent, but not the same sense of "agent". I want LLMs to be lightly sprinkled in a future "agent" OS, not the other way around.

Re: Towards a harness that can do anything

#32

Post by brainless »

visarga wrote: post #29
brainless wrote: post #11

I kind of have a different idea of agents. I totally believe in a deterministic scaffold but I really think that an agent should be as deterministic as possible - the more code, the better. Think of a typical loop we may ask of Claude Code today (assume we are not using TDD): run some test suite with fail fast mode, diagnose if the failure is due to recent feature changes (pass reference to backend/frontend, github i…

One of my policies for agentic coding is to spend much effort in developing tests, coded tests not LLM based vibes. My projects have around 1:1 LOC between code and tests. Tests are like skin, when the skin is pricked it hurts, agents need to feel pain too. OP's idea "everything is a text file" is good and I use it too. My plans are saved as task.md files, numbered and named. Work items are checkboxes inside the file…

Yes, I agree with this. I am not focusing on tests as much and I think that is a big mistake. Agents need to immediately understand something is off.

Re: Towards a harness that can do anything

#33

Post by bcjordan »

skybrian wrote: post #27

“That can do anything” gives me Zombo.com vibes. I would be more interested if it seemed to do particular things well.

Feel like there's now a lot of modern software that can do particular things well but there is new promise in finding universal abstractions that can ideally run recursively / safely run any software at their leaves. (Unix + tooling is most of the way there ofc)

I feel like Docker Compose / K8S / VM / Dagger.io layers are close but can't quite always recurse flexibly and aren't always simple to run with. Networking / devices / auth are often awkward choke points

Re: Towards a harness that can do anything

#35

Post by _superposition_ »

brainless wrote: post #24
_superposition_ wrote: post #18

Earlier quoted context omitted.

I know it's a type of blasphemy here, but deterministic workflows such as what you describe is where langgraph really shines imo.

It is not blasphemy if langgraph is trying to do that. As I understand langgraph manages orchestration in custom built agents. I usually stay away from systems which already make it seem as if building agents is a ritual. What I am saying is the opposite - use Claude Code or whatever else - generate actual "programs". Basically scripts. We have tons of ways for "programs" to interact with each other. Then have clearl…

I get what your saying, however we don't need to be stuck with the Claude code harness. You may find this interesting.

https://www.langchain.com/blog/tuning-the-harness-not-the-mo...

Re: Towards a harness that can do anything

#36

Post by ambicapter »

vishvananda wrote: post #20
brainless wrote: post #11

I kind of have a different idea of agents. I totally believe in a deterministic scaffold but I really think that an agent should be as deterministic as possible - the more code, the better. Think of a typical loop we may ask of Claude Code today (assume we are not using TDD): run some test suite with fail fast mode, diagnose if the failure is due to recent feature changes (pass reference to backend/frontend, github i…

I’m definitely on the deterministic code train as well. All of my success for long running tasks has been around wrapping the agentic harness (cc, codex-cli, etc.) in a deterministic workflow with deterministic gates. We need a name for this outer layer. In my mind that is the true harness because it constrains the agents failure mode. I think flow engineering has been proposed. Maybe it’s the agentic exoskeleton?

Why do you need a different name?

Re: Towards a harness that can do anything

#38

Post by ChicagoDave »

I think domain specific harnesses are already surpassing generic harnesses. I also think software development is its own domain.

My harness is a Claude Code plugin with its own brainstorming, adr, and planning skills with associated review and interview skills. Behavioral testing related to acceptance criteria is built in. Everything in my harness is gated to prevent ratholes.

I recently inflated a docker container to execute a set of work with Claude in unsafe mode and immediately saw problems with everything it was doing…and then I realized I had not installed my harness.

Running Claude without an engineering harness is like driving a car without brakes or a steering wheel.

Re: Towards a harness that can do anything

#39

Post by vishvananda »

ambicapter wrote: post #36
vishvananda wrote: post #20

Earlier quoted context omitted.

I’m definitely on the deterministic code train as well. All of my success for long running tasks has been around wrapping the agentic harness (cc, codex-cli, etc.) in a deterministic workflow with deterministic gates. We need a name for this outer layer. In my mind that is the true harness because it constrains the agents failure mode. I think flow engineering has been proposed. Maybe it’s the agentic exoskeleton?

Why do you need a different name?

From harness? Because people expect a squishy set of things from a harness that is different from what I end up building. I end up with a rigid internal structure that the harness uses in-turn (tests with clear error messages, tools, etc.) and a matched rigid external structure that drives the turn tracking progress and deterministically handles the overall progress. You could call that whole thing a harness but that makes the definition muddy and hard to talk about. So scaffold or skeleton seems more appropriate. The harness constrains the agent. The matched endoskeleton and exoskeleton gives it structure.

Re: Towards a harness that can do anything

#40

Post by everforward »

brainless wrote: post #24
_superposition_ wrote: post #18

Earlier quoted context omitted.

I know it's a type of blasphemy here, but deterministic workflows such as what you describe is where langgraph really shines imo.

It is not blasphemy if langgraph is trying to do that. As I understand langgraph manages orchestration in custom built agents. I usually stay away from systems which already make it seem as if building agents is a ritual. What I am saying is the opposite - use Claude Code or whatever else - generate actual "programs". Basically scripts. We have tons of ways for "programs" to interact with each other. Then have clearl…

You can drive agents via ACP these days, which I think is the layer you would want for what you’re talking about.

The API is basically what you see as a user of Claude Code or Pi or whatever. You can make new sessions, send messages to sessions, configure which MCPs get started, etc.

I’ve been poking at something similar to what you’re talking about via that route. My client prompts the agent to do a thing, and then afterwards launches deterministic things to check it which can either re-prompt the original session or start a new session.

Eg it automatically runs the tests afterwards, and will send a new prompt in the original chat to fix them if they fail. I also briefly poked at a security analyzer that gets changed files via git and makes a new session to check whether there are security issues and propose a fix that then gets sent to the original session.

If you want a circular loop where the LLM can adjust its own workflow while keeping it deterministic, you can let the agent modify the ACP client that drives it.

Post reply on HN