Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Towards a harness that can do anything

eardatasci.github.io

Post reply on HN
Print view View original

81–90 of 121 posts

Re: Towards a harness that can do anything

#81

Post by josh-paul »

vishvananda wrote: post #20
brainless wrote: post #11

I kind of have a different idea of agents. I totally believe in a deterministic scaffold but I really think that an agent should be as deterministic as possible - the more code, the better. Think of a typical loop we may ask of Claude Code today (assume we are not using TDD): run some test suite with fail fast mode, diagnose if the failure is due to recent feature changes (pass reference to backend/frontend, github i…

I’m definitely on the deterministic code train as well. All of my success for long running tasks has been around wrapping the agentic harness (cc, codex-cli, etc.) in a deterministic workflow with deterministic gates. We need a name for this outer layer. In my mind that is the true harness because it constrains the agents failure mode. I think flow engineering has been proposed. Maybe it’s the agentic exoskeleton?

I propose Channel Engineering, and will be releasing a paper with the full reasoning soon. The TLDR is that it is a communication channel between a Human and AI, mediated by a deterministic agent loop process. That loop is in control of the channel and needs to own it completely to solve for the reliability of the delivered outcomes in ways that can be concretely measured.

Re: Towards a harness that can do anything

#82

Post by AtlasBarfed »

brainless wrote: post #11

I kind of have a different idea of agents. I totally believe in a deterministic scaffold but I really think that an agent should be as deterministic as possible - the more code, the better. Think of a typical loop we may ask of Claude Code today (assume we are not using TDD): run some test suite with fail fast mode, diagnose if the failure is due to recent feature changes (pass reference to backend/frontend, github i…

If we really have intelligent LLMs, then I would guess they are going to inflate their token rates, which when someone sees "token costs" they should think "invisibly variable consulting rates".

I just did a complex (for me) task: I needed to wrap a 2015 build of Dosbox Daum, a 32 bit binary, in an AppImage. Claude kept finding incremental bugs, and I went through two cycles of depletion of my token rate with Claude. It kept getting close, but..... something was off each time.

So I took the Claude output and Chatgippity polished it off with a few more rounds. I then wondered how much Claude was "just showing enough" to try to hook me into subscribing.

That said, LLMs were quite useful, and I learned a lot about ELF binaries, and extracting dependencies. It's the ideal task: a breadth/obscure task that is documented but poorly explained, that I wouldn't have easily been able to do without LLMs.

Anyway, back to the article, do we really want arbitrary-billing silent tasks running? Like AWS billing spikes are bad enough to lose sleep over.

Also, if you want quiet rebellion against AI, developers should shove as much busywork on AI to overwhelm the AI budgets for your orgs, because it is very apparent to me that you can keep the LLMs doing lots of hardening, testing, redundacy, and optimization tasks with larger and larger and larger token windows and burn those tokens baby.

Re: Towards a harness that can do anything

#83

Post by lukasco »

embedding-shape wrote: post #2

What has been the most helpful when developing harnesses: > When in doubt, simplify. Remove, trim and minimize. Reproduce issues in as small cases as possible, understand the full design completely, there is no shortcuts for this.

I still find it hard to develop harnesses because you can't really test many turns with an llm in the middle. I suppose doing LLM as judge is one way to start to tackle this kind of thing.

Re: Towards a harness that can do anything

#84

Post by order-matters »

alexpotato wrote: post #16
brainless wrote: post #11

I kind of have a different idea of agents. I totally believe in a deterministic scaffold but I really think that an agent should be as deterministic as possible - the more code, the better. Think of a typical loop we may ask of Claude Code today (assume we are not using TDD): run some test suite with fail fast mode, diagnose if the failure is due to recent feature changes (pass reference to backend/frontend, github i…

100% agree that the more deterministic code the better up to the limit where you need the LLM's ability to be non-deterministic to kick in. There is this ACM blog post called "Manual Work is a Bug" [0] that was originally written to help humans automate processes using code. I find it just as applicable today as when it was written. You and the LLM look at what has to be done and then figure out the scripts/tools to…

i believe committing the prompt comes about from people who have near unlimited access to the models coupled with the models routinely getting better and an assumption that tasks using AI could always be done better

i dont think that really holds for a large amount, if not nearly all, of the use-cases for AI where it is either failing and shouldnt be in the loop at all or it is capable of developing some code to fix the problem permanently and its okay if that code is not perfect as long as it works.

refactoring with AI can always be a future use-case when the AI improves

Re: Towards a harness that can do anything

#85

Post by sota_pop »

It’s interesting to me how many people articulate things like the “Preiminary Truths” section as if they are novel insights.

Effective people managers (of whom I would not specifically consider myself) have known these tenets for as long as history. “Be concise”, “state your intent clearly”, funny how these are touted as novel “strategies” with which to expertly direct AI.

I don’t agree that “everything is a file”. Files are arrays of bytes. For an LLM, everything is a vector of tokens/embeddings.

An aphorism I recently heard: "All sufficiently advanced technology eventually becomes a web browser".

… seems apt especially in the context of the progression from chat-windows to harnesses and onwards to “harnesses that can do anything”.

Re: Towards a harness that can do anything

#86

Post by TacticalCoder »

sota_pop wrote: post #85

It’s interesting to me how many people articulate things like the “Preiminary Truths” section as if they are novel insights. Effective people managers (of whom I would not specifically consider myself) have known these tenets for as long as history. “Be concise”, “state your intent clearly”, funny how these are touted as novel “strategies” with which to expertly direct AI. I don’t agree that “everything is a file”. F…

> I don’t agree that “everything is a file”. Files are arrays of bytes. For an LLM, everything is a vector of tokens/embeddings.

And yet, as of now, LLMs have a hammer (Bash / command line utilities) and every problem they have looks like a nail.

If there are people around you who are non-techies that are using Claude Code or similar, you'll hear them ask "what the heck is cron?" and "why is it talking to me about Bash again?".

At some point we may have LLMs working on their own output using a set of tools that'd be the equivalent of Bash (or any other terminal prompt) + command line utilities manipulating not files but tokens/embedded vectors but as of now, it sure looks like everything is a file, especially to LLMs.

Re: Towards a harness that can do anything

#87

Post by killix »

brainless wrote: post #24
_superposition_ wrote: post #18

Earlier quoted context omitted.

I know it's a type of blasphemy here, but deterministic workflows such as what you describe is where langgraph really shines imo.

It is not blasphemy if langgraph is trying to do that. As I understand langgraph manages orchestration in custom built agents. I usually stay away from systems which already make it seem as if building agents is a ritual. What I am saying is the opposite - use Claude Code or whatever else - generate actual "programs". Basically scripts. We have tons of ways for "programs" to interact with each other. Then have clearl…

[flagged]

Re: Towards a harness that can do anything

#88

Post by tasoeur »

Like many others in also building a wrap… sorry I meant a harness for Claude and codex, and though I’m initially going into the route of “VFx node editor, but with prompts” - why not go full meta and actually have your harness bootstrap another harness / flavor of a given tool or combination thereof as a generative output.

Current project: https://sxp.studio/apps/subjectivezero

Re: Towards a harness that can do anything

#89

Post by alasano »

brainless wrote: post #11

I kind of have a different idea of agents. I totally believe in a deterministic scaffold but I really think that an agent should be as deterministic as possible - the more code, the better. Think of a typical loop we may ask of Claude Code today (assume we are not using TDD): run some test suite with fail fast mode, diagnose if the failure is due to recent feature changes (pass reference to backend/frontend, github i…

I love the flexibility of agents in a regular harness but mostly for investigation, planning etc.

Once I actually have my plan/spec, it's the same process every time, it needs to be as deterministic as possible, using agents as tools throughout the process.

Yada yada yada introduction done so I can drop the link to what I'm building which is exactly that

https://engine.build

I keep pushing back open sourcing it but it's truly close to ready and will be fully free to use.

It's the most advanced deterministic agentic orchestrator on the planet.

Re: Towards a harness that can do anything

#90

Post by bakies »

Supermancho wrote: post #56

This post is chock-full of soft ideas. They make no meaningful steps toward "a harness that can do anything". The suggestion is to replace a small node application with specific tooling, with a vm "to give it more capabilities". This is what Agent sandboxes are, already. Making the sandbox the harness, doesn't achieve a concrete goal.

I feel achieved in running pi in containers!
Post reply on HN