Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Towards a harness that can do anything

eardatasci.github.io

Post reply on HN
Print view View original

61–70 of 121 posts

Re: Towards a harness that can do anything

#61

Post by jaggederest »

no_wizard wrote: post #46
brainless wrote: post #11

I kind of have a different idea of agents. I totally believe in a deterministic scaffold but I really think that an agent should be as deterministic as possible - the more code, the better. Think of a typical loop we may ask of Claude Code today (assume we are not using TDD): run some test suite with fail fast mode, diagnose if the failure is due to recent feature changes (pass reference to backend/frontend, github i…

In the age of agents, I feel like BDD is more important than ever since it describes the behavior, and if it does so well it’ll make it much easier for these tools to pick up correct behavior. Gherkin style tests also come to mind

The best part about LLMs is that, if you write gherkin style tests, you don't even need to automate them yet to get value, the AI can "manually" validate initially and then only codify them into deterministic execution after they've been nailed down. That way you have a bulk set of tests that run automatically, and a few candidate tests that are not automatically run yet, and the boundaries expand over time.

Re: Towards a harness that can do anything

#62

Post by all2 »

brainless wrote: post #11

I kind of have a different idea of agents. I totally believe in a deterministic scaffold but I really think that an agent should be as deterministic as possible - the more code, the better. Think of a typical loop we may ask of Claude Code today (assume we are not using TDD): run some test suite with fail fast mode, diagnose if the failure is due to recent feature changes (pass reference to backend/frontend, github i…

What you are describing is precisely my goal with my agent framework.

One of the meta-processes designed in is pushing automated processes, both defined and discovered, down as far as possible. "Down" here means as far towards the metal as reasonable. So automate the automatable stuff, and leave the LLMs to do stuff LLMs are actually good at.

A trivial example is 'handle this bugfix ticket'. Many actions in a bugfix are pre-defined, for example a git commit at the end of the ticket. So Maelstrom will, at the end of a bugfix workflow, will force a git commit from the LLM that did the implementation. The LLM never even sees the git command, it just fills in a JSON field with a commit summary, and the workflow handles the commit.

Re: Towards a harness that can do anything

#63

Post by all2 »

chuckadams wrote: post #51

Mapping agent concepts to the Unix environment sounds like a great idea in general, but I get off the train at adopting the FHS, which is an ancient relic that has no place in a green-field system. I don't know exactly what shape it should take, but something along the lines of Nix seems more appropriate. Or maybe Plan 9.

One of the ideas I had (and I'm still tinkering with) is 'agent as a linux user'. All of the perms/groups/access rights are already built in, so it would make sense to leverage these for agents running on the same host. Heck, there's even mail built in for async comms between agents.

Re: Towards a harness that can do anything

#66

Post by zby »

brainless wrote: post #11

I kind of have a different idea of agents. I totally believe in a deterministic scaffold but I really think that an agent should be as deterministic as possible - the more code, the better. Think of a typical loop we may ask of Claude Code today (assume we are not using TDD): run some test suite with fail fast mode, diagnose if the failure is due to recent feature changes (pass reference to backend/frontend, github i…

I believe there should be easy ways to move logic between prompts and code in a smooth ways. Moving from prompts to code is for getting deterministic, fast, well defined and cheap execution - the other way is when you want to quickly extend your system or when you want to relax a pre-condition (which is extending if you look into it deeply).

There are some inroads into this vision - but I haven't seen anything build directly for this (beside my own experiment).

I have some 'vibe noted' notes on this: https://zby.github.io/commonplace/notes/unified-calling-conv..., https://zby.github.io/commonplace/notes/rlm-tendril-and-llm-...

Re: Towards a harness that can do anything

#67

Post by simonreiff »

polynomial wrote: post #59
simonreiff wrote: post #21

Earlier quoted context omitted.

I did indeed read every word. And read the code. Don't be rude.

You have used too many words. You could not have written that many words yourself, so you must be posting slop. In fact, let's just go ahead and remove your comment which is obviously diminishing the quality of our otherwise impeccable conversations on hn. Sorry, I don't make the rules (That being said, "the most powerful AI agent file-editing tool in the world" is a bit of a stretch.)

lol fair and I appreciate the feedback. I'm overhauling tbe home page to include a demo and will definitely revisit the tagline as well. (Even if I still do believe it's true!)

Re: Towards a harness that can do anything

#68

Post by alexpotato »

brainless wrote: post #26
alexpotato wrote: post #16

Earlier quoted context omitted.

100% agree that the more deterministic code the better up to the limit where you need the LLM's ability to be non-deterministic to kick in. There is this ACM blog post called "Manual Work is a Bug" [0] that was originally written to help humans automate processes using code. I find it just as applicable today as when it was written. You and the LLM look at what has to be done and then figure out the scripts/tools to…

This is an interesting share, thanks. Yes, that is my mental model. Use coding agents to generate more "programs" (scripts) to automate everything. Have edge case handlers - and these handlers can develop/update the original scripts.

You could even have a decision tree in code where the leaf nodes are primarily "run this script with these params" but some of the leaves are "ask the LLM"
Post reply on HN