Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What Is a Harness?

earendil.com

Post reply on HN
Print view View original

161–170 of 200 posts

Re: What Is a Harness?

#161

Post by grey-area »

qarl2 wrote: post #118
grey-area wrote: post #104

Earlier quoted context omitted.

Guided by humans, code generators which have ingested the worlds’ code and can recognise and generate patterns can be useful tools. I wouldn’t personally qualify it as a wild success as we are early and there are significant downsides. That doesn’t make them intelligent agents which think independently.

Would you like to see some of my own examples of wild success? I have a system that entirely reverse engineers old arcade games. Creates semantic symbol mappings that were considered impossible just a couple years ago. Granted, it took me a couple weeks to build the system. From impossible to a couple weeks in just a couple years. Would you like to see it or continue to pretend these things don't exist? Your call. (I…

I’ve made my own arcade games with an LLM. It was fun though the code wasn’t great.

Mostly I felt they reproduced games from the training data, some with more examples available worked better than others.

I use them most days for work, and for that reason don’t trust them that much and certainly don’t worry or fantasise about AGI.

Re: What Is a Harness?

#163

Post by visarga »

YZF wrote: post #92
Syntaf wrote: post #62

I’ve been working on a harness for accounting agents at my job recently and it’s been a pretty interesting experience. We originally started with building a CLI tool so our LLMs could more easily interact with our platform. I cannot recommend enough the value of having an internal CLI. It’s both fun to build and extremely useful for agents. We paired this with skills initially, but found that the way folks built skil…

This is the same "tension" I keep seeing in my day job. Some people approach LLMs like they're writing code. They give a long list of detailed instructions for specific scenarios. When I use LLMs I leave things as open as possible. I just give them the information they need and my ask. As you say frontier models are very good at figuring things out. Being too prescriptive is counterproductive, it over-constrains the…

> If you want to follow a process or a checklist you probably shouldn't use an LLM

I like to externalize tasks as markdown files with checklists, they are still planned by agents but I can pass the plan around to judge agents and fix some errors before implementing.

I also have the coding agents comment on each closed checklist item, so the same file becomes a log of what happened. This goes to the implementation judge. I can also switch agents anytime, or resume a task days later no problem.

I am avoiding internally provided tools for todo lists and planning because they do not leave the same artifact trail which makes judging with separate agents easy.

Re: What Is a Harness?

#164

Post by stymaar »

YZF wrote: post #92
Syntaf wrote: post #62

I’ve been working on a harness for accounting agents at my job recently and it’s been a pretty interesting experience. We originally started with building a CLI tool so our LLMs could more easily interact with our platform. I cannot recommend enough the value of having an internal CLI. It’s both fun to build and extremely useful for agents. We paired this with skills initially, but found that the way folks built skil…

This is the same "tension" I keep seeing in my day job. Some people approach LLMs like they're writing code. They give a long list of detailed instructions for specific scenarios. When I use LLMs I leave things as open as possible. I just give them the information they need and my ask. As you say frontier models are very good at figuring things out. Being too prescriptive is counterproductive, it over-constrains the…

The problem is that even Fable still make trivial yet high impact mistake when let on their own, and then you'd need to read the whole code to catch them…

Meanwhile they are very good at implementating an explicit algorithm that you feed it to them.

Re: What Is a Harness?

#165

Post by eliranlevi »

Syntaf wrote: post #62

I’ve been working on a harness for accounting agents at my job recently and it’s been a pretty interesting experience. We originally started with building a CLI tool so our LLMs could more easily interact with our platform. I cannot recommend enough the value of having an internal CLI. It’s both fun to build and extremely useful for agents. We paired this with skills initially, but found that the way folks built skil…

I’ve noticed it’s the performance that suffers when agents are paired with more than a single CLI and non-prescriptive skills. Since it seems to be out of its training data, anything non-trivial and the model just tries to brute force its way to a solution. Maybe it’s also about building them as self-improving, though I’ve been doing it manually for a CLI we don’t own.

It seems to be art at the moment.

Re: What Is a Harness?

#167

Post by rurban »

0x457 wrote: post #109
wonnage wrote: post #95

Earlier quoted context omitted.

That works for well trod paths, e.g “fix ci” works exceedingly well. “why app slow” obviously doesn’t work because the task is underspecified. But in order to properly specify you either need an experienced engineer who knows how to narrow the problem domain, or you have to provide some template instructions/output formats (e.g, skills) which will invariably never fit the problem perfectly

> . “why app slow” obviously doesn’t work because the task is underspecified. Not always. In my case LLM goes to grafana mcp, pulls metrics/traces/cpu profiles. Figures out what is slow and proposes a solution.

In my cases it always used linux perf to sample the calls, because that's the best tool for my jobs. Never had to tell it to use instrumentation.

Re: What Is a Harness?

#168

Post by rurban »

I have a completely different view on that. Maybe because I work with harnesses for decades already. The perl test suite is called harness.

It's a build and test system, which makes it easy to test and run all supported platform and config options, and app/lib features and regressions. Like a CI. Basically a CI is the best harness.

The agent is just a feedback loop between the model and the harness. Since the feedback loop must be fast, a real CI is too costly, and only serves as end verification. So a good harness enables fast tests for features per arch. With proper cleanup and parallelization.

Re: What Is a Harness?

#170

Post by throwawayqqq11 »

Am i the only one that does not trust the ootb harnesses?

My approach is a llama-server wrapped in a locked down systemd service and a goose client/harness with a from-scratch apparmor profile behind a filtered network interface.

Getting this running was alot of work and is certainly not perfect but with qwen3.8 and expected price hikes, i think it might be worth it.

Post reply on HN