Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Harness Is the Thing

scott-fryxell.github.io

Post reply on HN
Print view View original

61–70 of 91 posts

Re: The Harness Is the Thing

#61

Post by andai »

flyinglizard wrote: post #40
andai wrote: post #37

I've been very happy with Luna but my approach is "many bite sized edits" for which models basically hit saturation a year ago. (I also tried the "let a massive model make massive changes" approach and am still psychologically recovering from the experience. The codebase may never recover!) Also, Luna and DSV4 Flash seem to be on par now except Luna is faster and cheaper?

FWIW I run a clinical analysis backend and directly compared Luna with DSv4 Flash 0731 - DS was a bit ahead, and cheaper even considering it used 40% more tokens. The exposure to Deepseek made me question the valuation house of cards built on SOTA providers. There are more companies producing competitive and useful models than there are companies producing jet engines for airliners, and not for the lack of trying. Ch…

You reminded me of this recent post, which I found very interesting:

Why jet engines aren't made in China

https://news.ycombinator.com/item?id=48740971

That's a very interesting point of comparison though. I wonder why that should be the case? Why is it so much harder to make a jet engine than a language model?

Maybe with LLMs the iteration times are lower? Or there's more public information about technique? Or are jet engines just intrinsically a harder problem?

Re: The Harness Is the Thing

#62

Post by andai »

esalman wrote: post #23

What I've learned in last week is that a harness is basically a while loop. In each iteration you make an LLM call, perform some work (e.g. tool call), augment the prompt (append or compact etc.)- not necessarily in that other- and continue. Until an end condition is satisfied. Then you break out.

It's two, actually:

https://minimal-agent.com/

This example has one, but if you add a second one around their example code, and take user input, and there's a basic harness already.

Re: The Harness Is the Thing

#63

Post by apercu »

ChrisMarshallNY wrote: post #44
all2 wrote: post #4

Earlier quoted context omitted.

I've been using GPT-5.6 Luna for everything at this point. It is more than enough to do all the work I need to do. You have to lead it by the nose, but if you know where you're going it works really, really well.

I’ve just been using it, with a chat interface. Basically, as a “consultant.” For me, and my projects, it’s been great. It’s made an enormous difference. I guess my workflow may seem “quaint,” to many folks, here, but the end results speak for themselves. I suspect that one vocation that could get heavily impacted by AI, is the consulting business. That’s where many experienced people go, as they reach their career p…

The thing about consulting is that someone needs to verify the output, know what questions to ask in the first place, and provide a throat for the client to choke in the event of an issue. Also, professional insurance. I could be wrong but in the worlds I live in professionally, accountability is still a thing.

Re: The Harness Is the Thing

#64

Post by ChrisMarshallNY »

apercu wrote: post #63
ChrisMarshallNY wrote: post #44

Earlier quoted context omitted.

I’ve just been using it, with a chat interface. Basically, as a “consultant.” For me, and my projects, it’s been great. It’s made an enormous difference. I guess my workflow may seem “quaint,” to many folks, here, but the end results speak for themselves. I suspect that one vocation that could get heavily impacted by AI, is the consulting business. That’s where many experienced people go, as they reach their career p…

The thing about consulting is that someone needs to verify the output, know what questions to ask in the first place, and provide a throat for the client to choke in the event of an issue. Also, professional insurance. I could be wrong but in the worlds I live in professionally, accountability is still a thing.

Yup. There's some places that we'll need that accountability. I suspect that this may be filled by folks that act as "LLM brokers," using AI in the background, while dealing with the legalities, in the foreground.

But in my case, it wasn't nearly so exotic. The LLM helped me to do a much better job, preparing the App Store presentation, Web support, privacy policies, budget prognostication, and app glossary.

I have just had an extremely complex app, pass App Review, in record time (from going into review, to approval). No niggles or bounces at all.

I'm thrilled.

Re: The Harness Is the Thing

#66

Post by andunie »

trollbridge wrote: post #60
throw1234567891 wrote: post #54

Earlier quoted context omitted.

Have you laid those people off? You still have people on the payroll. If each one of them uses a $500 subscription, you are now spending $42k/month, not $40k.

No, but the 3 of us can now accomplish a lot more than we used to. We spend around $450 a month on hosted inference plus capital / utility costs of around $100 a month in local inference. And our 2 non-technical staff are now busy designing apps, so when they hand off something to be productised, it’s far more complete than the old days of a few Figma drawings

Are you sure you are?

Try laying some people off just to be sure.

Re: The Harness Is the Thing

#67

Post by permalac »

AirMax98 wrote: post #3

Reading this really makes me wish that I had a slightly better workflow. I'm really soley dependent of Fable to the point that I don't use other models, and I've already sort of hit a point where I'm running into usage limits every week. I am really living on borrowed time — when Anthropic finally collapses their 50% usage increase at the end of August, I'll definitely be forced to switch my workflow. When that happe…

Us fable to help you with the transition.

Re: The Harness Is the Thing

#68

Post by skohan »

andunie wrote: post #65

I don't understand why no one has tried to make a harness without full shell access yet. It would be so much safer.

Is that the harness' job? It seems to me the best place for sandboxing is at the OS level (i.e. running the harness inside a container with correct access configured).

Re: The Harness Is the Thing

#70

Post by adverbly »

skohan wrote: post #68
andunie wrote: post #65

I don't understand why no one has tried to make a harness without full shell access yet. It would be so much safer.

Is that the harness' job? It seems to me the best place for sandboxing is at the OS level (i.e. running the harness inside a container with correct access configured).

I would argue that it is the harness's job... Certainly for any harness which is targeting an end user.

They don't sell cars without seat belts, and it is the car manufacturer who has to do it.

Also, small nitpick but technically a container doesn't give full isolation compared to something like a VM.

Post reply on HN