Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Harness Is the Thing

scott-fryxell.github.io

Post reply on HN
Print view View original

91–100 of 114 posts

Re: The Harness Is the Thing

#91

Post by throw10920 »

dmantis wrote: post #85
dingaling911 wrote: post #81

Earlier quoted context omitted.

This problem is easily solved. "Check that the blog looks the same in XYZ browsers". I have LLMs align stuff for me all the time because I'm too lazy. It takes the screenshot, changes the code, code auto reloads and boom, done.

Sure. However I have a strange feeling when I read all AI praises and how productive everyone is while seeing more and more broken things everywhere. Sometimes from the same very productive people. Otherwise we might have said that maybe broken stuff goes from AI-deniers. If everyone is doing the job of hundreds, extremely productive and everything is so easily fixed, why everything feels so slow and broken, even so…

> If everyone is doing the job of hundreds, extremely productive and everything is so easily fixed, why everything feels so slow and broken, even so basic things?

This doesn't make sense. There were slow and broken things before AI. This was true even when the things were made by hundreds or thousands of engineers.

If an organization doesn't care about making their app or site fast or correct, then unless they have a truly ludicrous amount of free manpower (more than is available now with AI, because review and architecture are still bottlenecks), it probably won't happen.

AI doesn't change this. AI doesn't change the priorities of an organization, it just changes how quickly and cheaply they can build. And every org has a point in their priority queue where things are no longer worth it to spend resources on.

If Venmo didn't care about making their login page work well with Firefox (which, in my experience, they don't) before AI, why would you expect it to be better with AI?

Re: The Harness Is the Thing

#92

Post by criley2 »

bluegatty wrote: post #86
criley2 wrote: post #58

Earlier quoted context omitted.

Sonnet 5 is the worst model of 2026. Literally just turn effort slider down on Opus, it's smarter, faster and cheaper than whatever Sonnet is. Beyond that, I find this whole plan and build thing to be a pointless waste of tokens. If your planner made a detailed enough plan, then the cost of executing that plan is a just one turn more of cached tokens, and minimal time. Meanwhile: switching agents, reloading context a…

Completely disagree (except the Sonnet bit, yes, it's degrading). "then the cost of executing that plan is a just one turn more of cached tokens, and minimal time." This is just not true at all. There's a huge gap between 'figured out the hard stuff' and 'rock solid'. Dependencies, integration, corner cases, docs, testing, unforeseen issues, a lot of back and forth auditing making sure things are really tight. Audits…

I'm sorry, but just because you achieve results you consider acceptable with this method doesn't mean everyone does.

I don't work where we can ship slop. I don't work where PRs can be merged based on what the agents say. I work where a human has to read and approve and own every single line of code. I work where the stakes are actually high, so the cost of not using the best tools in terms of human time are big. A single turn around in a PR costs more in human time than the difference between deepseek and fable in API costs.

So, when you admit "There's a huge gap between 'figured out the hard stuff' and 'rock solid'." but then claim that the cheapest/dumbest agent in your arsenal is your go-to for "rock solid", I have to question the quality of your results.

Personally, "using plan mode" is a very 2025 way of using these tools, and I wouldn't be surprised to see "plan mode" be removed from codex/claude code/et al.

Realistically, I'm using the best models to think about a domain and problem (Fable High+), and I'm using a cheap daily driver with an advisor pattern (Opus High + Fable) to iterate through POCs, and I'm using human review to guide design. None of that is "plan mode", it's actual engineering. Then we decompose the solution, we stack it, and we use only really strong agents to build, review and refine.

This obsession with cheap agents leads to low quality outcomes. "Rock solid" deserves the best tools, and the "plan" will never be good enough. I'm going to be sending fable xhigh and sol 56 xhigh et al at it in adversarial review, why the heck am I cheaping out on the actual implementation?

And finally: my time costs way more than any of this. Cheaper models are slower overall and when combined with re-work time, are dramatically slower. I'm costing my company hundreds in my time to save a few bucks on the API bills. Nonsense!

Re: The Harness Is the Thing

#93

Post by adamtulinius »

tokai wrote: post #84
andunie wrote: post #65

I don't understand why no one has tried to make a harness without full shell access yet. It would be so much safer.

That exist? "permission": { "bash": "deny" } Or something equivalent in any agentic editor of your choice.

What does that achieve if the agent can still run processes on the host system?

Re: The Harness Is the Thing

#94

Post by trollbridge »

andunie wrote: post #66
trollbridge wrote: post #60

Earlier quoted context omitted.

No, but the 3 of us can now accomplish a lot more than we used to. We spend around $450 a month on hosted inference plus capital / utility costs of around $100 a month in local inference. And our 2 non-technical staff are now busy designing apps, so when they hand off something to be productised, it’s far more complete than the old days of a few Figma drawings

Are you sure you are? Try laying some people off just to be sure.

I'm very un-motivated to do so since then I'd have more work to do, and I'd be making less money.

Re: The Harness Is the Thing

#95

Post by SuperCuber »

adamtulinius wrote: post #93
tokai wrote: post #84

Earlier quoted context omitted.

That exist? "permission": { "bash": "deny" } Or something equivalent in any agentic editor of your choice.

What does that achieve if the agent can still run processes on the host system?

what do you mean? the agent can't do it anymore if there's no bash permission

Re: The Harness Is the Thing

#96

Post by alexgoodhart »

noir_lord wrote: post #79
dmantis wrote: post #76

> Single developer projects can build to the caliber and consistency of large development teams. Yet the simple blog website static page saying that looks very weird and broken on the desktop firefox. How large should be a development team to make proper margins in 2026?

I was a little "what a weird nitpick" then I opened it on FF and yeah... that's awful . Normally I'd just switch into FF reader view when it's that badly done but that doesn't work either.

I looked at it on my out of date FF and it looks fine.

Re: The Harness Is the Thing

#97

Post by bluegatty »

criley2 wrote: post #92
bluegatty wrote: post #86

Earlier quoted context omitted.

Completely disagree (except the Sonnet bit, yes, it's degrading). "then the cost of executing that plan is a just one turn more of cached tokens, and minimal time." This is just not true at all. There's a huge gap between 'figured out the hard stuff' and 'rock solid'. Dependencies, integration, corner cases, docs, testing, unforeseen issues, a lot of back and forth auditing making sure things are really tight. Audits…

I'm sorry, but just because you achieve results you consider acceptable with this method doesn't mean everyone does. I don't work where we can ship slop. I don't work where PRs can be merged based on what the agents say. I work where a human has to read and approve and own every single line of code. I work where the stakes are actually high, so the cost of not using the best tools in terms of human time are big. A si…

Yours was the casual dismissal; and based on a misunderstanding of what can be achieved.

Based your arbitrary dismissal and unwillingness to even try to consider new patterns with which you may be unfamiliar - it may be difficult to communicate with you.

I have the advantage of 'certainty' because I have the evidence over many projects / team members.

We ship near perfect code.

In addition to the hints above, we do this at least in part by explicitly anchoring and testing requirements into several aspects of the code, and ensuring that known 'weak spots' are managed.

The 'planning process' ensures the requirements are mechanically anchored and integrated into tests, that 'proportional' documentation is applied, and that module, library and project level documentation is perfect (and mechanically validated where possible), which FYI is what solves most of 'context problems'. (That's another hint, if you have extremely good docs, you don't need to load vast amounts of code).

Yes - I hear you that 'time matters' and that 'the stakes are high' - consider that you may be talking to people where the stakes are just as high, or higher - but more specifically, this is not about 'saving tokens' or cost so much as it is using the right level of model for the task.

Use the best models for background research and planning, use mediocre models for execution, and mid-high for auditing - in other words 'use the right model for the right work' - and in a certain methodology, dumber models are appropriate.

FYI this saves you the ugly 'Fable' problem which many are encountering as it burns though Max plans. Don't 'automate' with Fable, it's the wrong model for that.

I could go on, but consider that there are actually ways of organizing projects and orchestration that work well.

Re: The Harness Is the Thing

#98

Post by bitexploder »

KronisLV wrote: post #78
AirMax98 wrote: post #3

Reading this really makes me wish that I had a slightly better workflow. I'm really soley dependent of Fable to the point that I don't use other models, and I've already sort of hit a point where I'm running into usage limits every week. I am really living on borrowed time — when Anthropic finally collapses their 50% usage increase at the end of August, I'll definitely be forced to switch my workflow. When that happe…

> when Anthropic finally collapses their 50% usage increase at the end of August, I'll definitely be forced to switch my workflow I cancelled my Anthropic subscription until they fix how their models write and it's no longer unbearably annoying and obnoxious. The concise output format is a step in the right direction but I need a few months away from them. Kimi and GLM models on Max reasoning feel pretty close to Fab…

I only like talking to Opus 4.6 in its default form. I have some pretty aggressive prompting strategies ensuring my language guidance rules are front and center and it really helps with Opus 4.8+, something went wrong with those models.

Also, OMP has a solid subagent model and I like it with some tweaking.

Re: The Harness Is the Thing

#99

Post by sho »

AirMax98 wrote: post #3

Reading this really makes me wish that I had a slightly better workflow. I'm really soley dependent of Fable to the point that I don't use other models, and I've already sort of hit a point where I'm running into usage limits every week. I am really living on borrowed time — when Anthropic finally collapses their 50% usage increase at the end of August, I'll definitely be forced to switch my workflow. When that happe…

If you find yourself wishing for more Fable at the subscription rates, just buy another 20x account? I'm currently on 3, spaced out on Sunday, Wednesday and Friday. Just as one is running out the next resets. Problem solved!

I feel the value I get far exceeds $600/m. It's a straight expected value calculation for me and I'm happy to pay. I wish they'd make it easier though - just sell me a 100x account for $1k/m and save the messing around.

Re: The Harness Is the Thing

#100

Post by liampulles »

One will lose the opportunity to develop domain understanding if they do not get into the weeds of thinking through the problem.

I use Claude plan mode to do relatively small changes and even then I find that if I actually try and think through the problem and solve it myself that I find good metaphors that will aid future work, and I will discover tangential issues which are then important to look at.

Post reply on HN