Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Agents need control flow, not more prompts

bsuh.bearblog.dev

Post reply on HN
Print view View original

211–220 of 348 posts

Re: Agents need control flow, not more prompts

#211

Post by deadbabe »

julianlam wrote: post #176
827a wrote: post #100

1000% agree. I am increasingly hesitant to believe Anthropic's continual war drum of "build for the capabilities of future models, they'll get better". We've got a QA agent that needs to run through, say, 200 markdown files of requirements in a browser session. Its a cool system that has really helped improve our team's efficiency. For the longest time we tried everything to get a prompt like the following working: "…

> This started breaking down after ~30 files. Sometimes it would miss a file. Sometimes it would triple-test a bundle of files and take 10 minutes instead of 3. An error in one file would convince it it needs to re-test four previous files, for no reason. It was very frustrating. Sorry, you thought a prompt was a suitable replacement for a testing suite?

If you are invested in AI stocks, this is the way. You are basically funneling money from software companies into your brokerage account. Keep going.

Re: Agents need control flow, not more prompts

#212

Post by BalinKing »

zapataband1 wrote: post #196
827a wrote: post #100

1000% agree. I am increasingly hesitant to believe Anthropic's continual war drum of "build for the capabilities of future models, they'll get better". We've got a QA agent that needs to run through, say, 200 markdown files of requirements in a browser session. Its a cool system that has really helped improve our team's efficiency. For the longest time we tried everything to get a prompt like the following working: "…

[flagged]

From the site guidelines (https://news.ycombinator.com/newsguidelines.html):

> Be kind. Don't be snarky. Converse curiously; don't cross-examine. Edit out swipes.

Re: Agents need control flow, not more prompts

#213

Post by [deleted] »

tharkun__ wrote: post #208
cheshire_cat wrote: post #205

Earlier quoted context omitted.

Wouldn't it be more efficient to convert the requirements these 200 markdown files into Playwright tests? You could still use an LLM to write and extend the tests, but running the tests would be deterministic and would use less resources.

This type of thing so much. AI is being pushed so much at work right now. For non-dev stuff even. The amount of things that people think are "awesome never seen this" is staggering. Just because you haven't seen file format X converted to file format Y before and now you asked the LLM to do it and it worked, doesn't mean you needed an LLM for it nor that it's remarkable. The LLM knew how to do it because it learned f…

[deleted]

Re: Agents need control flow, not more prompts

#216

Post by devin »

cogman10 wrote: post #141
DrewADesign wrote: post #109

Earlier quoted context omitted.

I used to assume they pushed people into the prompt-only workflows because you’re paying them for the tokens, and not paying them for the scaffolding you built. However, I think that they’re really worried about is that a person needs to design and implement that stuff… It throws a wet blanket on their insistence that this will replace entire people in entire workflows or even projects, and I just don’t buy it. I do…

> However, I think that they’re really worried about is that a person needs to design and implement that stuff… It throws a wet blanket on their insistence that this will replace entire people in entire workflows or even projects, and I just don’t buy it. I think you are on to something. But I also think this sort of system lends itself to not needing really good LLMs to do impressive things. I've noticed that the qu…

And then you realize that what you’re using the smaller models for is ALSO decomposable and part of it is just a few if statements, and then you realize that for this feature you don’t actually need or want a model because the performance, reliability, reproducibility are cheaper and better for you and your users.

Re: Agents need control flow, not more prompts

#217

Post by gofreddygo »

tharkun__ wrote: post #208
cheshire_cat wrote: post #205

Earlier quoted context omitted.

Wouldn't it be more efficient to convert the requirements these 200 markdown files into Playwright tests? You could still use an LLM to write and extend the tests, but running the tests would be deterministic and would use less resources.

This type of thing so much. AI is being pushed so much at work right now. For non-dev stuff even. The amount of things that people think are "awesome never seen this" is staggering. Just because you haven't seen file format X converted to file format Y before and now you asked the LLM to do it and it worked, doesn't mean you needed an LLM for it nor that it's remarkable. The LLM knew how to do it because it learned f…

> It's magic

you'll be surprised with how many people are comfortable attributing something they do not understand to Magic.

more than anything, ai let people who couldn't and wouldn't bother to learn to write simple code, to side step ones who can and build solutions to scratch their own itch. that too faster.

now human behavior kicks in, and they don't want to hand control back into the hands of people who can code to solve problems.

put this together and you have a good model to understand the AI sales pitch... Its magic

like all magic, its but a trick.

Re: Agents need control flow, not more prompts

#218

Post by jimbokun »

devin wrote: post #216
cogman10 wrote: post #141

Earlier quoted context omitted.

> However, I think that they’re really worried about is that a person needs to design and implement that stuff… It throws a wet blanket on their insistence that this will replace entire people in entire workflows or even projects, and I just don’t buy it. I think you are on to something. But I also think this sort of system lends itself to not needing really good LLMs to do impressive things. I've noticed that the qu…

And then you realize that what you’re using the smaller models for is ALSO decomposable and part of it is just a few if statements, and then you realize that for this feature you don’t actually need or want a model because the performance, reliability, reproducibility are cheaper and better for you and your users.

So you have the model write the if statements and put itself out of a job.

Re: Agents need control flow, not more prompts

#219

Post by try-working »

Nizoss wrote: post #133
try-working wrote: post #108

that's why you need a recursive workflow that creates its own artifacts per step that can later be used for verification.

Sounds interesting, can you elaborate on your thinking? Got me curious.

how do you verify the work that was just done in the current stage? verify against the output artifacts from the previous stages. for example, if you have a requirement doc, then you can analyse the codebase for current state, and store as a doc. then generate the implementation plan based on the delta between requirements and current state. after implementation, create an implementation summary doc. to verify the implementation in the next stage, compare the implementation summary against the implementation plan, the previous codebase analysis and the original requirements doc, as well as codebase diffs.

so, every stage outputs a source of truth for that stage, which can be used by later stages for verification, alone or together with other artifacts. if you want to read more, here's the recursive-mode development workflow I built: https://recursive-mode.dev/introduction

Re: Agents need control flow, not more prompts

#220

Post by nickstinemates »

This is why we built swamp[1].

Swamp teaches your Agent to build and execute repeatable workflows, makes all the data they produce searchable, and enables your team to collaborate.

We also build swamp and swamp club using swamp. You can see that process in the lab[2]. This combines all of the creativity of the LLM for the parts that matter, while providing deterministic outcomes for the parts you need to be deterministic.

1: https://swamp.club

2: https://swamp.club/lab

Post reply on HN