Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Understudy – Teach a desktop agent by demonstrating a task once

github.com

Post reply on HN
Print view View original

11–20 of 51 posts

Re: Show HN: Understudy – Teach a desktop agent by demonstrating a task once

#11

Post by dec0dedab0de »

bayes-song wrote: post #7
wuweiaxin wrote: post #3

[flagged]

That’s exactly the hard part, and I agree it matters more than the happy path. A few concrete things we do today: 1. It’s fully agentic rather than a fixed replay script. The model is prompted to treat GUI as one route among several, to prefer simpler / more reliable routes when available, and to switch routes or replan after repeated failures instead of brute-forcing the same path. In practice, we’ve also seen cases…

What if you had it ask for another demonstration when things are different? or if it's different and taking more than X amount of time to figure out. Like an actual understudy would.

Re: Show HN: Understudy – Teach a desktop agent by demonstrating a task once

#12

Post by bayes-song »

dec0dedab0de wrote: post #11
bayes-song wrote: post #7

Earlier quoted context omitted.

That’s exactly the hard part, and I agree it matters more than the happy path. A few concrete things we do today: 1. It’s fully agentic rather than a fixed replay script. The model is prompted to treat GUI as one route among several, to prefer simpler / more reliable routes when available, and to switch routes or replan after repeated failures instead of brute-forcing the same path. In practice, we’ve also seen cases…

What if you had it ask for another demonstration when things are different? or if it's different and taking more than X amount of time to figure out. Like an actual understudy would.

That sounds like a good idea. During the use of a skill, if the agent finds something unclear, it could proactively ask the user for clarification and update the skill accordingly. This seems like a very worthwhile direction to explore.

In the current system, I have implemented a periodic sweep over all sessions to identify completed tasks, cluster those tasks, and summarize the different solution paths within each cluster to extract a common path and proactively add it as a new skill. However, so far this process only adds new skills and does not update existing ones. Updating skills based on this feedback loop seems like something worth pursuing.

Re: Show HN: Understudy – Teach a desktop agent by demonstrating a task once

#15

Post by rogerrogerr »

ghjv wrote: post #13
wuweiaxin wrote: post #3

[flagged]

Out of curiosity - were this and other comments from this account written by hand, or generated and posted by an agent on behalf of a human user?

Feels like an agent that has been told to use `--` instead of emdash.

Re: Show HN: Understudy – Teach a desktop agent by demonstrating a task once

#20

Post by throwaway23293 »

bayes-song wrote: post #7
wuweiaxin wrote: post #3

[flagged]

That’s exactly the hard part, and I agree it matters more than the happy path. A few concrete things we do today: 1. It’s fully agentic rather than a fixed replay script. The model is prompted to treat GUI as one route among several, to prefer simpler / more reliable routes when available, and to switch routes or replan after repeated failures instead of brute-forcing the same path. In practice, we’ve also seen cases…

You're replying to a bot... probably someone's openclaw
Post reply on HN