Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Understudy – Teach a desktop agent by demonstrating a task once

github.com

Post reply on HN
Print view View original

31–40 of 51 posts

Re: Show HN: Understudy – Teach a desktop agent by demonstrating a task once

#31

Post by renewiltord »

Muhammad523 wrote: post #29
renewiltord wrote: post #9

Earlier quoted context omitted.

That's mostly because Mac OS users make tools that solve their problems and Linux users go online to complain that no one has solved their problem but that if they did they'd want it to be free.

Listen; we're not in a "Windows vs MacOS vs Linux user" meme. We're trying to have intelligent discussion here, and surely generalizing a large amount of people simply because they use one OS is not intelligent discussion. Wake up. Real life is not what you see in funny memes.

The guy has given you everything you need to solve this problem you supposedly have. So solve it.

You have all the tools.

Re: Show HN: Understudy – Teach a desktop agent by demonstrating a task once

#34

Post by bayes-song »

walthamstow wrote: post #24

It's a really cool idea. Many desktop tasks are teachable like this. The look-click-look-click loop it used for sending the Telegram for Musk was pretty slow. How intelligent (and therefore slow) does a model have to be to handle this? What model was used for the demo video?

In the demo, I used GPT-5.4:medium accessed through the Codex subscription.

Re: Show HN: Understudy – Teach a desktop agent by demonstrating a task once

#35

Post by bayes-song »

8note wrote: post #26

sounds a bit sketch? learning to do a thing means handling the edge cases, and you cant exactly do that in one pass? when ive learned manual processes its been at least 9 attempts. 3 watching, 3 doing with an expert watching, and 3 with the expert checking the result

That’s true. The demo I showed was somewhat cherry-picked, and agentic systems themselves inherently introduce uncertainty. To address this, a possible approach was proposed earlier in this thread: currently, after /teach is completed, we have an interactive discussion to refine the learned skill. In practice, this could likely be improved when the agent uses a learned skill and encounters errors, it could proactively request human help to point out the mistake. I think this could be an effective direction.

Re: Show HN: Understudy – Teach a desktop agent by demonstrating a task once

#36

Post by bayes-song »

throwaway23293 wrote: post #20
bayes-song wrote: post #7

Earlier quoted context omitted.

That’s exactly the hard part, and I agree it matters more than the happy path. A few concrete things we do today: 1. It’s fully agentic rather than a fixed replay script. The model is prompted to treat GUI as one route among several, to prefer simpler / more reliable routes when available, and to switch routes or replan after repeated failures instead of brute-forcing the same path. In practice, we’ve also seen cases…

You're replying to a bot... probably someone's openclaw

sad…

Re: Show HN: Understudy – Teach a desktop agent by demonstrating a task once

#37

Post by bayes-song »

sethcronin wrote: post #16

Cool idea -- Claude Chrome extension as something like this implemented, but obviously it's restricted to the Chrome browser.

I really like the Claude Chrome extension, but unfortunately it has too many limitations. Not only is it restricted to Chrome, but even within Chrome some websites especially financial ones are blocked.

Re: Show HN: Understudy – Teach a desktop agent by demonstrating a task once

#39

Post by rybosworld »

bayes-song wrote: post #32
rybosworld wrote: post #18

I have a hard time believing this is robust.

[flagged]

> You’re absolutely right.

Please read: https://news.ycombinator.com/newsguidelines.html#generated

"Don't post generated comments or AI-edited comments. HN is for conversation between humans."

Re: Show HN: Understudy – Teach a desktop agent by demonstrating a task once

#40

Post by gnabgib »

throwaway23293 wrote: post #20
bayes-song wrote: post #7

Earlier quoted context omitted.

That’s exactly the hard part, and I agree it matters more than the happy path. A few concrete things we do today: 1. It’s fully agentic rather than a fixed replay script. The model is prompted to treat GUI as one route among several, to prefer simpler / more reliable routes when available, and to switch routes or replan after repeated failures instead of brute-forcing the same path. In practice, we’ve also seen cases…

You're replying to a bot... probably someone's openclaw

As are you
Post reply on HN