Well if these guys computer action model works as they intended (ground up video trained model) https://news.ycombinator.com/item?id=47125014 maybe this benchmark will be conquered far faster then expected
Nice, their training recipe seems unique.
PA bench: Evaluating web agents on real world personal assistant workflows
11–13 of 13 posts
Re: PA bench: Evaluating web agents on real world personal assistant workflows
#12I just don't get why would you would want an agent to use the browser to do these mundane things (check email, work with calendar etc), when you can simply give it a few tools, and save maybe six gazillion tokens per task?
Adversarial interoperability.
Re: PA bench: Evaluating web agents on real world personal assistant workflows
#13I just don't get why would you would want an agent to use the browser to do these mundane things (check email, work with calendar etc), when you can simply give it a few tools, and save maybe six gazillion tokens per task?
Using existing enterprise apps probably - this solution is scalable for the vendor and it's easier to sell using existing software as-is than to start out by writing new custom tools.
Yes, I can see the usecase for legacy desktop apps etc, but the web? it's a DOM. WebMCP coming now too, no need for screenshotting or DOM querying then either..