Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PA bench: Evaluating web agents on real world personal assistant workflows

vibrantlabs.com

Post reply on HN
Print view View original

1–10 of 13 posts

PA bench: Evaluating web agents on real world personal assistant workflows

#1

Post by shahules »

We’re the team at Vibrant Labs (W24). We’ve been building envs for browser agents and quickly realized that existing benchmarks in this space didn’t capture the primary failure modes we were seeing in production (which scaled up as the number of applications and horizon length increase).

We built PA Bench (Personal Assistant Benchmark) to evaluate frontier computer/web use models on their ability to handle multi-step workflows across simulated clones of Gmail and Calendar.

*What’s next:*

We’re currently scaling the dataset to 3+ tabs and are building more high-fidelity simulations for common enterprise workflows. We’d love to hear feedback on the benchmark and notes about what was/wasn’t surprising about the results.

Blog post: https://vibrantlabs.com/blog/pa-bench

PA bench: Evaluating web agents on real world personal assistant workflows
vibrantlabs.com

Re: PA bench: Evaluating web agents on real world personal assistant workflows

#4

Post by abhijithneil »

Is there a possible way computer use can be automated using multiple computer use agents from different providers, but also with some sort of routing setup so the best course of action can be chosen without hitting failures (for eg: permission issues in OpenAI could be rerouted to Gemini)

Re: PA bench: Evaluating web agents on real world personal assistant workflows

#5

Post by shahules »

abhijithneil wrote: post #4

Is there a possible way computer use can be automated using multiple computer use agents from different providers, but also with some sort of routing setup so the best course of action can be chosen without hitting failures (for eg: permission issues in OpenAI could be rerouted to Gemini)

There are few agents like browser-use, skyvern etc that may provide this capability.

Re: PA bench: Evaluating web agents on real world personal assistant workflows

#8

Post by shenberg »

mrorigo wrote: post #6

I just don't get why would you would want an agent to use the browser to do these mundane things (check email, work with calendar etc), when you can simply give it a few tools, and save maybe six gazillion tokens per task?

Using existing enterprise apps probably - this solution is scalable for the vendor and it's easier to sell using existing software as-is than to start out by writing new custom tools.

Re: PA bench: Evaluating web agents on real world personal assistant workflows

#10

Post by shahules »

shenberg wrote: post #8
mrorigo wrote: post #6

I just don't get why would you would want an agent to use the browser to do these mundane things (check email, work with calendar etc), when you can simply give it a few tools, and save maybe six gazillion tokens per task?

Using existing enterprise apps probably - this solution is scalable for the vendor and it's easier to sell using existing software as-is than to start out by writing new custom tools.

After doing few experiments, I think that having Agents work on browser for all tasks wouldn't be best due to many factors like token cost, safety, etc. But browser/computer can be a tool that the agent can be alongside MCPs to complete tasks that requires interaction with such modalities.
Post reply on HN