Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Launch HN: Browser Use (YC W25) – open-source web agents

github.com

Post reply on HN
Print view View original

91–100 of 107 posts

Re: Launch HN: Browser Use (YC W25) – open-source web agents

#91

Post by MagMueller »

p0deje wrote: post #90

Have you experimented with using text-only models and DOM/accessibility tree for interaction with a ? I'm currently working on the open-source test automation tool ( https://alumnium.ai ) and the accessibility tree w/o screenshots works pretty well as long as the website provides decent support for ARIA attributes or at least has proper HTML5 structure.

On most pages, we don't need vision, and the DOM alone is sufficient. We have not worked with the accessibility tree yet, but it's a great idea to include that. Do you have any great resources on where to get started?

Re: Launch HN: Browser Use (YC W25) – open-source web agents

#92

Post by MagMueller »

hakaneskici wrote: post #44

What is your overall vision and roadmap about automated testing for web apps by bringing value from AI into the process? When I worked on the accessibilityinsights.io team, dealing with inconsistent or complicated xPaths was also an issue. Is AI vision helping there much?

It could be useful to run a prompt/test once, get the xPaths, and rerun it deterministically. When it breaks, you know something is wrong, and the LLM could be used as a fallback to fix the script.

Re: Launch HN: Browser Use (YC W25) – open-source web agents

#94

Post by MagMueller »

echelon wrote: post #89

Gregor and Magnus, the only thing I want in the world is an agent that removes all of the bad from the web: - No more ads. No more banner ads. No more Google search ads. No more promoted stories. No more submarine ads. - No more spam. Low quality content is nuked. - No more clickbait. Inauthentic headlines from dubious sources are removed. - No more rage comments. Angry commenters are muted. I have enough to worry ab…

We see people replacing UIs and using browser-use to fill out the real UI. So there could be a world where everyone has their own UI, and you could have that filter option.

Furthermore, valid point: if Pepsi spends $1M on ads, why don't you get a piece of it if they pitch to you?

Re: Launch HN: Browser Use (YC W25) – open-source web agents

#95

Post by MagMueller »

alex1115alex wrote: post #57

Awesome job launching guys! We used Browser Use last week to order burgers from our smart glasses: https://x.com/caydengineer/status/1889835639316807980 One thing I'm hoping for is an increase in speed. Right now, the agent is slow for complex tasks, so we're still in an era where it might be better to codify popular tasks (eg: sending a WhatsApp message) instead of handling them with browser automation. Have yall lo…

One option could be for the main apps like WhatsApp to have defined custom actions, which are almost like an API to the service. I think the interplay between LLM and automation scripts will succeed here:

Agent call 1: Send WhatsApp message (to=Magnus, text=hi) Inside, you open WhatsApp and search for Magnus (without LLM)

Agent call 2: Select contact from all possible Magnus contacts Script 3: Type the message and click send

So in total, 2 calls - with Gemini, you could already achieve this in 10-15 seconds.

Re: Launch HN: Browser Use (YC W25) – open-source web agents

#96

Post by p0deje »

MagMueller wrote: post #91
p0deje wrote: post #90

Have you experimented with using text-only models and DOM/accessibility tree for interaction with a ? I'm currently working on the open-source test automation tool ( https://alumnium.ai ) and the accessibility tree w/o screenshots works pretty well as long as the website provides decent support for ARIA attributes or at least has proper HTML5 structure.

On most pages, we don't need vision, and the DOM alone is sufficient. We have not worked with the accessibility tree yet, but it's a great idea to include that. Do you have any great resources on where to get started?

> On most pages, we don't need vision, and the DOM alone is sufficient.

I misunderstood looking at demo videos, it seemed like you constantly update elements with borders/IDs so I assumed that's what is then passed to vision.

> Do you have any great resources on where to get started?

A great place to start is https://chromium.googlesource.com/chromium/src/+/main/docs/a....

Re: Launch HN: Browser Use (YC W25) – open-source web agents

#97

Post by arjunchint »

jerpint wrote: post #83
arjunchint wrote: post #8

Have you inspected or thought through the security of your open source library? You are using debugger tools such as CDP, launching playwright without a sandbox, and guiding users to launch Chrome in debugger mode to connect to browser-use on their main browser. The debugging tools you use have active exploits that Google doesn't fix because they are supposed to be for debugging and not for production/general use. Th…

If they’re launching a cloud-based service, doesn’t this effectively remove the risk of running it locally ?

Their key offering is an open source solution that you can run on your own laptop and Chrome browser, but their approach to doing this presents a huge security risk.

They do have a cloud offering that should not have these risks but then you have to enter your passwords into their cloud browser environment, presenting a different set of risks. Their cloud offering is basically similar to SkyVerne or even a higher cost tier subscription we have at rtrvr.ai

Re: Launch HN: Browser Use (YC W25) – open-source web agents

#99

Post by arjunchint »

rainforest wrote: post #84
arjunchint wrote: post #8

Have you inspected or thought through the security of your open source library? You are using debugger tools such as CDP, launching playwright without a sandbox, and guiding users to launch Chrome in debugger mode to connect to browser-use on their main browser. The debugging tools you use have active exploits that Google doesn't fix because they are supposed to be for debugging and not for production/general use. Th…

Could you go into a bit more detail about this? Why is exposing devtools to the agent a problem? What's the attack vector? That the agent might do something malicious to exfil saved passwords?

Forget the agent, browser-use's published setup instructions to use with your own Chrome profile and passwords [https://docs.browser-use.com/customize/real-browser, https://github.com/browser-use/browser-use/blob/495714e2dd38...] launches a Chrome session with Remote Debugging enabled.

These tools they are guiding users to setup and execute are "inherently insecure" [https://issues.chromium.org/issues/40056642].

So if you go to a site that can take advantage of these loopholes then your browser is likely to be compromised and could escalate from their.

Re: Launch HN: Browser Use (YC W25) – open-source web agents

#100

Post by rahimnathwani »

cookiengineer wrote: post #10

I've been following your progress for a while now and I'm super impressed how far you've got already. Are you working on unifying the tools that the LLM uses with the MCP / model context protocol? As far as I understand, lots of other providers (like Bolt/Stackblitz etc) are migrating towards this. Currently, there's not many tools available in the upstream specification other than File I/O and some minor interaction…

You can write a wrapper to use it with MCP, or use one someone else has created:

https://github.com/Saik0s/mcp-browser-use/blob/main/README.m...

Post reply on HN