Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: I wrote an open-source browser alternative for Computer Use for any LLM

github.com

Post reply on HN
Print view View original

1–10 of 75 posts

Show HN: I wrote an open-source browser alternative for Computer Use for any LLM

#1

Post by gregpr07 »

Hey HN,

I made Browser-Use, an open-source tool that lets (all Langchain supported) LLMs execute tasks directly in the browser just with function calling.

It allows you to build agents that interact with web elements using natural language prompts. We created a layer that simplifies website interaction for LLMs by extracting xPaths and interactive elements like buttons and input fields (and other fancy things). This enables you to design custom web automation and scraping functions without manual inspection through DevTools.

Hasn't this been done a lot of times? Good question, as a general SaaS tool yes, but I think a lot of people are going to try to make their own web automation agents from scratch, so the idea is to provide groundwork/library for the hard part so that not everyone has to repeat these steps:

- parse html in a LLM friendly way (clickable items + screenshots)

- provide a nice function calls for everything inside the browser

- create reusable agent classes

What this is NOT? An all knowing AI agent that can solve all your problems.

The vision: create repeatable tasks on the web just by prompting your agent and not care about the hows.

To better showcase the power of text extraction we made a few demos such as:

- Applying for multiple software engineering jobs in San Francisco

- Opening new tabs to search for images of Albert Einstein, Oprah Winfrey, and Steve Jobs

- Finding the cheapest one-way flight from London to Kyrgyzstan for December 25th

I’d be interested in feedback on how this tool fits into your automation workflows. Try it out and let me know how it performs on your end.

We are Gregor & Magnus and we built this in 5 days.

Show HN: I wrote an open-source browser alternative for Computer Use for any LLM
github.com

Re: Show HN: I wrote an open-source browser alternative for Computer Use for any LLM

#2

Post by coreyp_1 »

This looks really interesting. The first hurdle, though, that prevents me from experimenting with this on my job is the lack of a license.

I see in the readme that it claims that it is MIT licensed, but there is no actual license file or information in any of the source files that I could find.

Re: Show HN: I wrote an open-source browser alternative for Computer Use for any LLM

#4

Post by gregpr07 »

coreyp_1 wrote: post #2

This looks really interesting. The first hurdle, though, that prevents me from experimenting with this on my job is the lack of a license. I see in the readme that it claims that it is MIT licensed, but there is no actual license file or information in any of the source files that I could find.

Thanks you, love the feedback! Will add the license.

Let me know how it goes if you try it.

Re: Show HN: I wrote an open-source browser alternative for Computer Use for any LLM

#5

Post by bravura »

It would be amazing if you:

a) There were a test / eval suite to determine which model works best for what. It could be divided into a training suite and test suite. (Training tasks can be used for training, test tasks only for evaluation.) Possibly a combination of unit tests against known xpaths, and integration tests that are multi-step and end in a measurable result. I know the web is constantly changing, so I'm not 100% sure how this should work.

b) There were some sort of wiki, or perhaps another repo or discussion board, of community-generated prompt recipes for particular actions.

Re: Show HN: I wrote an open-source browser alternative for Computer Use for any LLM

#6

Post by firejake308 »

Is it decided then that screenshots are better input for LLMs than HTML, or is that still an active area of investigation? I see that y'all elected for a mostly screenshot-based approach here, wondering if that was based on evidence or just a working theory.

Re: Show HN: I wrote an open-source browser alternative for Computer Use for any LLM

#8

Post by gregpr07 »

rahimnathwani wrote: post #7

In case anyone else was looking for the functions available to the LLM: https://github.com/gregpr07/browser-use/blob/68a3227c8bc97fe...

Yes, this plus reasoning and ask user for additional info. More here https://github.com/gregpr07/browser-use/blob/main/src/agent/...

Re: Show HN: I wrote an open-source browser alternative for Computer Use for any LLM

#9

Post by gregpr07 »

WillAdams wrote: post #3

Does it work with COM objects/Java applications? I'd give my interest in Hell for a way to have a script plug in data into a Java app.

No I think that’s a completely different beast, this is only for html/websites.

But would be interesting to see what happenes with our pipeline with pure vision model. Did you mean something else?

Re: Show HN: I wrote an open-source browser alternative for Computer Use for any LLM

#10

Post by gregpr07 »

firejake308 wrote: post #6

Is it decided then that screenshots are better input for LLMs than HTML, or is that still an active area of investigation? I see that y'all elected for a mostly screenshot-based approach here, wondering if that was based on evidence or just a working theory.

Not sure, I think there is a lot of research being done here.

Actually, browser use works quite well with vision turned off, it just sometimes gets stuck at some trivial vision tasks. The interesting thing is that screenshot approach is often cheaper than cleaned up html, because some websites have HUGE action spaces.

We looked at some papers (like ferret ui) but i think we can do much better on html tasks. Also, there is a lot of space to improve the current pipeline.

Post reply on HN