Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: I wrote an open-source browser alternative for Computer Use for any LLM

github.com

Post reply on HN
Print view View original

41–50 of 75 posts

Re: Show HN: I wrote an open-source browser alternative for Computer Use for any LLM

#41

Post by fragmede »

refulgentis wrote: post #39
gregpr07 wrote: post #34

Earlier quoted context omitted.

Hmm, but this how we handle it? We just have a CLI that outputs exactly, goal, state, and asks user for more clarity if needed, no GUI. The original idea was to make it completely headless.

I'm sorry I'm definitely off today, and am missing it, I appreciate your patience. I'm thinking maybe the goal/state stuff might have clouded my point. Setting aside prompt engineering, just thinking of the stock AI UIs today, i.e. chat based. Then, we want to accomplish some goal using GUI and/or CLI. Given the premise that I'd avoid GUI automation, why am I saying CLI is the way to go? A toy example: let's say the…

try openinterpreter for cli computer automation.

Re: Show HN: I wrote an open-source browser alternative for Computer Use for any LLM

#42

Post by refulgentis »

fragmede wrote: post #41
refulgentis wrote: post #39

Earlier quoted context omitted.

I'm sorry I'm definitely off today, and am missing it, I appreciate your patience. I'm thinking maybe the goal/state stuff might have clouded my point. Setting aside prompt engineering, just thinking of the stock AI UIs today, i.e. chat based. Then, we want to accomplish some goal using GUI and/or CLI. Given the premise that I'd avoid GUI automation, why am I saying CLI is the way to go? A toy example: let's say the…

try openinterpreter for cli computer automation.

Nice, thanks

Re: Show HN: I wrote an open-source browser alternative for Computer Use for any LLM

#43

Post by namuorg »

Oras wrote: post #12

This looks interesting. I am really impressed with MultiOn [0], and I tried to make something similar, but it's quite challenging doing it with a Chrome extension. I also saw one doing Captcha solving with Selenium [1]. I will keep an eye on your development, good luck! [0] https://www.multion.ai/ [1] https://github.com/VRSEN/agency-swarm

I actually built a Chrome extension that runs Claude computer use if you’d like to try it out! [0] It’s currently awaiting approval in the Chrome Web Store.

After having spent the last several years building a popular Chrome extension for browser automation [1], I was excited to see if LLMs could actually build automations end-to-end based on a high-level description. Unfortunately, they still get confused quite easily so the holy grail has yet to come. Still fun to play around with though!

[0] https://autobrowser.ai/

[1] https://news.ycombinator.com/item?id=29254147

Re: Show HN: I wrote an open-source browser alternative for Computer Use for any LLM

#44

Post by maggreenWAI »

gregpr07 wrote: post #11
bravura wrote: post #5

It would be amazing if you: a) There were a test / eval suite to determine which model works best for what. It could be divided into a training suite and test suite. (Training tasks can be used for training, test tasks only for evaluation.) Possibly a combination of unit tests against known xpaths, and integration tests that are multi-step and end in a measurable result. I know the web is constantly changing, so I'm…

A) we plan on thoroughly testing that with Mind2Web dataset. They have a very robust set of (persistant) selectors B) so, shadcn for prompts for web agents haha :) but I agree, that would be SICK! Just go to browseruse and get the prompt for your specific use case

A) For Mind2Web: because there are multiple ways to reach a goal state - any thoughts how to evaluate if a task was successful? Should we let the LLM/ other LLM evaluate it?

Re: Show HN: I wrote an open-source browser alternative for Computer Use for any LLM

#45

Post by maggreenWAI »

EGreg wrote: post #37
its_down_again wrote: post #28

Earlier quoted context omitted.

Screenshots aren't as accurate or context-rich as HTML, but they let you bypass the hassle of building logic for permissions and authentication across different apps to pull in text content for the LLM.

Can’t you just make a browser extension to haveaccess to the HTML and CSS, and use LLMs from that?

Context length + API cost is right now main bottleneck for huge HTML + CSS files. The extraction here is already quite efficient but still: with past messages + system prompt + sometimes extracted text + extracted interactive elements you are quickly already around 2500 tokens (for gpt-4o 0.01$).

If you extract entire HTML and CSS your cost + inference time are quickly 10x.

Re: Show HN: I wrote an open-source browser alternative for Computer Use for any LLM

#46

Post by maggreenWAI »

firejake308 wrote: post #6

Is it decided then that screenshots are better input for LLMs than HTML, or is that still an active area of investigation? I see that y'all elected for a mostly screenshot-based approach here, wondering if that was based on evidence or just a working theory.

Next step is to represent also the structure of the HTML tree in the extracted elements for better understanding, maybe images are then less needed.

Re: Show HN: I wrote an open-source browser alternative for Computer Use for any LLM

#47

Post by maggreenWAI »

Let's say in 1 year, more agents than humans interact with the web.

Do you think: 1. Websites release more API functions for agents to interact with them or 2. We will transform with tools like this the UI into functions callable by agents and maybe even cache all inferred functions for websites in a third party service?

Re: Show HN: I wrote an open-source browser alternative for Computer Use for any LLM

#48

Post by EGreg »

maggreenWAI wrote: post #45
EGreg wrote: post #37

Earlier quoted context omitted.

Can’t you just make a browser extension to haveaccess to the HTML and CSS, and use LLMs from that?

Context length + API cost is right now main bottleneck for huge HTML + CSS files. The extraction here is already quite efficient but still: with past messages + system prompt + sometimes extracted text + extracted interactive elements you are quickly already around 2500 tokens (for gpt-4o 0.01$). If you extract entire HTML and CSS your cost + inference time are quickly 10x.

Aren't screenshots far larger than this?

Re: Show HN: I wrote an open-source browser alternative for Computer Use for any LLM

#49

Post by maggreenWAI »

rahimnathwani wrote: post #7

In case anyone else was looking for the functions available to the LLM: https://github.com/gregpr07/browser-use/blob/68a3227c8bc97fe...

You can just extend this e.g. with adding data to database, sending notifications, extracting specific data format ect... make sure to also accept your added function when its called in act() https://github.com/gregpr07/browser-use/blob/main/src/contro...

Re: Show HN: I wrote an open-source browser alternative for Computer Use for any LLM

#50

Post by gregpr07 »

EGreg wrote: post #48
maggreenWAI wrote: post #45

Earlier quoted context omitted.

Context length + API cost is right now main bottleneck for huge HTML + CSS files. The extraction here is already quite efficient but still: with past messages + system prompt + sometimes extracted text + extracted interactive elements you are quickly already around 2500 tokens (for gpt-4o 0.01$). If you extract entire HTML and CSS your cost + inference time are quickly 10x.

Aren't screenshots far larger than this?

Nope: 1280x1024 low resolution with gpt-4o are 85 tokens so approx $0.0002 (so 100x cheaper). For high resolution its apporx $0.002 https://openai.com/api/pricing/
Post reply on HN