Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: I wrote an open-source browser alternative for Computer Use for any LLM

github.com

Post reply on HN
Print view View original

31–40 of 75 posts

Re: Show HN: I wrote an open-source browser alternative for Computer Use for any LLM

#31

Post by refulgentis »

gregpr07 wrote: post #30
refulgentis wrote: post #25

Earlier quoted context omitted.

Per research across companies, both help, screenshots are worse, but marginally. The computer use stuff gets me fired up enough that I end up always sharing this, even though when delivered concisely without breaking NDAs, it can sound like a hot take: The whole thing is a dead end. I saw internal work at a FAANG on this for years, and even in the case where the demo is cooked up to "get everything right", intentiona…

Could you elaborate on the CLI idea? I am intrigued but not exactly sure what you mean.

(handwaving) I'd rather be in a loop of "here's our goal. here's latest output from the CLI. what do we type into the CLI" than the GUI version of that loop.

I hope that's clearer, I'm a bit over-caffeinated

Re: Show HN: I wrote an open-source browser alternative for Computer Use for any LLM

#32

Post by alexeichhorn »

craftkiller wrote: post #15
daft_pink wrote: post #13

I was really excited about the original claude computer use until I watched the youtube videos and saw it was only running in a docker container. I wish I could run something like this on a real machine.

What makes a docker container not a "real machine"? Docker programs are running natively, just like any other program, without emulation/virtualization. Its not like a virtual machine (unless you're on one of the lesser operating systems like Windows or OSX), its just configuring some settings in the Linux kernel to isolate the process from other processes. Its basically just an enhanced chroot.

Running natively doesn't make it a real machine. If I run iOS Simulator, it also runs natively, but I'm pretty sure it's not a real iPhone ;)

Re: Show HN: I wrote an open-source browser alternative for Computer Use for any LLM

#33

Post by Oras »

gregpr07 wrote: post #29
Oras wrote: post #23

Earlier quoted context omitted.

You need to call an API to screenshot the page, then figure out the JavaScript code to execute it. It’s not as easy as it might sound. Playwright and selenium automate the browser itself, but with the chrome extension you need to use the context of the current browser. I’m not an expert in browser automation so found it challenging moving from playwright to make it completely browser based.

I don’t know a lot about this but do you have full power of Selenium or not? That would be also very interesting aproach especially when “local” browser models get very good

From 3 days playing around it, I couldn’t find a way to use selenium or playwright in the browser.

What I did though is having a loop to send instructions from playwright.

For instance, I will open the browser, and then enter a loop to await for instructions (can be from event such as redis) to execute again in the same browser. But still, it’s based on the session instantiated by playwright.

Re: Show HN: I wrote an open-source browser alternative for Computer Use for any LLM

#34

Post by gregpr07 »

refulgentis wrote: post #31
gregpr07 wrote: post #30

Earlier quoted context omitted.

Could you elaborate on the CLI idea? I am intrigued but not exactly sure what you mean.

(handwaving) I'd rather be in a loop of "here's our goal. here's latest output from the CLI. what do we type into the CLI" than the GUI version of that loop. I hope that's clearer, I'm a bit over-caffeinated

Hmm, but this how we handle it? We just have a CLI that outputs exactly, goal, state, and asks user for more clarity if needed, no GUI. The original idea was to make it completely headless.

Re: Show HN: I wrote an open-source browser alternative for Computer Use for any LLM

#36

Post by EGreg »

firejake308 wrote: post #6

Is it decided then that screenshots are better input for LLMs than HTML, or is that still an active area of investigation? I see that y'all elected for a mostly screenshot-based approach here, wondering if that was based on evidence or just a working theory.

I doubt screenshots would be better input considering that eg box options and other markup are hisden visually until a user interacts with something

Re: Show HN: I wrote an open-source browser alternative for Computer Use for any LLM

#37

Post by EGreg »

its_down_again wrote: post #28
firejake308 wrote: post #6

Is it decided then that screenshots are better input for LLMs than HTML, or is that still an active area of investigation? I see that y'all elected for a mostly screenshot-based approach here, wondering if that was based on evidence or just a working theory.

Screenshots aren't as accurate or context-rich as HTML, but they let you bypass the hassle of building logic for permissions and authentication across different apps to pull in text content for the LLM.

Can’t you just make a browser extension to haveaccess to the HTML and CSS, and use LLMs from that?

Re: Show HN: I wrote an open-source browser alternative for Computer Use for any LLM

#39

Post by refulgentis »

gregpr07 wrote: post #34
refulgentis wrote: post #31

Earlier quoted context omitted.

(handwaving) I'd rather be in a loop of "here's our goal. here's latest output from the CLI. what do we type into the CLI" than the GUI version of that loop. I hope that's clearer, I'm a bit over-caffeinated

Hmm, but this how we handle it? We just have a CLI that outputs exactly, goal, state, and asks user for more clarity if needed, no GUI. The original idea was to make it completely headless.

I'm sorry I'm definitely off today, and am missing it, I appreciate your patience.

I'm thinking maybe the goal/state stuff might have clouded my point. Setting aside prompt engineering, just thinking of the stock AI UIs today, i.e. chat based.

Then, we want to accomplish some goal using GUI and/or CLI. Given the premise that I'd avoid GUI automation, why am I saying CLI is the way to go?

A toy example: let's say the user says "get my current IP".

If our agent is GUI-based, maybe it does: open Chrome > type in whatismyip.com > recognize IP from screenshot.

If our agent is CLI-based, maybe it does: run the curl command to fetch the user's IP from a public API (e.g. curl whatismyip.com) > parse the output to extract the IP address > return the IP address to the user as text.

In the CLI example, the agent interacts with the system using native commands (in this case, curl) and text outputs, rather than trying to simulate GUI actions and parse screenshot contents.

Why do I believe thats preferable over GUI-based automation?

1. More direct/efficient - no need for browser launching, screenshot processing, etc.

2. More reliable - dealing with only structured text output, rather than trying to parse visual elements

3. Parallelizable: I can have N CLI shells, but only 1 GUI shell, which is shared with the user.

4. In practice, I'm basing that off observations of the GUI-automation project I mentioned, accepting computer automation is desirable, and...work I did to build an end-to-end testing framework for devices paired to phones, both iOS and Android.

What the? Where did that come from?

TL;DR: I love E2E tests, for years, and it was stultifying to see how little they were used beyond the testing team due to flakiness. Even small things like "Launch the browser" are extremely fraught. How long to wait? How often do we poll? How do we deal with some dialog appearing in front of the app? How do we deal with not having the textual view hierarchy for the entire OS?

Re: Show HN: I wrote an open-source browser alternative for Computer Use for any LLM

#40

Post by fragmede »

wants to have cron, so I can ask it to check with my local parking agency, every day or every 12 hours, do I have a parking ticket, and to raise a warning if I do. Or to check with county jail and see if someone is still there/not there. Or check the price of a product on Amazon every hour and warn when it's changed (aka camelcamelcamel but local). Search craigslist/zillow/Facebook marketplace for items until one shows up. etc.
Post reply on HN