Live data from Hacker News

Show HN: I wrote an open-source browser alternative for Computer Use for any LLM

github.com

21–30 of 75 posts

Re: Show HN: I wrote an open-source browser alternative for Computer Use for any LLM

#21

Awesome project, starred! Here are some other projects for agentic browser interactions: * Cerebellum (Typescript): https://github.com/theredsix/cerebellum * Skyvern: https://github.com/Skyvern-AI/skyvern Disclaimer: I am the author of Cerebellum

Thanks man, starred yours too, it's super cool to see all these projects getting spun up!

I see Cerebellum is vision only. Did you try adding HTML + screenshot? I think that improves the performance like crazy and you don't have to use Claude only.

Just saw Skyvern today on previous Show HNs haha :)

Re: Show HN: I wrote an open-source browser alternative for Computer Use for any LLM

#22
post #12

This looks interesting. I am really impressed with MultiOn [0], and I tried to make something similar, but it's quite challenging doing it with a Chrome extension. I also saw one doing Captcha solving with Selenium [1]. I will keep an eye on your development, good luck! [0] https://www.multion.ai/ [1] https://github.com/VRSEN/agency-swarm

Thanks! Have you tried captcha solving with [1]? It's very tricky sometimes, especially with non standard "verify human" - maybe you could solve it by writing Selenium/Javascript code directly and then execute it.

I haven’t but watched a video doing it with this framework.

With captcha, worst case scenario is using a service to do it as part of the agent flow. See 2captcha service

Re: Show HN: I wrote an open-source browser alternative for Computer Use for any LLM

#23
post #12

This looks interesting. I am really impressed with MultiOn [0], and I tried to make something similar, but it's quite challenging doing it with a Chrome extension. I also saw one doing Captcha solving with Selenium [1]. I will keep an eye on your development, good luck! [0] https://www.multion.ai/ [1] https://github.com/VRSEN/agency-swarm

what are the challenges with the Chrome extension path?

You need to call an API to screenshot the page, then figure out the JavaScript code to execute it. It’s not as easy as it might sound.

Playwright and selenium automate the browser itself, but with the chrome extension you need to use the context of the current browser.

I’m not an expert in browser automation so found it challenging moving from playwright to make it completely browser based.

Re: Show HN: I wrote an open-source browser alternative for Computer Use for any LLM

#24

Awesome project, starred! Here are some other projects for agentic browser interactions: * Cerebellum (Typescript): https://github.com/theredsix/cerebellum * Skyvern: https://github.com/Skyvern-AI/skyvern Disclaimer: I am the author of Cerebellum

Thanks man, starred yours too, it's super cool to see all these projects getting spun up! I see Cerebellum is vision only. Did you try adding HTML + screenshot? I think that improves the performance like crazy and you don't have to use Claude only. Just saw Skyvern today on previous Show HNs haha :)

I had an older version that used simplified HTML, and it got to decent performance with GPT-4o and Gemini but at the cost of 10x token usage. You are right, identifying the interactable elements and pulling out their values into a prompt structure to explicitly allow the next actions can boost performance, especially if done with grammar like structured outputs or guidance-llm. However, I saw that Claude had similar levels of performance with pure vision, and I felt that vision + more training would beat a specialized DOM algorithm due to "the bitter lesson".

BTW I really like your handling of browser tabs, I think it's really clever.

Re: Show HN: I wrote an open-source browser alternative for Computer Use for any LLM

#25

Is it decided then that screenshots are better input for LLMs than HTML, or is that still an active area of investigation? I see that y'all elected for a mostly screenshot-based approach here, wondering if that was based on evidence or just a working theory.

Per research across companies, both help, screenshots are worse, but marginally.

The computer use stuff gets me fired up enough that I end up always sharing this, even though when delivered concisely without breaking NDAs, it can sound like a hot take:

The whole thing is a dead end.

I saw internal work at a FAANG on this for years, and even in the case where the demo is cooked up to "get everything right", intentionally, to figure out the value of investing in chasing this further...its undesirable, for design reasons.

It's easy to imagine being wow'd by the computer doing something itself, but when its us, its a boring and slow way to get things done thats scary to watch.

Even with the stilted 100% success rate, our meatbrains cheerily emulated knowing its I humbly and strongly suggest to anyone interested in this space to work towards CLI versions of this concept. Now, you're nonblocking, are in a more "native" environment for the LLM, and are much cheaper.

If that sounds regressive and hardheaded, Microsoft, in particular, has plenty of research on this subject, and there's a good amount from diverse sources.

Note the 20%-40% success rates they report, then, note that completing a full task successfully represents a product series of 20%-40%. To get an intuition for how this affects the design experience, think how annoying it is to have to repeat a question because Siri/Assistant/whatever voice assistant don't understand it, and they have roughly ~5 errors per 100 words.

Re: Show HN: I wrote an open-source browser alternative for Computer Use for any LLM

#26

Earlier quoted context omitted.

Thanks man, starred yours too, it's super cool to see all these projects getting spun up! I see Cerebellum is vision only. Did you try adding HTML + screenshot? I think that improves the performance like crazy and you don't have to use Claude only. Just saw Skyvern today on previous Show HNs haha :)

I had an older version that used simplified HTML, and it got to decent performance with GPT-4o and Gemini but at the cost of 10x token usage. You are right, identifying the interactable elements and pulling out their values into a prompt structure to explicitly allow the next actions can boost performance, especially if done with grammar like structured outputs or guidance-llm. However, I saw that Claude had similar…

Fair, also Claude probably only gets better on this since they kinda want people to use Computer use. We are gonna try to do best of both worlds.

Thanks man, Magnus came up with it this morning haha!

Re: Show HN: I wrote an open-source browser alternative for Computer Use for any LLM

#27
post #22

Earlier quoted context omitted.

Thanks! Have you tried captcha solving with [1]? It's very tricky sometimes, especially with non standard "verify human" - maybe you could solve it by writing Selenium/Javascript code directly and then execute it.

I haven’t but watched a video doing it with this framework. With captcha, worst case scenario is using a service to do it as part of the agent flow. See 2captcha service

Will def try it.

Re: Show HN: I wrote an open-source browser alternative for Computer Use for any LLM

#28

Is it decided then that screenshots are better input for LLMs than HTML, or is that still an active area of investigation? I see that y'all elected for a mostly screenshot-based approach here, wondering if that was based on evidence or just a working theory.

Screenshots aren't as accurate or context-rich as HTML, but they let you bypass the hassle of building logic for permissions and authentication across different apps to pull in text content for the LLM.

Re: Show HN: I wrote an open-source browser alternative for Computer Use for any LLM

#29
post #23

Earlier quoted context omitted.

what are the challenges with the Chrome extension path?

You need to call an API to screenshot the page, then figure out the JavaScript code to execute it. It’s not as easy as it might sound. Playwright and selenium automate the browser itself, but with the chrome extension you need to use the context of the current browser. I’m not an expert in browser automation so found it challenging moving from playwright to make it completely browser based.

I don’t know a lot about this but do you have full power of Selenium or not? That would be also very interesting aproach especially when “local” browser models get very good

Re: Show HN: I wrote an open-source browser alternative for Computer Use for any LLM

#30

Is it decided then that screenshots are better input for LLMs than HTML, or is that still an active area of investigation? I see that y'all elected for a mostly screenshot-based approach here, wondering if that was based on evidence or just a working theory.

Per research across companies, both help, screenshots are worse, but marginally. The computer use stuff gets me fired up enough that I end up always sharing this, even though when delivered concisely without breaking NDAs, it can sound like a hot take: The whole thing is a dead end. I saw internal work at a FAANG on this for years, and even in the case where the demo is cooked up to "get everything right", intentiona…

Could you elaborate on the CLI idea? I am intrigued but not exactly sure what you mean.
Post reply on HN