Show HN: I wrote an open-source browser alternative for Computer Use for any LLM • Hacker News Board

Re: Show HN: I wrote an open-source browser alternative for Computer Use for any LLM

#51

Post by arjvik » Tue, Nov 05, 2024, 10:53 PM UTC

gregpr07 wrote: post #10

firejake308 wrote: post #6

Is it decided then that screenshots are better input for LLMs than HTML, or is that still an active area of investigation? I see that y'all elected for a mostly screenshot-based approach here, wondering if that was based on evidence or just a working theory.

Not sure, I think there is a lot of research being done here. Actually, browser use works quite well with vision turned off, it just sometimes gets stuck at some trivial vision tasks. The interesting thing is that screenshot approach is often cheaper than cleaned up html, because some websites have HUGE action spaces. We looked at some papers (like ferret ui) but i think we can do much better on html tasks. Also, the…

Would be really cool if you could tie this into Claude's computer use APIs!

Re: Show HN: I wrote an open-source browser alternative for Computer Use for any LLM

#52

Post by gregpr07 » Tue, Nov 05, 2024, 11:02 PM UTC

arjvik wrote: post #51

gregpr07 wrote: post #10

Earlier quoted context omitted.

Not sure, I think there is a lot of research being done here. Actually, browser use works quite well with vision turned off, it just sometimes gets stuck at some trivial vision tasks. The interesting thing is that screenshot approach is often cheaper than cleaned up html, because some websites have HUGE action spaces. We looked at some papers (like ferret ui) but i think we can do much better on html tasks. Also, the…

Would be really cool if you could tie this into Claude's computer use APIs!

Do you think they do any super fancy magic other than for example how ferret ui does their classification of ui elements? It could be very interesting to test head to head hope much better you can make computer use by adding html (it’s much better from our quick testing, just don’t know the numbers).

Re: Show HN: I wrote an open-source browser alternative for Computer Use for any LLM

#53

Post by e-clinton » Tue, Nov 05, 2024, 11:46 PM UTC

EGreg wrote: post #37

its_down_again wrote: post #28

Earlier quoted context omitted.

Screenshots aren't as accurate or context-rich as HTML, but they let you bypass the hassle of building logic for permissions and authentication across different apps to pull in text content for the LLM.

Can’t you just make a browser extension to haveaccess to the HTML and CSS, and use LLMs from that?

I do this for my extension [0] but the HTML is often too large for context window sizes . I end up doing scraping of the relevant pieces before sending to LLM.

[0] https://chromewebstore.google.com/detail/namebrand-check-for...

Re: Show HN: I wrote an open-source browser alternative for Computer Use for any LLM

#54

Post by KaoruAoiShiho » Tue, Nov 05, 2024, 11:53 PM UTC

Maybe can build a database for which sites / pages work best with HTML vs Screenshots, and then can choose to use HTML to save on token cost / improve latency if possible.

Re: Show HN: I wrote an open-source browser alternative for Computer Use for any LLM

#55

Post by G_o_D » Wed, Nov 06, 2024, 12:27 AM UTC

It is called screen scraping, where text rendered on screen/monitors are being scraped either in browser or even in windows os even on android screen , thats how softwares like autohotkey and all do automation windows or android screen can be dumped into heirarchical xml along with x y coordinates of its ui elements along with text they contain which can be uses o click scroll scrape text

Re: Show HN: I wrote an open-source browser alternative for Computer Use for any LLM

#56

Post by teaxio » Wed, Nov 06, 2024, 1:31 AM UTC

firejake308 wrote: post #6

Is it decided then that screenshots are better input for LLMs than HTML, or is that still an active area of investigation? I see that y'all elected for a mostly screenshot-based approach here, wondering if that was based on evidence or just a working theory.

[dead]

Re: Show HN: I wrote an open-source browser alternative for Computer Use for any LLM

#57

Post by gitgud » Wed, Nov 06, 2024, 3:07 AM UTC

It's impressive, but to me it seems like the saddest development experience...

    agent = Agent(
        task='Go to hackernews on show hn and give me top 10 post titels, their points and hours. Calculate for each the ratio of points per hour.',
        llm=ChatOpenAI(model='gpt-4o'),
    )
    
    await agent.run()

Passing prompts to a LLM agent... waiting for the black box to run and do something...

Re: Show HN: I wrote an open-source browser alternative for Computer Use for any LLM

#58

Post by atrus » Wed, Nov 06, 2024, 3:30 AM UTC

gitgud wrote: post #57

It's impressive, but to me it seems like the saddest development experience... agent = Agent( task='Go to hackernews on show hn and give me top 10 post titels, their points and hours. Calculate for each the ratio of points per hour.', llm=ChatOpenAI(model='gpt-4o'), ) await agent.run() Passing prompts to a LLM agent... waiting for the black box to run and do something...

I mean, is that really much different than an API? I pass a query, and get data back, and rarely do I get to inspect the mechanisms behind what's returning that data.

Re: Show HN: I wrote an open-source browser alternative for Computer Use for any LLM

#59

Post by dbacar » Wed, Nov 06, 2024, 6:09 AM UTC

gregpr07 wrote: post #21

theredsix wrote: post #20

Awesome project, starred! Here are some other projects for agentic browser interactions: * Cerebellum (Typescript): https://github.com/theredsix/cerebellum * Skyvern: https://github.com/Skyvern-AI/skyvern Disclaimer: I am the author of Cerebellum

Thanks man, starred yours too, it's super cool to see all these projects getting spun up! I see Cerebellum is vision only. Did you try adding HTML + screenshot? I think that improves the performance like crazy and you don't have to use Claude only. Just saw Skyvern today on previous Show HNs haha :)

I starred both of you

Re: Show HN: I wrote an open-source browser alternative for Computer Use for any LLM

#60

Post by gitgud » Wed, Nov 06, 2024, 6:34 AM UTC

atrus wrote: post #58

gitgud wrote: post #57

It's impressive, but to me it seems like the saddest development experience... agent = Agent( task='Go to hackernews on show hn and give me top 10 post titels, their points and hours. Calculate for each the ratio of points per hour.', llm=ChatOpenAI(model='gpt-4o'), ) await agent.run() Passing prompts to a LLM agent... waiting for the black box to run and do something...

I mean, is that really much different than an API? I pass a query, and get data back, and rarely do I get to inspect the mechanisms behind what's returning that data.

Okay that’s a pretty good point actually.

I guess I wrongly assume regular API’s are more reliable, but you’re right they’re basically black boxes too…