Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Using GPT-4 Vision with Vimium to browse the web

github.com

Post reply on HN
Print view View original

51–60 of 133 posts

Re: Using GPT-4 Vision with Vimium to browse the web

#51

Post by keepamovin »

haswell wrote: post #15
transistorfan wrote: post #6

At my work there are a large contingent of people who essentially do manual data copying between legacy programs (govt), because the tech debt is so large that we can't figure out a way to plug these things together. Excited for tools like this to eventually act as a layer that can run over these sort of problems, as bizarre a solution as it is from a compute perspective

The industry buzzword is "Robotic Process Automation", which as a category of products has been focused on using various forms of ML/AI to glue these things together in a common/structured way (in addition to good old fashioned screen scraping). Up this this point, these products have been quite brittle. The recent explosion of AI tech seems like quite a boon for this space.

I totally agree on all points, especially around what AI means for this.

I'm kind of in a happy accident situation because I was working on something for RPA, which then became a layer that was factored as its own product, but now might be able to come full circle as a result of AI.

Essentially this layer can function as a "delivery medium" for RPA agent creation, that you can use on any device without download. However, as it has many others uses I've been working on those, but I've been seeking a great reason to get back into RPA.

I have a cool idea to leverage human-guided AI creation of data maps and action tours for RPA, but similar to what you say, unless great care is taken you can end up with a brittle approach. Also, as the market has been quite saturated many reasonable approaches, I just haven't felt compelled.

Yet now I think the possible merging of GPT level AIs with browser instrumentation to deliver an augmented way to browse the web makes that incredibly compelling.

So I'm incredibly thrilled that I have this happy accident of BrowserBox^0 (the factored out layer originally from RPA work above) which provides a pluggable/iframe-emebeddable interface for remotely controlling a headless browser. So now I want to look at unifying BrowserBox with this kind of GPT driven exploration.

It's even cooler, because, as BB enables co-browsing by default (multiplayer browsing) and turns the browser into a "client-server" architecture, I can see plugging in GPT-4V as a connecting client with some kind of minimal API affordance for it to use would, like the very cool vimium keyboard-enabled browsing in the OP, would be such interesting project to try!

We're open source so if you want to check us out or get involved in this quest, come say hi, maybe get involved if you're game!

0: https://github.com/BrowserBox/BrowserBox

Re: Using GPT-4 Vision with Vimium to browse the web

#52

Post by Roark66 »

transistorfan wrote: post #41
leovander wrote: post #28

Earlier quoted context omitted.

In the OP's specific instance when would you reach out for a traditional ETL tool vs an RPA solution?

How much does the involvement of a bank of fax machines complicate things?

A little perhaps, but not much. One can replace a bank of physical fax machines with modems.

It would be an interesting job for sure. Why wasn't it done before? I can imagine only two reasons. One, there isn't that much data to move and it makes no sense to build software for what few people spend 30min per day on. Two, the data in the legacy system is images and people are not just moving it between systems, but they also do categorisation, verification etc. In which case an AI model may be useful, but almost always hard coded rules will be faster.

Re: Using GPT-4 Vision with Vimium to browse the web

#53

Post by Garlef »

transistorfan wrote: post #6

At my work there are a large contingent of people who essentially do manual data copying between legacy programs (govt), because the tech debt is so large that we can't figure out a way to plug these things together. Excited for tools like this to eventually act as a layer that can run over these sort of problems, as bizarre a solution as it is from a compute perspective

"Chinese Room Automation"

Re: Using GPT-4 Vision with Vimium to browse the web

#54

Post by [deleted] »

Garlef wrote: post #53
transistorfan wrote: post #6

At my work there are a large contingent of people who essentially do manual data copying between legacy programs (govt), because the tech debt is so large that we can't figure out a way to plug these things together. Excited for tools like this to eventually act as a layer that can run over these sort of problems, as bizarre a solution as it is from a compute perspective

"Chinese Room Automation"

[deleted]

Re: Using GPT-4 Vision with Vimium to browse the web

#55

Post by bboygravity »

transistorfan wrote: post #6

At my work there are a large contingent of people who essentially do manual data copying between legacy programs (govt), because the tech debt is so large that we can't figure out a way to plug these things together. Excited for tools like this to eventually act as a layer that can run over these sort of problems, as bizarre a solution as it is from a compute perspective

Funny that you and others on here don't seem to realize that literally everybody who uses the internet has the exact same data entry problem all the time. Blame it on "old software", but how about the entire internet?

copying (or in most cases even worse: re-typing) form data from one location on the screen into yet another webform.

Username, password, email address, physical address, credit card info etc etc.

Some extensions try to help with data entry, but none of them work properly and consistently enough to really help. Even consistently filling just username and pw is too much to ask.

It's my number 1 frustration when using the internet (worse than ads) and I find it mind-blowing that this hasn't been solved yet with or without LLMs.

I would pay a montly fee for any software that solves this once and for all and it sounds like it's coming (and I'm already paying their monthly fee).

Re: Using GPT-4 Vision with Vimium to browse the web

#56

Post by jimmySixDOF »

ishan0102 wrote: post #4

Hey! Creator here, thanks for sharing! Let me know if anyone has questions and feel free to contribute, I've left some potential next steps in the README.

Nice. I know Open Interpreter are trying to get Selenium automated to natural language control and quite a few other projects are also popping up on HN lately. The vimium approach is a lot lighter so looks promising. One way or another the as-published world wide web is turning into its own dynamic API overlay server. Ingest all the Sources!

Re: Using GPT-4 Vision with Vimium to browse the web

#58

Post by jatins »

snake_doc wrote: post #22

Ah, very similar to Adept’s[1] concept? Though, their product seems not yet ready. [1] https://www.adept.ai/

It's also a little insane to me that what Adept has been supposedly building for years with 300+ mil in funding can now be built in a day with Open AI APIs?

I think Adept pivoted along the way but original concept was very similar to this.

Re: Using GPT-4 Vision with Vimium to browse the web

#59

Post by pseudosaid »

bboygravity wrote: post #55
transistorfan wrote: post #6

At my work there are a large contingent of people who essentially do manual data copying between legacy programs (govt), because the tech debt is so large that we can't figure out a way to plug these things together. Excited for tools like this to eventually act as a layer that can run over these sort of problems, as bizarre a solution as it is from a compute perspective

Funny that you and others on here don't seem to realize that literally everybody who uses the internet has the exact same data entry problem all the time. Blame it on "old software", but how about the entire internet? copying (or in most cases even worse: re-typing) form data from one location on the screen into yet another webform. Username, password, email address, physical address, credit card info etc etc. Some e…

use a password manager. i havent copy pasted form data twice on a site in a long time

Re: Using GPT-4 Vision with Vimium to browse the web

#60

Post by fragmede »

bboygravity wrote: post #55
transistorfan wrote: post #6

At my work there are a large contingent of people who essentially do manual data copying between legacy programs (govt), because the tech debt is so large that we can't figure out a way to plug these things together. Excited for tools like this to eventually act as a layer that can run over these sort of problems, as bizarre a solution as it is from a compute perspective

Funny that you and others on here don't seem to realize that literally everybody who uses the internet has the exact same data entry problem all the time. Blame it on "old software", but how about the entire internet? copying (or in most cases even worse: re-typing) form data from one location on the screen into yet another webform. Username, password, email address, physical address, credit card info etc etc. Some e…

consistently filling out username and password is all I wanted from my password manager, but it turns out it handles credit card number and other bits of information for me as well.
Post reply on HN