Live data from Hacker News

Using GPT-4 Vision with Vimium to browse the web

github.com

71–80 of 133 posts

Re: Using GPT-4 Vision with Vimium to browse the web

#71

At my work there are a large contingent of people who essentially do manual data copying between legacy programs (govt), because the tech debt is so large that we can't figure out a way to plug these things together. Excited for tools like this to eventually act as a layer that can run over these sort of problems, as bizarre a solution as it is from a compute perspective

Working on this layer at https://autotab.com. This sounds like an amazing problem for browser automation to solve, would love to talk with you if you’re interested!

Re: Using GPT-4 Vision with Vimium to browse the web

#72

At my work there are a large contingent of people who essentially do manual data copying between legacy programs (govt), because the tech debt is so large that we can't figure out a way to plug these things together. Excited for tools like this to eventually act as a layer that can run over these sort of problems, as bizarre a solution as it is from a compute perspective

Funny that you and others on here don't seem to realize that literally everybody who uses the internet has the exact same data entry problem all the time. Blame it on "old software", but how about the entire internet? copying (or in most cases even worse: re-typing) form data from one location on the screen into yet another webform. Username, password, email address, physical address, credit card info etc etc. Some e…

Yeah, my dream would be using this to scrape pages, pop the content into my provide db, serving it up in my own format (which is going to be a white page with letters with inline images and videos that are not ads. And my interactions fed back to the vision model to post in the original. So I never have to see a ‘design’ (heavy js riddled unreadable crap) again in my life. And so I can, with my own tooling, browse and reuse my history including content instead rely on all the broken stuff bolted on the web.

Re: Using GPT-4 Vision with Vimium to browse the web

#73

Many Dutch companies pay salaries by 1. receiving payslips from the accountant, and then 2. manually initiating bank transfers to each employee for the amount in the corresponding payslip, and then 3. manually initiating a bank transfer to the tax authority to pay the withholded salary taxes. This is completely useless manual labor. There should be no reason for this to be a manual procedure. And yet it's almost impo…

I don't think this really has much to do with AI. In the UK there are solutions like Pento now which do all this, including automating payments via open banking to the user and the tax authority and automatically filing tax filings:

https://www.pento.io/la/payroll-software

Re: Using GPT-4 Vision with Vimium to browse the web

#74

Looks extremely cool. Trying to run it though, I get stuck at "Getting actions for the given objective..." (using the example on the repo)

Huh weird, I'm getting that too. OpenAI has been having periodic outages today, think that might be why since it was working fine earlier.

https://www.bleepingcomputer.com/news/security/openai-confir... News reports and their https://status.openai.com/incidents/21vl32gvx3hb incident reports indicate they are mitigating / fighting off attacks recently

Re: Using GPT-4 Vision with Vimium to browse the web

#75

At my work there are a large contingent of people who essentially do manual data copying between legacy programs (govt), because the tech debt is so large that we can't figure out a way to plug these things together. Excited for tools like this to eventually act as a layer that can run over these sort of problems, as bizarre a solution as it is from a compute perspective

Funny that you and others on here don't seem to realize that literally everybody who uses the internet has the exact same data entry problem all the time. Blame it on "old software", but how about the entire internet? copying (or in most cases even worse: re-typing) form data from one location on the screen into yet another webform. Username, password, email address, physical address, credit card info etc etc. Some e…

FTL. See NiagraFiles.

Re: Using GPT-4 Vision with Vimium to browse the web

#76

At my work there are a large contingent of people who essentially do manual data copying between legacy programs (govt), because the tech debt is so large that we can't figure out a way to plug these things together. Excited for tools like this to eventually act as a layer that can run over these sort of problems, as bizarre a solution as it is from a compute perspective

Funny that you and others on here don't seem to realize that literally everybody who uses the internet has the exact same data entry problem all the time. Blame it on "old software", but how about the entire internet? copying (or in most cases even worse: re-typing) form data from one location on the screen into yet another webform. Username, password, email address, physical address, credit card info etc etc. Some e…

Bash pipes? The free flow of information through composable tools.

The commercial web? Not the above.

This is just a baseline. I’m sure that an LLM can help the issue but the biggest problem is that these varied HTTP-with-datastores are islands passing messages in bottles back and forth while a bash pipeline is akin to fiber optics.

Re: Using GPT-4 Vision with Vimium to browse the web

#78
post #15

Earlier quoted context omitted.

The industry buzzword is "Robotic Process Automation", which as a category of products has been focused on using various forms of ML/AI to glue these things together in a common/structured way (in addition to good old fashioned screen scraping). Up this this point, these products have been quite brittle. The recent explosion of AI tech seems like quite a boon for this space.

I totally agree on all points, especially around what AI means for this. I'm kind of in a happy accident situation because I was working on something for RPA, which then became a layer that was factored as its own product, but now might be able to come full circle as a result of AI. Essentially this layer can function as a "delivery medium" for RPA agent creation, that you can use on any device without download. Howe…

I have watched your project for a while as a possible option for embedded browsers for XR applications like WebXR but the high licensing cost was a factor and solutions like Hyperbeam or Vueplex in Unity have been possible. Defiantly agree that multimodal LLM integration is a huge opportunity and multiplayer browsing with AI in realtime is a super cool idea if you package it right.

Re: Using GPT-4 Vision with Vimium to browse the web

#79
Been playing with this through the ChatGPT interface for the past few weeks. Couple of tips. Update the css to get rid of the gradients and rounded corners. I found red with bold white text to be most consistent. Increase the font size. If two labels overlap, push them apart and add an arrow to the element. Send both images to the API, a version with the annotations added and a version without.

Re: Using GPT-4 Vision with Vimium to browse the web

#80

Ah, very similar to Adept’s[1] concept? Though, their product seems not yet ready. [1] https://www.adept.ai/

Yep, took inspiration from them and a couple other startups

What other startups did you use for inspiration?
Post reply on HN