Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Using GPT-4 Vision with Vimium to browse the web

github.com

Post reply on HN
Print view View original

121–130 of 133 posts

Re: Using GPT-4 Vision with Vimium to browse the web

#121

Post by rpigab »

This is amazing that it's possible and works, but I wonder if the electricity cost is sustainable in the long run.

For handicapped people who depend on tools like this for accessibility, it's justified, but I wouldn't use it myself if it uses too much power.

I'm sure OpenAI and friends love operating at a loss until everyone uses their products, then enshittify or raise prices, like Netflix, Microsoft, Google, etc., but CO2 emissions can't be easily reversed.

I'd be glad to listen to other points of view though, maybe everything we do on computers is already bad for the environment anyway and comparing which one pollutes more is vain, idk.

Re: Using GPT-4 Vision with Vimium to browse the web

#122

Post by abrichr »

transistorfan wrote: post #6

At my work there are a large contingent of people who essentially do manual data copying between legacy programs (govt), because the tech debt is so large that we can't figure out a way to plug these things together. Excited for tools like this to eventually act as a layer that can run over these sort of problems, as bizarre a solution as it is from a compute perspective

This type of use case is exactly why are building https://github.com/OpenAdaptAI/OpenAdapt

Re: Using GPT-4 Vision with Vimium to browse the web

#123

Post by abrichr »

jatins wrote: post #58
snake_doc wrote: post #22

Ah, very similar to Adept’s[1] concept? Though, their product seems not yet ready. [1] https://www.adept.ai/

It's also a little insane to me that what Adept has been supposedly building for years with 300+ mil in funding can now be built in a day with Open AI APIs? I think Adept pivoted along the way but original concept was very similar to this.

Agreed! This is part of the motivation behind https://github.com/OpenAdaptAI/OpenAdapt

Re: Using GPT-4 Vision with Vimium to browse the web

#124

Post by abrichr »

FooBarWidget wrote: post #63

Many Dutch companies pay salaries by 1. receiving payslips from the accountant, and then 2. manually initiating bank transfers to each employee for the amount in the corresponding payslip, and then 3. manually initiating a bank transfer to the tax authority to pay the withholded salary taxes. This is completely useless manual labor. There should be no reason for this to be a manual procedure. And yet it's almost impo…

Thanks for the tip!

Automating repetitive GUI workflows is the goal of https://github.com/OpenAdaptAI/OpenAdapt

Re: Using GPT-4 Vision with Vimium to browse the web

#125

Post by gumballindie »

transistorfan wrote: post #119
gumballindie wrote: post #94

Earlier quoted context omitted.

Wow. Leaking confidential tax payer data.

I should have been clearer, it's between two apps that we host internally - applications on our own intranet cannot talk to each other. If you want to get any data out of either of these apps to the world, you need to do a manual export and email/usb which would obviously flag

Correct, but chat gpt reads screen data to be able to "click" around. So you would need to expose at least data that is displayed on screen to this external product.

Re: Using GPT-4 Vision with Vimium to browse the web

#126

Post by heroprotagonist »

pms wrote: post #116
Maxion wrote: post #110

Earlier quoted context omitted.

The speed at which this is moving at is mind boggling. This may become crazier than the dot.com boom.

Until you realize that it doesn't work well with less popular videos (any items really), because "Large Language Models Struggle to Learn Long-Tail Knowledge" [1]. [1] https://proceedings.mlr.press/v202/kandpal23a.html

Except in this case, the knowledge is 'how to search the web for X" instead of 'an understanding or familiarity with X'.

Re: Using GPT-4 Vision with Vimium to browse the web

#128

Post by silentguy »

Usually there are a lot of comments about how text is the best interface and it's making a comeback in the LLMs but in this case picture is the better medium since parsing the webpage js would prove too difficult. I think a screenshot of a webpage has a smaller footprint than the raw payloads (js, assets, etc.).

Re: Using GPT-4 Vision with Vimium to browse the web

#129

Post by sebastiennight »

ternaus wrote: post #112

Love the idea. It also shows that GPT-4V created a new angle in web scraping. I guess, this or similar code would be leveraged in many projects like: 1. Scrape XXX websites, say LinkedIn or Twitter use all types of methods in the DOM to prevent it, but fighting working well GPT-4V + OCR would be ultra hard. 2. Give me an analysis of what these XXX companies are doing. And this could be done for competitors, to unders…

It took me a while to get what you meant, because... I'm not sure "XXX websites" usually means what you intended to convey here :)

Re: Using GPT-4 Vision with Vimium to browse the web

#130

Post by circuit10 »

dangerwill wrote: post #109

How is this making your browsing experience any better? You still have to know what you want to do, and it is just faster to type Rick roll into youtube directly and click the links directly instead of having to type k, or vh, or whatever. You are just adding a useless chatgpt middleman between you and the browser that you likely spend all day in anyway and should be adept at navigating

It's a proof of concept for how it could do more complicated tasks
Post reply on HN