Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Using GPT-4 Vision with Vimium to browse the web

github.com

Post reply on HN
Print view View original

101–110 of 133 posts

Re: Using GPT-4 Vision with Vimium to browse the web

#101

Post by musha68k »

TeMPOraL wrote: post #67
bboygravity wrote: post #55

Earlier quoted context omitted.

Funny that you and others on here don't seem to realize that literally everybody who uses the internet has the exact same data entry problem all the time. Blame it on "old software", but how about the entire internet? copying (or in most cases even worse: re-typing) form data from one location on the screen into yet another webform. Username, password, email address, physical address, credit card info etc etc. Some e…

> It's my number 1 frustration when using the internet (worse than ads) and I find it mind-blowing that this hasn't been solved yet with or without LLMs. Simple: it's because not solving this problem is how our godawful industry makes most of its money. Empowering the user means relinquishing control over their "journey"[0]. Ergonomics means fewer opportunities to upsell or show ads. I don't have the link handy, but…

Ted Nelson’s “intertwingularity” isn’t far off from the data entry problem described. He argues for universal data access where duplication is obsolete. Imagine form data as a single, linkable object across the web, editable in one place, reflected everywhere—no re-typing, just seamless auto-fill. That’s the unrealized potential of hypertext.

Re: Using GPT-4 Vision with Vimium to browse the web

#102

Post by simbolit »

yjftsjthsd-h wrote: post #24
aikinai wrote: post #16

Earlier quoted context omitted.

I remember years ago thinking it was weird in Ghost in the Shell when a robot had fingers on its fingers to type really fast. Maybe that really won’t happen since they can plug into USB at least, but they will probably use the screen and keyboard input sometimes at least.

USB is an attack vector; if it's not exploiting your USB driver it's connecting your data pins to mains power. Keyboards are an air gap.

Isn't the keyboard connected to the computer via USB?

If I have access to the keyboard, I have access to a USB cable plugged into the computer, right?

Perhaps I misunderstand something....

Re: Using GPT-4 Vision with Vimium to browse the web

#103

Post by gsuuon »

lachlan_gray wrote: post #61

I think vim is unintentionally a great “embodiment” for chatgpt. There’s nothing that can’t be done with a stream of text, and the internet is full of vimscript already I started a similar experiment if anyone else is thinking along the same lines :) https://github.com/LachlanGray/vim-agent

This is a neat idea!

Re: Using GPT-4 Vision with Vimium to browse the web

#104

Post by e12e »

It's insane that this is now possible:

https://github.com/ishan0102/vimGPT/blob/682b5e539541cd6d710...

> "You need to choose which action to take to help a user do this task: {objective}. Your options are navigate, type, click, and done. Navigate should take you to the specified URL. Type and click take strings where if you want to click on an object, return the string with the yellow character sequence you want to click on, and to type just a string with the message you want to type. For clicks, please only respond with the 1-2 letter sequence in the yellow box, and if there are multiple valid options choose the one you think a user would select. For typing, please return a click to click on the box along with a type with the message to write. When the page seems satisfactory, return done as a key with no value. You must respond in JSON only with no other fluff or bad things will happen. The JSON keys must ONLY be one of navigate, type, or click. Do not return the JSON inside a code block."

Re: Using GPT-4 Vision with Vimium to browse the web

#105

Post by kspacewalk2 »

yreg wrote: post #91
transistorfan wrote: post #6

At my work there are a large contingent of people who essentially do manual data copying between legacy programs (govt), because the tech debt is so large that we can't figure out a way to plug these things together. Excited for tools like this to eventually act as a layer that can run over these sort of problems, as bizarre a solution as it is from a compute perspective

A long, long time ago I worked on a small project for a major multinational grocery chain. I made them a tool that parses an Excel file with a specific structure and calls some endpoints in their internal system to submit the data. I was curious, so I asked how they are doing it currently. They led me to a computer at the back of their office. The wallpaper had two rectangles, one of them said MS EXCEL and the other…

I wonder if it used something like AutoIt[0]. I remember using it at one of my more boring co-op jobs about 20 years ago to automate moving data between a spreadsheet and some obscure database product.

[0] https://en.wikipedia.org/wiki/AutoIt

Re: Using GPT-4 Vision with Vimium to browse the web

#106

Post by DalasNoin »

I tried to use it, but unfortunately it often did not add the little annotations for the different options to the screen and it got stuck in a loop. This bot works by adding a two letter combination to each clickable option, but sometimes they don't show up. It managed to sign in to twitter ones, but really quickly I burned through the 100 images api limit.

Maybe for a future version it only uses vision for difficult situations in which it gets stuck and otherwise uses the text based browser?

Re: Using GPT-4 Vision with Vimium to browse the web

#107

Post by yjftsjthsd-h »

simbolit wrote: post #102
yjftsjthsd-h wrote: post #24

Earlier quoted context omitted.

USB is an attack vector; if it's not exploiting your USB driver it's connecting your data pins to mains power. Keyboards are an air gap.

Isn't the keyboard connected to the computer via USB? If I have access to the keyboard, I have access to a USB cable plugged into the computer, right? Perhaps I misunderstand something....

I meant the reverse; the computer attacking the robot using it

Re: Using GPT-4 Vision with Vimium to browse the web

#109

Post by dangerwill »

How is this making your browsing experience any better? You still have to know what you want to do, and it is just faster to type Rick roll into youtube directly and click the links directly instead of having to type k, or vh, or whatever. You are just adding a useless chatgpt middleman between you and the browser that you likely spend all day in anyway and should be adept at navigating

Re: Using GPT-4 Vision with Vimium to browse the web

#110

Post by Maxion »

e12e wrote: post #104

It's insane that this is now possible: https://github.com/ishan0102/vimGPT/blob/682b5e539541cd6d710... > "You need to choose which action to take to help a user do this task: {objective}. Your options are navigate, type, click, and done. Navigate should take you to the specified URL. Type and click take strings where if you want to click on an object, return the string with the yellow character sequence you want to c…

The speed at which this is moving at is mind boggling. This may become crazier than the dot.com boom.
Post reply on HN