Live data from Hacker News

Using GPT-4 Vision with Vimium to browse the web

github.com

111–120 of 133 posts

Re: Using GPT-4 Vision with Vimium to browse the web

#111

Earlier quoted context omitted.

Isn't the keyboard connected to the computer via USB? If I have access to the keyboard, I have access to a USB cable plugged into the computer, right? Perhaps I misunderstand something....

I meant the reverse; the computer attacking the robot using it

Uhhhhh, thanks. That makes a lot of sense!

Re: Using GPT-4 Vision with Vimium to browse the web

#112
Love the idea.

It also shows that GPT-4V created a new angle in web scraping.

I guess, this or similar code would be leveraged in many projects like:

1. Scrape XXX websites, say LinkedIn or Twitter use all types of methods in the DOM to prevent it, but fighting working well GPT-4V + OCR would be ultra hard.

2. Give me an analysis of what these XXX companies are doing. And this could be done for competitors, to understand the landscape of some industry, or even plainly to get news.

Large-scale scrapping, not depending on the source code of the pages is a powerful infrastructural change.

Re: Using GPT-4 Vision with Vimium to browse the web

#114
post #97
post #91

Earlier quoted context omitted.

A long, long time ago I worked on a small project for a major multinational grocery chain. I made them a tool that parses an Excel file with a specific structure and calls some endpoints in their internal system to submit the data. I was curious, so I asked how they are doing it currently. They led me to a computer at the back of their office. The wallpaper had two rectangles, one of them said MS EXCEL and the other…

I worked with a client who used a multi-millon dollar system for moving goods automatically into packaging stations. The system was built and maintained by a major european company. All the data was transfered automatically between systems normally, but one day, for some reason, there was an internal communication error inside the machine which caused a lot of packages to be sent without being recorded as such. Now n…

For all its flaws as a programming language, VBA made an excellent bodging language and I salute your expedient field hack.

Re: Using GPT-4 Vision with Vimium to browse the web

#115
i think this is actively dangerous. well not yet. but getting there.

i know - ai isn't meant to be sentient. but if it looks like a duck and quacks like a duck...

how do i know that the comments here aren't done by dedicated hacker news ai bots?

the potential danger could come from lack of supervision down the road.

i didn't get much sleep last night so this is less coherent than it could be.

Re: Using GPT-4 Vision with Vimium to browse the web

#116
post #110
post #104

It's insane that this is now possible: https://github.com/ishan0102/vimGPT/blob/682b5e539541cd6d710... > "You need to choose which action to take to help a user do this task: {objective}. Your options are navigate, type, click, and done. Navigate should take you to the specified URL. Type and click take strings where if you want to click on an object, return the string with the yellow character sequence you want to c…

The speed at which this is moving at is mind boggling. This may become crazier than the dot.com boom.

Until you realize that it doesn't work well with less popular videos (any items really), because "Large Language Models Struggle to Learn Long-Tail Knowledge" [1].

[1] https://proceedings.mlr.press/v202/kandpal23a.html

Re: Using GPT-4 Vision with Vimium to browse the web

#117
post #50

How will tools like this affect web tracking or generally advertisements on the internet? Imagine you could have an agent browse the web for you and fetch exactly what you are seraching for without you seeing any ads/pop ups or being tracked along the way! Could be a great ”ad blocker”! Could it perhaps also make SEO useless and thus improve the quality of internet? But I wonder if it also could have negative effects…

Since this is sending screenshots of pages to GPT, won't it see the ads as well?

Re: Using GPT-4 Vision with Vimium to browse the web

#118

Earlier quoted context omitted.

Funny that you and others on here don't seem to realize that literally everybody who uses the internet has the exact same data entry problem all the time. Blame it on "old software", but how about the entire internet? copying (or in most cases even worse: re-typing) form data from one location on the screen into yet another webform. Username, password, email address, physical address, credit card info etc etc. Some e…

consistently filling out username and password is all I wanted from my password manager, but it turns out it handles credit card number and other bits of information for me as well.

Doesn't chrome out of the box handle all of that?

Re: Using GPT-4 Vision with Vimium to browse the web

#119

At my work there are a large contingent of people who essentially do manual data copying between legacy programs (govt), because the tech debt is so large that we can't figure out a way to plug these things together. Excited for tools like this to eventually act as a layer that can run over these sort of problems, as bizarre a solution as it is from a compute perspective

Wow. Leaking confidential tax payer data.

I should have been clearer, it's between two apps that we host internally - applications on our own intranet cannot talk to each other. If you want to get any data out of either of these apps to the world, you need to do a manual export and email/usb which would obviously flag
Post reply on HN