Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Using GPT-4 Vision with Vimium to browse the web

github.com

Post reply on HN
Print view View original

91–100 of 133 posts

Re: Using GPT-4 Vision with Vimium to browse the web

#91

Post by yreg »

transistorfan wrote: post #6

At my work there are a large contingent of people who essentially do manual data copying between legacy programs (govt), because the tech debt is so large that we can't figure out a way to plug these things together. Excited for tools like this to eventually act as a layer that can run over these sort of problems, as bizarre a solution as it is from a compute perspective

A long, long time ago I worked on a small project for a major multinational grocery chain.

I made them a tool that parses an Excel file with a specific structure and calls some endpoints in their internal system to submit the data.

I was curious, so I asked how they are doing it currently. They led me to a computer at the back of their office. The wallpaper had two rectangles, one of them said MS EXCEL and the other said INTERNET EXPLORER. Then the person opened these apps, carefully positioned both windows exactly into those rectangles and ran some auto-clicker - the kind cheaters would use in RuneScape – which moved the cursor and copied and pasted the values from the Excel into the various forms on the website.

Amazing.

Re: Using GPT-4 Vision with Vimium to browse the web

#92

Post by arbuge »

manmal wrote: post #37
maccam912 wrote: post #25

I've been playing with a similar idea of screenshots and actions from GPT-4 Vision for browsing, but after trying and failing to overlay info in the screenshot, I ended up just getting the accessibility tree from playwright and sending that along as text so the model would know what options it had for interaction. In my case it seemed to work better, I see the creator is here and has a list of future ideas, maybe add…

Probably better to capture all the content and not just what fits on one screen. Most pages should fit as text (or HTML?) in the new extended token window.

Better watch token costs. The per token costs are lower now but even so a full context load still costs almost $4.

Re: Using GPT-4 Vision with Vimium to browse the web

#93

Post by nvm0n2 »

FooBarWidget wrote: post #63

Many Dutch companies pay salaries by 1. receiving payslips from the accountant, and then 2. manually initiating bank transfers to each employee for the amount in the corresponding payslip, and then 3. manually initiating a bank transfer to the tax authority to pay the withholded salary taxes. This is completely useless manual labor. There should be no reason for this to be a manual procedure. And yet it's almost impo…

That's just a bank problem. Certainly this isn't how payroll works for large companies. Banks usually let you upload XML files that define a set of SWIFT payments, this is how I do payroll even for a small company. The accountants supply the XML file too, presumably they have an app that generates it.

Re: Using GPT-4 Vision with Vimium to browse the web

#94

Post by gumballindie »

transistorfan wrote: post #6

At my work there are a large contingent of people who essentially do manual data copying between legacy programs (govt), because the tech debt is so large that we can't figure out a way to plug these things together. Excited for tools like this to eventually act as a layer that can run over these sort of problems, as bizarre a solution as it is from a compute perspective

Wow. Leaking confidential tax payer data.

Re: Using GPT-4 Vision with Vimium to browse the web

#97

Post by Valgrim »

yreg wrote: post #91
transistorfan wrote: post #6

At my work there are a large contingent of people who essentially do manual data copying between legacy programs (govt), because the tech debt is so large that we can't figure out a way to plug these things together. Excited for tools like this to eventually act as a layer that can run over these sort of problems, as bizarre a solution as it is from a compute perspective

A long, long time ago I worked on a small project for a major multinational grocery chain. I made them a tool that parses an Excel file with a specific structure and calls some endpoints in their internal system to submit the data. I was curious, so I asked how they are doing it currently. They led me to a computer at the back of their office. The wallpaper had two rectangles, one of them said MS EXCEL and the other…

I worked with a client who used a multi-millon dollar system for moving goods automatically into packaging stations. The system was built and maintained by a major european company. All the data was transfered automatically between systems normally, but one day, for some reason, there was an internal communication error inside the machine which caused a lot of packages to be sent without being recorded as such.

Now normally we would just have contacted the company and asked them for a data extraction so we could cross-reference the data. But since it wasn't clear who was at fault, and we knew it would take weeks for that extraction, we looked for an internal solution first.

Now there was a subsystem in the machine that worked only in Internet Explorer, with an old authentication scheme, that we could use to see the information we needed, so I, being the only person in the team without formal analysis training but having made my way there from a clerk job, knew exactly what to do.

I fired up the old IE, Excel, wrote in 5 minutes a VBA script that did exactly what you described, click there copy that etc, and 30 minutes later we had our extraction, and resolved the issue completely before the packages were even shipped.

All hail Excel.

Re: Using GPT-4 Vision with Vimium to browse the web

#98

Post by spaceman_2020 »

Roark66 wrote: post #49
transistorfan wrote: post #6

At my work there are a large contingent of people who essentially do manual data copying between legacy programs (govt), because the tech debt is so large that we can't figure out a way to plug these things together. Excited for tools like this to eventually act as a layer that can run over these sort of problems, as bizarre a solution as it is from a compute perspective

Whenever I hear about such a thing (people doing legacy system data extraction manually) I wonder if in every case someone got the estimate for the "proper" solution and just decided a bunch of people typing is cheaper? Integrating things like Chatgpt will still require people who know what they are doing to look at it, and I wouldn't be surprised if the first advice they give is "don't use chatgpt for it".

If the market forces work as they’re supposed to (not a given anymore), then corporations that adopt better tech will see better profits through lower expenses. And then the laggards will have to adapt or die.

Also remember that this is essentially v1 of the software- the Windows 95 of this adoption cycle

Re: Using GPT-4 Vision with Vimium to browse the web

#99

Post by poulpy123 »

ishan0102 wrote: post #4

Hey! Creator here, thanks for sharing! Let me know if anyone has questions and feel free to contribute, I've left some potential next steps in the README.

could it be used to make a bot that visit and parse websites to extrat relevant information without writing a parser for each websites ?
Post reply on HN