Live data from Hacker News

Using GPT-4 Vision with Vimium to browse the web

github.com

61–70 of 133 posts

Re: Using GPT-4 Vision with Vimium to browse the web

#61
I think vim is unintentionally a great “embodiment” for chatgpt. There’s nothing that can’t be done with a stream of text, and the internet is full of vimscript already

I started a similar experiment if anyone else is thinking along the same lines :)

https://github.com/LachlanGray/vim-agent

Re: Using GPT-4 Vision with Vimium to browse the web

#63
Many Dutch companies pay salaries by

1. receiving payslips from the accountant, and then

2. manually initiating bank transfers to each employee for the amount in the corresponding payslip, and then

3. manually initiating a bank transfer to the tax authority to pay the withholded salary taxes.

This is completely useless manual labor. There should be no reason for this to be a manual procedure. And yet it's almost impossible to automate this. The accountant portal either has no API, or it has an API but lets you download the data as PDF, and/or the API costs good money. The bank either has no API, or it requires you to sign up for a developer account as if you're going to publish a public app, when you're just looking to automate some internal procedures.

So the easiest way to pay salaries and taxes is still to hire a person to do it manually. Hopefully one day that won't be necessary anymore. I wouldn't trust an AI to actually initiate the bank transfers, but maybe they can just prepare the transactions and then a person has to approve the submission.

Re: Using GPT-4 Vision with Vimium to browse the web

#64

At my work there are a large contingent of people who essentially do manual data copying between legacy programs (govt), because the tech debt is so large that we can't figure out a way to plug these things together. Excited for tools like this to eventually act as a layer that can run over these sort of problems, as bizarre a solution as it is from a compute perspective

It's bizarre computationally, but at this point maybe we have to compare it to the alternative: hiring a person. At least the AI only consumes electricity (which is hopefully green), while a person consumes food (grown with mined fertilizers), or meat (which we know is really bad for the environment).

Re: Using GPT-4 Vision with Vimium to browse the web

#67

At my work there are a large contingent of people who essentially do manual data copying between legacy programs (govt), because the tech debt is so large that we can't figure out a way to plug these things together. Excited for tools like this to eventually act as a layer that can run over these sort of problems, as bizarre a solution as it is from a compute perspective

Funny that you and others on here don't seem to realize that literally everybody who uses the internet has the exact same data entry problem all the time. Blame it on "old software", but how about the entire internet? copying (or in most cases even worse: re-typing) form data from one location on the screen into yet another webform. Username, password, email address, physical address, credit card info etc etc. Some e…

> It's my number 1 frustration when using the internet (worse than ads) and I find it mind-blowing that this hasn't been solved yet with or without LLMs.

Simple: it's because not solving this problem is how our godawful industry makes most of its money. Empowering the user means relinquishing control over their "journey"[0]. Ergonomics means fewer opportunities to upsell or show ads.

I don't have the link handy, but I'm reminded of one of the earliest Windows user interface guidelines documents, back from Windows 95/98 era, which, in a section about theming/visual style, already recognized that they have to allow for full flexibility, because vendors will insist on fucking the experience up for the sake of branding anyway, and resisting it is futile[1].

--

[0] - I'm trying really hard to hold back my contempt towards terms like this, and the whole salesy way of viewing human-computer interactions.

[1] - They put it in much more polite terms, but the feeling of helplessness was already there.

Re: Using GPT-4 Vision with Vimium to browse the web

#69
post #40

Earlier quoted context omitted.

From OpenAI docs[1]: "For safety reasons, we have implemented a system to block the submission of CAPTCHAs." [1] https://platform.openai.com/docs/guides/vision

Yeah, I've been feeding screenshots from selenium to the vision API, and when I trigger bot detection on a website, chatgpt refuses to process the image.

It does solve, or at least try to solve, captchas for me. It gets like half the characters correct, it's very bad at it.

Re: Using GPT-4 Vision with Vimium to browse the web

#70

Earlier quoted context omitted.

Funny that you and others on here don't seem to realize that literally everybody who uses the internet has the exact same data entry problem all the time. Blame it on "old software", but how about the entire internet? copying (or in most cases even worse: re-typing) form data from one location on the screen into yet another webform. Username, password, email address, physical address, credit card info etc etc. Some e…

consistently filling out username and password is all I wanted from my password manager, but it turns out it handles credit card number and other bits of information for me as well.

I've used Bitwarden to faster fill out job applications.
Post reply on HN