Live data from Hacker News

MiniGPT-4

minigpt-4.github.io

111–120 of 337 posts

Re: MiniGPT-4

#111
post #21

Earlier quoted context omitted.

I never know what to expect anymore. We live in a world where computers can describe paintings and write sonnets about them but a half-trillion dollar car company can't figure out how to parallel park with eight cameras.

DriveGPT as it hits 2 parked cars and runs over a dog: "Apologies for the confusion earlier. You are correct that the cars were in the planned local planner path..."

Sorry, yes, that was a dog.

Let me try again and this time I will definitely not hit anything.

Sorry, that was another dog.

Re: MiniGPT-4

#112

Earlier quoted context omitted.

You can just ask GPT how to do it. Much like a lot of web dev!

at some point someone makes a service where you can let AI take over your computer directly. Easier that way! Curling straight to shell taken to next level.

So...AutoGPT? Now with command-line access! Have fun :)

https://github.com/Significant-Gravitas/Auto-GPT/

Re: MiniGPT-4

#113
post #26

Earlier quoted context omitted.

Someone needs to write a buyer's guide for GPUs and LLMs. For example, what's the best course of action if don't need to train anything but do want to eventually run whatever model becomes the first local-capable equivalent to ChatGPT? Do you go with Nvidia for the CUDA cores or with AMD for more VRAM? Do you do neither and wait another generation?

Nvidia and the highest amount of vram you can get. Currently the 4090, the rumor is the 4090ti will have 48gb of vram, idk if its worth waiting or not. The more VRAM the higher paremeter count you can run all in memory (fastest by far). AMD is almost a joke in ML. The lack of CUDA support (which is nvidia proprietary) is straight lethal, and also even though ROCM does have much better support these days, from what I'…

That's good to know. A lot of people might look at the 7900XTX with its 24GB of ram for a grand and think "ah, a good deal for fitting an LLM in".

Re: MiniGPT-4

#114
post #86

Earlier quoted context omitted.

Depends on whether you just want to play around or have actual business needs, but to play around I'm waiting for the next generation of unified memory architectures

Can you point any source re: next generation of unified memory architectures? How far are they away?

Well currently there's just apple and they are super secretive about their roadmap, the M1/2 top line chip already perform quite well, and while the top version look expensive, a pair of a6000 will definitely eat more money both to purchase and to run.

M3 is around the corner tho, and there's some announcement to come from intel or arm following their partnership. There's also the new card coming from intel that is supposed to be aimed squarely at machine learning workloads, and they don't have to segment their market by memory sizing like Nvidia do, but they aren't well supported as device targets, but a pair of these will likely be very cost effective if and only if they will get credible compatibility with the libraries and models

Re: MiniGPT-4

#115
post #6

On a technical level, they're doing something really simple -- take BLIP2's ViT-L+Q-former, connect it to Vicuna-13B with a linear layer, and train just the tiny layer on some datasets of image-text pairs. But the results are pretty amazing. It completely knocks Openflamingo && even the original blip2 models out of the park. And best of all, it arrived before OpenAI's GPT-4 Image Modality did. Real win for Open Sourc…

Indeed, really simple. And yes, the results are shockingly good. But what I find most remarkable about this is that the ViT-L+Q-former's hidden states are related by only a linear projection (plus bias) to the Vicuna-13B's token embeddings:

  emb_in_vicuna_space = emb_in_qformer_space @ W + B
These two models are trained independently of each other, on very different data (RGB images vs integer token ids representing subwords), and yet somehow they learn to embed different data in feature vectors that are so... similar. WHY should that be the case?

It suggests to me there may be something universal about the embedding layers and hidden states of all trained deep learning models.

Re: MiniGPT-4

#116
post #14
post #6

On a technical level, they're doing something really simple -- take BLIP2's ViT-L+Q-former, connect it to Vicuna-13B with a linear layer, and train just the tiny layer on some datasets of image-text pairs. But the results are pretty amazing. It completely knocks Openflamingo && even the original blip2 models out of the park. And best of all, it arrived before OpenAI's GPT-4 Image Modality did. Real win for Open Sourc…

> they're doing something really simple -- take BLIP2's ViT-L+Q-former, connect it to Vicuna-13B with a linear layer, and train just the tiny layer on some datasets of image-text pairs Oh yes. Simple! Jesus, this ML stuff makes a humble web dev like myself feel like a dog trying to read Tolstoy.

FWIW I work in LLMs and I consistently fail to do simple webdev stuff

Re: MiniGPT-4

#117
post #58
post #14

Earlier quoted context omitted.

> they're doing something really simple -- take BLIP2's ViT-L+Q-former, connect it to Vicuna-13B with a linear layer, and train just the tiny layer on some datasets of image-text pairs Oh yes. Simple! Jesus, this ML stuff makes a humble web dev like myself feel like a dog trying to read Tolstoy.

Web stuff probably makes ML devs feel the same way. ML is just a different field, using a different set of technologies from those you’re familiar with.

The best ML PhDs can’t do what frontend devs can: understand CSS :D

Re: MiniGPT-4

#118
post #76

Earlier quoted context omitted.

In practice, it's a lot more like web dev than you might imagine. The above means that the approach is web-dev like gluing, almost literally just, from existingliba import someop from existinglibb import anotherop from someaifw import glue a = someop(X) b = glue(a) Y = anotherop(b)

It's more like gardening: 1. plant seed 2. ...wait a very long time... 3. observe completely unexpected but cool result The unexpected part of step 3 is what makes this very different from any kind of engineering, even webdev. Of course, there is a lot of engineering involved in good ML, but that is more comparable to agricultural engineering in the sense that it's just a lot of dumb plumbing that any engineer can do…

I mean, for me, the unexpected part of 3 is what got me into programming in general. The first time you type a mysterious incantation into an editor and a few more mysterious incantations into the console and the console prints "Hello, world" like it was supposed to, it's unexpected because it's hard to believe that any of this mysterious incantation stuff actually works at all.

As you get better at programming you have to take on harder problems to create the surprise of something working, because you gain confidence, and as you gain confidence, you start expecting your code to work. It's only when you've compiled the thing 6 times with small corrections and gotten segfaults each time and the 7th time you finally find the place you weren't updating the pointer and you correct it, but this is the 7th error you've corrected without the segfault going away, so you don't really expect it to fix the problem, but then you run it and it's fixed!

And then you get a job and the reality is that most of the jobs you're just writing CRUD apps and for a little while you can get some surprise out of learning the frameworks, but eventually you actually get really, really knowledgeable about the Postrgres/Django/React stack and nothing surprises you any more, but because nothing surprises you any more, you're really effective and you start being able to bill the big bucks but only for work on that stack because it takes time to struggle enough to get surprised, and the time that takes means your time is worth less to your clients. Money ruins everything. And if you don't do anything non-billable, it's easy to forget what programming felt like when you didn't know how your tools all worked inside and out. Not everyone takes this path but it's certainly the easiest path to take.

I think for a lot of folks who have been doing this for a long time, the reason ML is so exciting is it's getting them back out of their comfort zone, and into a space where they can experience surprise again.

But that surprise has always been available if you continue to find areas of programming that push you out of your comfort zone. For me it's been writing compilers/interpreters for programming languages. Crafting Interpreters was awesome: for the first time I benchmarked a program written in my language against a Python program, and my program was faster: I never expected I'd be able to do that! More recently, I wrote a generational GC. It's... way too memory-intensive to be used in my language which uses one-GC-per-thread for potentially millions of threads, but it certainly was a surprise when that worked.

Personally, I'm keeping track of ML enough to know broad strokes of things but I'm not getting my hands dirty with code until there are some giants to stand on the shoulders of. Those may already exist but it's not clear who they are yet. And I've got very little interest in plugging together opaque API components; I know how to make an API call. I want to write the model code and train it myself.

Re: MiniGPT-4

#119
post #59

Earlier quoted context omitted.

Near term it’s a frustrating decision, but if these gpt4 vision LLMs are anything to go by it will prove to be the right decision in the long term.

Remember that "cameras" aren't as good as human perception because human eyes interact with the environment instead of being passive sensors. (That is, if you can't see something you can move your head.) Plus we have ears, are under a roof so can't get rained on, are self cleaning, temperature regulating, have much better dynamic range, wear driving glasses…

And we still get into millions of accidents every year…

Re: MiniGPT-4

#120
A modified BLIP2 that works with diffusion models rather than vision transformers would be pretty cool. Using Vicuna-13B or another large language model as the language model of Stable Diffusion, instead of the text encoder of CLIP, would be a game changer, it would completely change how you prompt Stable Diffusion from something kinda like a tag list to it being able to follow instructions in English instead.
Post reply on HN