Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT

thesequence.substack.com

Post reply on HN
Print view View original

261–270 of 527 posts

Re: The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT

#261

Post by blatant303 »

jonplackett wrote: post #236
FLT8 wrote: post #211

Earlier quoted context omitted.

This is what I get out of Vicuna-13b: Human: A glass door has 'push' written on it in mirror writing. To open the door should you 'push' or 'pull' it? Assistant: Since the word "push" is written in mirror writing on the glass door, you should actually "pull" the door open instead of "push" it. Mirror writing is a writing method where the characters are reversed, so when you see the word "push" written in mirror writi…

The difference for me with GPT-4 is its ‘understanding’ of the scene and its explanation of WHY you should push or pull. It talks an out a door with people approaching from different directions. It has some idea of what those people would be thinking. That seems different to just ‘mirror writing means do the opposite’.

I asked GPT4 to draw a dog or a skull in openscad and even though the end result was buggy, commenting things in the code here and there and making some volumes transparent I figured out he got it okay. For instance the dog had two eyes two ears one long nose (potatoids). It understood the symmetry of both pairs but was unable to place them at the right place. It's not like it was just misaligned, things were in the wrong planes, but they where there. As if he hadn't understood a face is like one face of a cube.

Re: The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT

#262

Post by pulvinar »

causality0 wrote: post #223
PoignardAzur wrote: post #190

Earlier quoted context omitted.

Wait, how does GPT-4 even... Does it benefit from its visual attention, or is it a case of "the question wasn't in GPT-3's training set but it was in GPT-4's"?

Wish I had access to 4. Chatgpt fails when I give it a problem like "you feel a tremor as you walk toward a glass door. When you approach the door you see a sign laying on the ground beyond the door that says "pull". Should you push or pull the door to open it? "

GPT-4 gets it "wrong" too, until you interrogate it and discover that it assumed that by "beyond" you meant beyond as in if you were approaching the door from the outside west, and the sign was laying outside but east of the door.

It's the old word-problem problem.

Re: The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT

#263

Post by pharrington »

greyface- wrote: post #157
Mathnerd314 wrote: post #145

Earlier quoted context omitted.

That's a contributor to the repo, not someone with commit access.

A contributor who is also a Facebook employee and co-author of the LLaMA paper, presumably speaking in official capacity.

You would not believe the amount of internal and public facing sabotage done by FB employees.

Re: The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT

#265

Post by alfor »

superkuh wrote: post #140
alfor wrote: post #131

What kind of GPU is needed to run the 65B models, can a consumer grade card do the job? Is it worth it? Or do you use cloud instances?

You can't fit a 65B even at 4bits (~50GB) on a consumer grade GPU. With an average geek's home desktop computer running 65B inference on a CPU with 64GB of system RAM you could expect about 1 token every 2 seconds.

Thanks, so to make usable you would need a GPU with at least 50GB. That probably mean A100 with 80GB at around 10K$.

Ok I understand why people use CPU and main memory.

After a quick check up you can rent a A100/80G at 1$-2$/h.

Re: The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT

#266

Post by whimsicalism »

int_19h wrote: post #245
sbierwagen wrote: post #238

Earlier quoted context omitted.

5950x is a CPU model. Integer-quantized models are generally run with CPU inference. For the larger models the problem then becomes generation time per token.

Quantized models are used aplenty with GPUs as well - 4-bit quantization is the only way you can squeeze llama-30b into 24Gb of VRAM (i.e. RTX 3090 or 4090). In fact, I would say that, at this point, most people running LLaMA locally are likely using 4-bit quantization regardless of model size and hardware, just to get the most out of the latter.

Most people running llama locally are doing CPU inference, period.

Re: The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT

#267

Post by alfor »

AeiumNE wrote: post #133
alfor wrote: post #131

What kind of GPU is needed to run the 65B models, can a consumer grade card do the job? Is it worth it? Or do you use cloud instances?

It's possible to run with a cpu and then use system memory.

Do you know what kind of speed you can expect with inference on CPU?

Re: The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT

#268

Post by bradleyjg »

superkuh wrote: post #118

I've spent an embarassing amount of time since the llamas leaked playing with them, the tools to run them, and writing wrappers for them. They are technically alternatives in the sense that they're incomparably better chat bots than anything in the past. But at least for the 30B and under versions (65B is too big for me to run), no matter what fine tuning is done (alpaca, gpt4all, vicuna, etc), the llamas themselves…

I haven’t seen to much discussion of what’s possible at various sizes for an early stage start up, which is a discussion I’d expect to see on yc.

Clearly a company with $5-5MM in the bank can’t train a competitive LLM from scratch but what would it cost to fine tune and/or run a 65B parameter model or a hypothetical future open source 165B parameter model?

Re: The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT

#269

Post by Aeolun »

drumhead wrote: post #228
jonplackett wrote: post #162

Earlier quoted context omitted.

The difference between 3.5 and 4 is gigantic even in my fairly limited experience. I gave them both some common sense tests and this one stuck out to me. Q: A glass door has ‘push’ written on it in mirror writing. Should you push or pull it GPT-3.5: If the word "push" is written in mirror writing on a glass door, you should push the door to open it GPT-4: Since the word "push" is written in mirror writing, it suggest…

I tried the same question with Bard, and it got it right but it also gave me the following: Here are some other things to keep in mind when opening a glass door: Use your hands to open the door. Do not use your feet or other body parts. Be careful not to break the glass. Glass doors can be very fragile, so it is important to be gentle when opening them. If you are unsure how to open a glass door, ask for help from so…

This feels like Bard has read too many US instruction manuals.

Re: The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT

#270

Post by whimsicalism »

famouswaffles wrote: post #242
PoignardAzur wrote: post #190

Earlier quoted context omitted.

Wait, how does GPT-4 even... Does it benefit from its visual attention, or is it a case of "the question wasn't in GPT-3's training set but it was in GPT-4's"?

It's pretty much looking like anything can be extracted from language. Some harder than others for sure but with enough scale it does look like eventually everything falls. Text only GPT-4 has a pretty solid understanding of space that 3.5 definitely lacks. You can see more thorough experiments in the microsoft agi paper where they test it's ability to track the visual space of a maze.

There is no such thing as text-only GPT-4 unless you are referring to at inference time.
Post reply on HN