The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT • Hacker News Board

Re: The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT

#161

Post by [deleted] » Sun, Apr 09, 2023, 9:29 PM UTC

ShannonLimiter wrote: post #117

isoprophlex wrote: post #94

Earlier quoted context omitted.

I find the thing incredibly smart and yet utterly useless at times. I just spent 20 minutes getting the current iteration of ChatGPT to agree with me that a certain sentence is palindromic. Even when you make it print the unaccented characters one by one, spaces excluded, backwards and forwards, it still insists "Élu par cette crapule" isn't palindromic. I understand how tokenization makes this difficult but come on.…

Just ask it to figure it out step-by-step and to remove accents: GPT-4: Figure this out step by step and check your work: Remove the accents and tell me if this a palindrome: Élu par cette crapule GPT-3.5 (Needs to know that removing spaces and lowercasing is okay): Figure this out step by step and check your work: Lowercase it and remove the accents and spaces and tell me if this a palindrome: Élu par cette crapule

[deleted]

Re: The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT

#162

Post by jonplackett » Sun, Apr 09, 2023, 9:29 PM UTC

superkuh wrote: post #118

I've spent an embarassing amount of time since the llamas leaked playing with them, the tools to run them, and writing wrappers for them. They are technically alternatives in the sense that they're incomparably better chat bots than anything in the past. But at least for the 30B and under versions (65B is too big for me to run), no matter what fine tuning is done (alpaca, gpt4all, vicuna, etc), the llamas themselves…

The difference between 3.5 and 4 is gigantic even in my fairly limited experience.

I gave them both some common sense tests and this one stuck out to me.

Q: A glass door has ‘push’ written on it in mirror writing. Should you push or pull it

GPT-3.5:

If the word "push" is written in mirror writing on a glass door, you should push the door to open it

GPT-4:

Since the word "push" is written in mirror writing, it suggests that the instruction is intended for people on the other side of the door. Therefore, you should pull the door to open it.

Re: The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT

#163

Post by morrbo » Sun, Apr 09, 2023, 9:30 PM UTC

lolinder wrote: post #154

Scene_Cast2 wrote: post #150

Earlier quoted context omitted.

Could you quantify "quite slow"?

A token per second-ish with a Ryzen 7 5800X. If I run it for too long it gets slower as heat throttling kicks in, I need a better cooling system if I'm going to run it non-stop.

i've had the same experience tbh, 7/13/30 on ryzen (local) and intel (server) both on rhel/centos. It's a shame really

Re: The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT

#164

Post by tyfon » Sun, Apr 09, 2023, 9:35 PM UTC

kristianp wrote: post #134

tyfon wrote: post #66

Earlier quoted context omitted.

> Why do these takes around open-source AI remain so popular? I can only speak for myself, but I have a great desire to run these things locally, without network and without anyone being able to shut me out of it and without a running cost except the energy needed for the computations. Putting powerful models behind walls of "political correctness" and money is not something that fits well with my personal beliefs. T…

How do you run it locally? llama.cpp + 64GB RAM + 4bit quantized?

I have a 5950x with 64 gb ram and they are quantized to 4 bit yes :)

The weights are stored on a samsung 980 pro so the load time is very fast too. I get about 2 tokens/second with this setup.

edit: forgot to confirm, it is llama.cpp

edit2: I am going to try the FP16 version after easter as I ordered 64 GB of additional ram. But I suspect the speed will be abyssal with the 5950x having to calculate through 120 gb of weights. Hopefully some smart person will come up with a way to allow the GPU to run off system memory via the amd infinity fabric or something.

Re: The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT

#165

Post by barbariangrunge » Sun, Apr 09, 2023, 9:36 PM UTC

greyface- wrote: post #157

Mathnerd314 wrote: post #145

Earlier quoted context omitted.

That's a contributor to the repo, not someone with commit access.

A contributor who is also a Facebook employee and co-author of the LLaMA paper, presumably speaking in official capacity.

Why would you presume that by default? Need a quote to conclude that

Re: The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT

#166

Post by seydor » Sun, Apr 09, 2023, 9:37 PM UTC

superkuh wrote: post #118

I've spent an embarassing amount of time since the llamas leaked playing with them, the tools to run them, and writing wrappers for them. They are technically alternatives in the sense that they're incomparably better chat bots than anything in the past. But at least for the 30B and under versions (65B is too big for me to run), no matter what fine tuning is done (alpaca, gpt4all, vicuna, etc), the llamas themselves…

> (just a hobby, won't be big and professional like gnu)

Llamas are creating the linux of AI and the ecosystem around it. Even though openAI has a head start, this whole thing is just starting. Llammas are showing the world that it doesn't take monopoly-level hardware to run those things. And because it's fun, like, video-game-fun there is going to be a lot of attention on them. Running a fully-owned, uncensored chat is the kind of thing that gets people creative

Re: The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT

#167

Post by barbariangrunge » Sun, Apr 09, 2023, 9:38 PM UTC

guwop wrote: post #114

halotrope wrote: post #106

Earlier quoted context omitted.

You can just ask if there is no output filtering

The future is going to be hilarious. Just ask the model who made it!

Does the model know, or will it just hallucinate an answer?

Re: The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT

#168

Post by barbariangrunge » Sun, Apr 09, 2023, 9:40 PM UTC

tyfon wrote: post #164

kristianp wrote: post #134

Earlier quoted context omitted.

How do you run it locally? llama.cpp + 64GB RAM + 4bit quantized?

I have a 5950x with 64 gb ram and they are quantized to 4 bit yes :) The weights are stored on a samsung 980 pro so the load time is very fast too. I get about 2 tokens/second with this setup. edit: forgot to confirm, it is llama.cpp edit2: I am going to try the FP16 version after easter as I ordered 64 GB of additional ram. But I suspect the speed will be abyssal with the 5950x having to calculate through 120 gb of…

I thought it needed 64gb of vram. 64gb of ram is easy to obtain

Re: The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT

#169

Post by [deleted] » Sun, Apr 09, 2023, 9:41 PM UTC

tyfon wrote: post #66

whimsicalism wrote: post #11

It appears there is this genre of articles pretending that LLAMA or its RL-HF tuned variants are somehow even close to an alternative to ChatGPT. Spending more than a few moments interacting even with the larger instruct-tuned variants of these models quickly dispels that idea. Why do these takes around open-source AI remain so popular? What is the driving force?

> Why do these takes around open-source AI remain so popular? I can only speak for myself, but I have a great desire to run these things locally, without network and without anyone being able to shut me out of it and without a running cost except the energy needed for the computations. Putting powerful models behind walls of "political correctness" and money is not something that fits well with my personal beliefs. T…

[deleted]

Re: The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT

#170

Post by digitallyfree » Sun, Apr 09, 2023, 9:42 PM UTC

whimsicalism wrote: post #11

It appears there is this genre of articles pretending that LLAMA or its RL-HF tuned variants are somehow even close to an alternative to ChatGPT. Spending more than a few moments interacting even with the larger instruct-tuned variants of these models quickly dispels that idea. Why do these takes around open-source AI remain so popular? What is the driving force?

Privacy and security is a big desire for the people running these lower-quality local models. These assistants are becoming more and more powerful and people want to use them for personal tasks. For instance someone might want to chat about their inner feelings or ask the AI to pen a confidential letter, things that they wouldn't want to send to a cloud service. We saw the same thing with Stable Diffusion and how people would spend a lot of effort getting it to run on their machines so they wouldn't have to rely on a public instance.

I'm talking about individual people here as the fact that this is a leak means that corps probably won't take the legal risk of trying this out (maybe some are doing so in secret). In the business world there definitely is a want for locally hosted models for employees that can safely handle confidential inputs and outputs.

The Llama models are not as good as ChatGPT but there are new variants like Alpaca and Vicuna with improved quality. People are actively using them already to help with writing and as chatbots.