Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT

thesequence.substack.com

Post reply on HN
Print view View original

171–180 of 527 posts

Re: The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT

#171

Post by sp332 »

apetresc wrote: post #135
tyfon wrote: post #66

Earlier quoted context omitted.

> Why do these takes around open-source AI remain so popular? I can only speak for myself, but I have a great desire to run these things locally, without network and without anyone being able to shut me out of it and without a running cost except the energy needed for the computations. Putting powerful models behind walls of "political correctness" and money is not something that fits well with my personal beliefs. T…

How have you managed to run the 65B model? Cloud resources, or you have a very kitted-out homelab?

If you're not running on GPU, you can upgrade your system RAM instead of finding a card with lots of VRAM. 64GB of DDR4 is only $120.

Re: The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT

#172

Post by [deleted] »

jhbadger wrote: post #37
anonzzzies wrote: post #31

Earlier quoted context omitted.

It is vastly better than anything else so far though. The rest will catch up but openai is not sleeping and they are well funded.

I thought that was the case before trying Vicuna. I agree that LLaMA and Alpaca are inferior to ChatGPT but I'm really not sure Vicuna is. It even (unfortunately) copies some of ChatGPT's quirks, like getting prudish when asking it to write a love scene ("It would not be appropriate for me to write...")

[deleted]

Re: The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT

#173

Post by danShumway »

superkuh wrote: post #129
cubefox wrote: post #122

Earlier quoted context omitted.

Your conclusion seems not to be warranted since you haven't tried out the 65B model.

I agree, but I think my experience is representative. So far most human people don't have the resources to be able to use 65B. And most small companies / university groups don't have the resources to fine-tune a 65B. I've talked to a couple dozen people in real time who've played with up to 30B but no one I know has the resources to run the 65B at all or fast enough to actually use and get an opinion of. None of the…

This is a good point. Even if ordinary people did have the resources to run the 65B well on their existing devices, the speed would limit its usefulness quite a bit. In practice, 30B is what most people are going to interact with (if even, I've seen a lot of projects use 13B).

My experience here is pretty similar. I'm heavily (emotionally at least) invested in models running locally, I refuse to build something around a remote AI that I can only interact with through an API. But I'm not going to pretend that LLaMA has been amazing locally. I really couldn't figure out what to build with it that would be useful.

I'm vaguely hoping that compression actually gets better and that targeted reinforcement/alignment training might change that. GPT can handle a wide range of tasks, but for a smaller AI it wouldn't be too much of a problem to have a much more targeted domain, and at that point maybe the 30B model is actually good enough if it's been refined around a very specific problem domain.

For that to happen, training needs to get more accessible though. Or communities need to start getting together and deciding to build very targeted models and then distributing the weights as "plug-and-play" models you can swap out for different tasks.

And if there's a way to get 65B more accessible, that would be great too.

Re: The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT

#174

Post by trifurcate »

apetresc wrote: post #135
tyfon wrote: post #66

Earlier quoted context omitted.

> Why do these takes around open-source AI remain so popular? I can only speak for myself, but I have a great desire to run these things locally, without network and without anyone being able to shut me out of it and without a running cost except the energy needed for the computations. Putting powerful models behind walls of "political correctness" and money is not something that fits well with my personal beliefs. T…

How have you managed to run the 65B model? Cloud resources, or you have a very kitted-out homelab?

All you need is 2 3090s.

Re: The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT

#175

Post by sp332 »

jhbadger wrote: post #159
Tepix wrote: post #87

Earlier quoted context omitted.

They are all built on top of Llama…

Yes, but that doesn't matter mow. The University of California has released Vicuna as open source. It doesn't need the Llama model to be installed at this point. Nor do you need any of Meta's code to run it either as you can use llama.cpp (not created by Meta). That's the whole point of the article. It's open source now. There's nothing Meta can do.

This is incorrect. According to the official https://github.com/lm-sys/FastChat#vicuna-weights you need the original Llama weights before applying the Vicuna diff.

Re: The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT

#176

Post by matthewdgreen »

superkuh wrote: post #129
cubefox wrote: post #122

Earlier quoted context omitted.

Your conclusion seems not to be warranted since you haven't tried out the 65B model.

I agree, but I think my experience is representative. So far most human people don't have the resources to be able to use 65B. And most small companies / university groups don't have the resources to fine-tune a 65B. I've talked to a couple dozen people in real time who've played with up to 30B but no one I know has the resources to run the 65B at all or fast enough to actually use and get an opinion of. None of the…

Is it hard to spin up an appropriate EC2 instance with 64GB+ of additional RAM? The m6a.8xlarge seems to have 128GB and costs $1.38 per hour. Was going to try this myself, but now I'm wondering if even that won't be worth the trouble. (I know this is not "your own devices" but would give a feel for what you're missing with the smaller models.)

Re: The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT

#177

Post by sp332 »

throwawayapples wrote: post #83

"The training and serving code, along with an online demo, are publicly available for non-commercial use." (from Vicuna's home page.) In what universe is that "open source"?!

They also said they have no plans to release the dataset, so it's not reproduceable either.

Re: The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT

#178

Post by YetAnotherNick »

itake wrote: post #88
YetAnotherNick wrote: post #82

Earlier quoted context omitted.

It is just a copyright violation. My guess is that it would be fine if you use already scraped data as you haven't accepted TOS, but they have every right to block you or access to your business if you violate this.

I thought the copyright office said that ai generated material isn’t copyrighted?

Sorry, can't edit my comment now. I meant it is "not a copyright violation, but just a TOS violation."

Re: The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT

#179

Post by danShumway »

seydor wrote: post #166
superkuh wrote: post #118

I've spent an embarassing amount of time since the llamas leaked playing with them, the tools to run them, and writing wrappers for them. They are technically alternatives in the sense that they're incomparably better chat bots than anything in the past. But at least for the 30B and under versions (65B is too big for me to run), no matter what fine tuning is done (alpaca, gpt4all, vicuna, etc), the llamas themselves…

> (just a hobby, won't be big and professional like gnu) Llamas are creating the linux of AI and the ecosystem around it. Even though openAI has a head start, this whole thing is just starting. Llammas are showing the world that it doesn't take monopoly-level hardware to run those things. And because it's fun , like, video-game-fun there is going to be a lot of attention on them. Running a fully-owned, uncensored cha…

This is my hope as well. It would be disastrous if the future of AI is one where only megacorps can run it and where they control all access to it. In that sense, LLaMA is really encouraging and I'm seriously rooting for it to improve.

It's just not there yet. I tend to be kind of bearish on LLMs in general, I think there's a lot more hype than is warranted, and people are overlooking some pretty significant downsides like prompt-injection that are going to end up making them a lot harder to use in ubiquitous contexts in practice, but... I mean, the big LLMs (even GPT-3.5) are definitely still in a class above LLaMA. I understand why they're hyped.

I look at GPT and think, "I'm not sure this is worth the trouble of using." But I look at LLaMA and I'm not sure how/where to use it at all. It's a whole different level of output.

But that doesn't mean I'm not rooting for the "hobbyists" to succeed. And it doesn't mean LLaMA can't succeed, it doesn't necessarily need to be better than GPT-4, it just needs to be good enough at a lot of the stuff GPT-4 does to be usable, and to have the accessibility and access outweigh everything else. It's just not there yet.

Re: The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT

#180

Post by muyuu »

Szpadel wrote: post #148
muyuu wrote: post #141

Earlier quoted context omitted.

his conclusion is simultaneously not warranted and correct a like-for-like comparison would be GPT-4 against the larger models like LLaMA 65B, but those cannot be run on consumer-grade hardware so one ends up comparing the stuff one can run... against the top stuff from OpenAI running on high-end GPU farms, and this technology clearly benefits a lot still from much larger scale than most people can afford the great r…

if "leaks" are true then even a 65B model is a toy when GPT-4 is supposed to be 1000B I would really like to have an open source model that could compete with even gpt3, but basically all lammas I tried failed to create correct email validation regex, where gpt3 is able to even refer to what rfc it is

13B on my computer does produce useful output, it's pretty hit and miss though

GPT-3.5 OTOH is much better, but it's also much better at producing convincing-sounding but completely incorrect answers

Post reply on HN