Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Coming of Local LLMs

nickarner.com

Post reply on HN
Print view View original

91–100 of 214 posts

Re: The Coming of Local LLMs

#91

Post by whimsicalism »

seydor wrote: post #89
whimsicalism wrote: post #69

It appears there is this genre of articles pretending that LLAMA or its RL-HF tuned variants are somehow even close to an alternative to ChatGPT. Spending more than a few moments interacting even with the larger instruct-tuned variants of these models quickly dispels that idea. Why do these takes around open-source AI remain so popular? What is the driving force? I've posted this before, but it seems like this genre…

You make it sound as if ChatGPT itself isnt severly limited. Vicuna and newer 13B models are quite close . And the uncensored models have a capability that ChatGPT will never have.

They just aren't quite close. Maybe you guys are asking shallow questions, but I'm asking subject matter questions and this is just not true.

It would be great if we could have detailed QA evaluation to show this, but of course then the open source people would train their models on it as a fine-tuning datasaet.

Re: The Coming of Local LLMs

#92

Post by whimsicalism »

simion314 wrote: post #83
whimsicalism wrote: post #69

It appears there is this genre of articles pretending that LLAMA or its RL-HF tuned variants are somehow even close to an alternative to ChatGPT. Spending more than a few moments interacting even with the larger instruct-tuned variants of these models quickly dispels that idea. Why do these takes around open-source AI remain so popular? What is the driving force? I've posted this before, but it seems like this genre…

People are enthusiastic about the possibilities, imagine a black box on your desk contains only RAM and matrix multiplication chips, you install on it your favorite AI assistant and you train it with your private data/code, you remove all prudish restrictions and get productive on your work and on your off times. Llama has the potential to reach ChatGPT it needs tunning to get better at responding to questions, llama…

I am similarly enthusiastic and super, super impressed by Llama.cpp

But I just don't see the need to mislead and suggest that these models are "on par" with ChatGPT or something like that. They just aren't.

Re: The Coming of Local LLMs

#93

Post by gigel82 »

I don't understand why people are so excited to build this big thing on top of Llama, which is closed source, severely license restricted and we now know for a fact that Meta is going after users with the legal hammer.

I'm sure if we'd pool resources together we could build a truly open alternative worthy of building on top of.

Re: The Coming of Local LLMs

#94

Post by gitfan86 »

whimsicalism wrote: post #69

It appears there is this genre of articles pretending that LLAMA or its RL-HF tuned variants are somehow even close to an alternative to ChatGPT. Spending more than a few moments interacting even with the larger instruct-tuned variants of these models quickly dispels that idea. Why do these takes around open-source AI remain so popular? What is the driving force? I've posted this before, but it seems like this genre…

Because OpenAI or Google being the gate keepers is a show stopper for most serious people in the space.

They will be able to shutdown your startup on a whim. Even if they didn't politicians and regulators would be a huge risk. Without democratization we get blade runner . Not that democratization has no problems, just that it is the way a lot of us are wanting it to go.

Re: The Coming of Local LLMs

#95

Post by avereveard »

seydor wrote: post #89
whimsicalism wrote: post #69

It appears there is this genre of articles pretending that LLAMA or its RL-HF tuned variants are somehow even close to an alternative to ChatGPT. Spending more than a few moments interacting even with the larger instruct-tuned variants of these models quickly dispels that idea. Why do these takes around open-source AI remain so popular? What is the driving force? I've posted this before, but it seems like this genre…

You make it sound as if ChatGPT itself isnt severly limited. Vicuna and newer 13B models are quite close . And the uncensored models have a capability that ChatGPT will never have.

What are the newer models? I am testing them in batteries across complex tasks, so far vicuña is the most flexible but they all choke on reflective instructions (I. E. Knowledge to extract is not in the model but in the user text)

Re: The Coming of Local LLMs

#96

Post by whimsicalism »

gitfan86 wrote: post #94
whimsicalism wrote: post #69

It appears there is this genre of articles pretending that LLAMA or its RL-HF tuned variants are somehow even close to an alternative to ChatGPT. Spending more than a few moments interacting even with the larger instruct-tuned variants of these models quickly dispels that idea. Why do these takes around open-source AI remain so popular? What is the driving force? I've posted this before, but it seems like this genre…

Because OpenAI or Google being the gate keepers is a show stopper for most serious people in the space. They will be able to shutdown your startup on a whim. Even if they didn't politicians and regulators would be a huge risk. Without democratization we get blade runner . Not that democratization has no problems, just that it is the way a lot of us are wanting it to go.

And basing your startup on a model pirated from Meta isn't risky?

I get it, I understand why people like decentralization - but the open source community doesn't even have close to the capability to train an actually open-source LLaMA equivalent.

Re: The Coming of Local LLMs

#97

Post by turmeric_root »

wing-_-nuts wrote: post #54
kbrkbr wrote: post #7

Earlier quoted context omitted.

From my experience with llama.cpp and oobaboogas webui I can say they are amazing, at least on my gaming pc. I’m absolutely astonished at the speed and quality of llama, alpaca, galactica and vicuna (the >10B parameters ones). Make no mistake, it’s for tinkerers that do not expect each prompt to be answered human like. I see them as creativity and thought testing tools, also knowledge exploratory.

is there a difference in the quality of llm one would be able to train or run on a gpu with 8, 12, 16, all the way up to 24gb? I'm trying to decide whether it's worth while to splurge on a more expensive 4090 vs a 4070 or whatever.

More VRAM => larger models. IME it is absolutely worth maxing out VRAM for the significant improvement in quality, especially with LLaMA (though even with a 4090, you won't be able to run the largest 65-billion parameter model even with 4-bit quantization).

That said, I recommend renting a cloud GPU for a few hours and trying the larger models on them before buying a GPU of your own, just to see if the models meet your requirements.

Re: The Coming of Local LLMs

#98

Post by whimsicalism »

dpflan wrote: post #16
jjoonathan wrote: post #9

Earlier quoted context omitted.

Low precision: agreed Attention: Isn't it quadratic in context length? I dunno, this feels like the crude first iteration of something that will get inevitably passed by something that scales better.

Sharing a comment from a similar line of discussion: """ Complexity is quadratic in sequence length. For 512 tokens it is 262K, but for 4000 tokens it becomes 16M and goes OOM on a single GPU. We need about 100K-1M tokens to load whole books at once. Since 2017 there have been hundreds of attempts to bring O(N^2) to O(N), but none of them replaced the vanilla attention yet in large models. They lose on accuracy. Mayb…

There is literally no way that GPT-4 is using classic O(N^2) attention exclusively. They haven't released the results, but I promise you this is not what they are using exclusively.

Re: The Coming of Local LLMs

#99

Post by kbrkbr »

whimsicalism wrote: post #78
kbrkbr wrote: post #7

Earlier quoted context omitted.

From my experience with llama.cpp and oobaboogas webui I can say they are amazing, at least on my gaming pc. I’m absolutely astonished at the speed and quality of llama, alpaca, galactica and vicuna (the >10B parameters ones). Make no mistake, it’s for tinkerers that do not expect each prompt to be answered human like. I see them as creativity and thought testing tools, also knowledge exploratory.

If they are amazing, then ChatGPT must be God-like in your view. I've been underwhelmed by Alpaca and Alpaca-LORA and LLaMA all at 13B but I have not tried higher params.

In my opinion the problem with these is engineering a good prompt. I read of lots of people only getting nonsense or repetitions, and learned a bit from what they shared. These models are not chat bots.

Vicuna is more friendly in that regard.

But I’m well aware of their limitations also, and I can see how one can be underwhelmed. They are not jacks of all trades

Re: The Coming of Local LLMs

#100

Post by mcculley »

cs702 wrote: post #87
bick_nyers wrote: post #68

Earlier quoted context omitted.

On the other hand, securing the biological coprocessor in your homelab and running queries on it 24/7 appears to have legal implications.

Moral implications too. The first Matrix movie is about the moral implications.

In polite company it is considered rude to suggest that there was more than one Matrix movie.
Post reply on HN