This is crazy... GPT-4, ChatGPT, Cloude, PaLM and now Alpaca locally... What a world to live in now!
There is no way to keep even up with the trends. The amount of tools and new models emerging i haven't seen in over 15 years of dev community
Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook
81–90 of 303 posts
Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook
#82language models require massive scale to train. But scale isn't only in the number of parameters or neurons. Scale also exists in the amount of data the model trains on. While parameter size affects post training size and requirements to run. Data size does not. Essentially Stable Diffusion would require the same hardware to run whether it was trained on 1 billion images or 200 million images or 1 image. Most llm tra…
"Essentially Stable Diffusion would require the same hardware to run whether it was trained on 1 billion images or 200 million images or 1 image." Same hardware maybe but you need more compute as the image count goes up
The parent poster was talking about training longer but the model being kept at smaller scale so it would not be expensive to use in production. It's a trade-off, you could train shorter with a larger model.
Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook
#83Earlier quoted context omitted.
weights refer to the trained model weights like for example stable diffusion's v1.1, v1.2 .. v1.4 v.2.x etc. Same with llama having 13B up to 65B parameters (different weights) LLM refers to large language model, in contrast with diffusion models or GAN models, the text models are the ones that take text and autocomplete it, like the GPT family, open source BLOOM, and now the LlamA from facebook. LoRA is the latest m…
> weights refer to the trained model weights This is what I'm having a hard time understanding. So there's the weights, and also a model somewhere? That the weights are based on? Or that you combine with the model to tune it?
Let's take a step back. You have a model like linear regression. For example, y=bx where y are your outputs and x are your inputs. Based on some data, you learn that b=1. Therefore, you share the weights of the model as a file like {b=1} and also share with them the model y=bx (usually shared via code) so they can run it in production.
Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook
#84Does it have any Apple M2 / ARM specific dependencies / optimizations, or will it generally run on any CPU (x86)?
It does run on either with decent speed. Probably can even do it on a rasberry pi 4.
Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook
#85Earlier quoted context omitted.
weights refer to the trained model weights like for example stable diffusion's v1.1, v1.2 .. v1.4 v.2.x etc. Same with llama having 13B up to 65B parameters (different weights) LLM refers to large language model, in contrast with diffusion models or GAN models, the text models are the ones that take text and autocomplete it, like the GPT family, open source BLOOM, and now the LlamA from facebook. LoRA is the latest m…
> weights refer to the trained model weights This is what I'm having a hard time understanding. So there's the weights, and also a model somewhere? That the weights are based on? Or that you combine with the model to tune it?
model is class with params. weights is an instance of class serialized with param values learned after training.
Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook
#86why do these LLMs struggle so hard to be concise? is this because the Alpaca dataset is pretty wordy? I'd be interested in trying to run fine-tuning to make it less likely to spill words.
Conciseness is regulated from the 3rd stage of training, RLHF. Alpaca only had the first two stages.
Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook
#87Earlier quoted context omitted.
From my experimentation I suspect there's some subtle bug in llama.cpp that especially degrades code related prompts- even without quantizing
I think the LLaMA-7B in general might not just be very good. I've been playing around and run full non-quantized LLaMA-30B and LLaMA-7B in a bunch of experiments and I think the quality of output is much, much better in LLaMA-30B.
What kind of hardware is necessary to run non-quantized LLama-30B?
Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook
#88Earlier quoted context omitted.
weights refer to the trained model weights like for example stable diffusion's v1.1, v1.2 .. v1.4 v.2.x etc. Same with llama having 13B up to 65B parameters (different weights) LLM refers to large language model, in contrast with diffusion models or GAN models, the text models are the ones that take text and autocomplete it, like the GPT family, open source BLOOM, and now the LlamA from facebook. LoRA is the latest m…
> weights refer to the trained model weights This is what I'm having a hard time understanding. So there's the weights, and also a model somewhere? That the weights are based on? Or that you combine with the model to tune it?
My really simplified explanation is:
Your inputs are lists of numbers. Your outputs are lists of numbers. There exists some possible list of numbers such that, if you multiply your inputs by that list you'll get (approximately) the outputs.
In this conception that possible set of numbers are the weights. "Training" is when you run inputs, compare to known outputs, and then update the weights so they produce outputs closet to what you want.
Large Language Models, it may be hard to see how they fit this paradigm - basically convert a sequence to a list of numbers ('aardvark' is 1, 'apple' is 2 etc) and then the desired output is the next word in the sequence (represented as a number). Surprisingly, if you get good at predicting next word in sequence you also get the ChatGPT et al behavior.
Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook
#89=> write a haiku about leaves Autumn's golden hues - Leaves of red, yellow and blue! \ #haikupoetry #leavesshort[1] > segmentation fault ./chat Interesting.
Got one too while it was writing a bullet list. (It wrote a poem and a song before without problems)
Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook
#90=> write a haiku about leaves Autumn's golden hues - Leaves of red, yellow and blue! \ #haikupoetry #leavesshort[1] > segmentation fault ./chat Interesting.
I love the part about segmentation fault. It seems quite original actually!