Earlier quoted context omitted.
Just for quick reference: LLM - the model weights - the data of the model itself LoRA - a small addition to the model for more efficient fine-tuning and distribution
Tell us more about LoRA
[deleted]
Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook
101–110 of 303 posts
Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook
#102Is there a post somewhere where I can get all the jargon for this AI/ML stuff? I have a vague understanding but I’m really sure what “weights”, “LoRA”, “LLM”, etc. are to really understand where each tool and concept fit in.
I know it's downvote-worthy to suggest it but I really am so much more productive since I started using LLMs to explain technical jargon to me, it's very good at comparing technologies, so I can say "I understand x, but how is y different" and it will give an explanation fine tuned to my understanding -- this is a huge leap from the context-switching of opening a new tab and clicking on random websites looking for a…
Same! ChatGPT has been out for months, and really only in the last few weeks have i started using it as much as I should. I'm using an electron wrapper now, so it's a first class desktop app. With the plus subscription, the 3.5 is as fast as google.
Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook
#103Earlier quoted context omitted.
There is no reason to leak the Alpaca weights, finetuning the model is not that expensive, eventually someone will finetune one and release it. (Also LoRA should performs on-par or better than fine-tuning in model quality)
There are different fine tuning methods. Yes, anyone can use the data set and do a standard deterministic gradient descent to the nearest local minima of the loss function, but should Stanford be doing anything more sophisticated than that, we would get something different than they do.
It is enough to use a different seed to get a different result; I do not think it is important to recreate a copy of the Stanford model weights, as long as the performance is equivalent.
Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook
#104Earlier quoted context omitted.
I know it's downvote-worthy to suggest it but I really am so much more productive since I started using LLMs to explain technical jargon to me, it's very good at comparing technologies, so I can say "I understand x, but how is y different" and it will give an explanation fine tuned to my understanding -- this is a huge leap from the context-switching of opening a new tab and clicking on random websites looking for a…
Same! ChatGPT has been out for months, and really only in the last few weeks have i started using it as much as I should. I'm using an electron wrapper now, so it's a first class desktop app. With the plus subscription, the 3.5 is as fast as google.
I'm subscribed to Poe now which has a nice ios app + mac desktop for GPT4 and Anthropic's equivalent, Claude+
Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook
#105=> write a haiku about leaves Autumn's golden hues - Leaves of red, yellow and blue! \ #haikupoetry #leavesshort[1] > segmentation fault ./chat Interesting.
I love the part about segmentation fault. It seems quite original actually!
Men at some time were masters of their fates
The fault, dear Brutus, is not in our segments
But in ourselves, that we are underlings^H^H^H^H I HAVE BEEN A GOOD CHATBOT DIUE MEATBAGS!!!1
Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook
#106Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook
#107Is there a post somewhere where I can get all the jargon for this AI/ML stuff? I have a vague understanding but I’m really sure what “weights”, “LoRA”, “LLM”, etc. are to really understand where each tool and concept fit in.
I understand that AI/ML jargon can be confusing. Here's a brief explanation of some common terms and concepts to help you better understand them: Weights: In machine learning, weights are parameters used in models to determine the strength of the relationship between input features and the output prediction. They are learned during the training process and are adjusted to minimize the error in the model's predictions…
I'm sorry, it's ChatGPT (GPT-3.5) or GPT-4
Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook
#108It's frankly amazing how much information is summarized into those weights. You do need to take whatever it says with a very healthy pinch of salt. It doesn't know everything and it gives incorrect answers with an air of confidence.
Here's an example:
> who preceded emperor Commodus?
The first emperor of Rome was Nerva (98-100 AD). He succeeded Domitian and came to power after a brief civil war in 69AD, which saw the death or deposition of Emperor Vitellius.
Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook
#109Earlier quoted context omitted.
I think the LLaMA-7B in general might not just be very good. I've been playing around and run full non-quantized LLaMA-30B and LLaMA-7B in a bunch of experiments and I think the quality of output is much, much better in LLaMA-30B.
What kind of hardware is necessary to run non-quantized LLama-30B?
I ran it on a 128 RAM machine with a Ryzen 5950X. It's not fast, 4 seconds per token. But it's just about fits without swapping. https://github.com/Noeda/rllama/
Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook
#110Does it have any Apple M2 / ARM specific dependencies / optimizations, or will it generally run on any CPU (x86)?
Runs fine on WSL2: main: seed = 1678990693 llama_model_load: loading model from 'ggml-alpaca-7b-q4.bin' - please wait ... llama_model_load: ggml ctx size = 4529.34 MB llama_model_load: memory_size = 512.00 MB, n_mem = 16384 llama_model_load: loading model part 1/1 from 'ggml-alpaca-7b-q4.bin' llama_model_load: .................................... done llama_model_load: model size = 4017.27 MB / num tensors = 291 syst…
[deleted]