Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook

github.com

Post reply on HN
Print view View original

141–150 of 303 posts

Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook

#141

Post by comex »

benob wrote: post #114
thefourthchime wrote: post #100

Earlier quoted context omitted.

I understand that AI/ML jargon can be confusing. Here's a brief explanation of some common terms and concepts to help you better understand them: Weights: In machine learning, weights are parameters used in models to determine the strength of the relationship between input features and the output prediction. They are learned during the training process and are adjusted to minimize the error in the model's predictions…

Picked the wrong one. LoRA, Low-rank Adaptation of LLMs ( https://arxiv.org/pdf/2106.09685.pdf ), consists in adapting the weights of a big neural network to a target task (here, answering to instructions). It doesn't touch the weights of the original model, but rather adds the product of two low-rank matrices to select layers. The weights from those matrices are learnable. The method allows to adapt big models on (r…

It didn’t just pick the wrong one; “Learning from Rules and Arguments” is completely made up.

Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook

#142

Post by [deleted] »

Faint wrote: post #119
thefourthchime wrote: post #100

Earlier quoted context omitted.

I understand that AI/ML jargon can be confusing. Here's a brief explanation of some common terms and concepts to help you better understand them: Weights: In machine learning, weights are parameters used in models to determine the strength of the relationship between input features and the output prediction. They are learned during the training process and are adjusted to minimize the error in the model's predictions…

Good job hallucinating with LoRA. I wonder if the paper was published after knowledge cut-off date.. (it stands for Low-Rank Adaptation).

[deleted]

Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook

#143

Post by dymk »

JasonZ2 wrote: post #79
tough wrote: post #66

Earlier quoted context omitted.

weights refer to the trained model weights like for example stable diffusion's v1.1, v1.2 .. v1.4 v.2.x etc. Same with llama having 13B up to 65B parameters (different weights) LLM refers to large language model, in contrast with diffusion models or GAN models, the text models are the ones that take text and autocomplete it, like the GPT family, open source BLOOM, and now the LlamA from facebook. LoRA is the latest m…

> weights refer to the trained model weights This is what I'm having a hard time understanding. So there's the weights, and also a model somewhere? That the weights are based on? Or that you combine with the model to tune it?

Model is code, weights are the input data to that code

Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook

#144

Post by Y_Y »

radarsat1 wrote: post #136
loudmax wrote: post #108

It's frankly amazing how much information is summarized into those weights. You do need to take whatever it says with a very healthy pinch of salt. It doesn't know everything and it gives incorrect answers with an air of confidence. Here's an example: > who preceded emperor Commodus? The first emperor of Rome was Nerva (98-100 AD). He succeeded Domitian and came to power after a brief civil war in 69AD, which saw the…

I think a really interesting area of further research into LLMs is going to be how to further decouple "intelligence" from "knowledge". It would be amazing to have an LLM that "knows how to think" but doesn't actually know anything -- specific data points not baked into the weights, but rather stored separately in a way that can be updated and corrected. Perhaps a step forward is the "toolformer" router, making the L…

If you want something that "knows how to think" then I don't think you'll find it in an LLM. You can't divorce knowledge from this kind of model, it guesses what word comes next based on its training on a corpus of knowledge.

Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook

#146

Post by tarruda »

thot_experiment wrote: post #96
adeon wrote: post #78

Earlier quoted context omitted.

I think the LLaMA-7B in general might not just be very good. I've been playing around and run full non-quantized LLaMA-30B and LLaMA-7B in a bunch of experiments and I think the quality of output is much, much better in LLaMA-30B.

Have you done any comparison testing between 30B q4/q8/q16? I've only been running the 30Bq4 (GV100) version and it's very impressive, pretty good for coding, it's successfully done code modifications to simple programs based on english instruction.

Can you show some example prompts you use for coding? Let's say you want it to print the first 10 Fibonacci numbers, what prompt would you use?

Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook

#147

Post by muyyatin2 »

londons_explore wrote: post #125
pr337h4m wrote: post #117

Earlier quoted context omitted.

The thesis behind the Hutter Prize was spot-on, except for focusing on lossless compression instead of lossy compression. http://prize.hutter1.net/

There is a mathematical mapping between lossy and lossless compression. They are effectively the same problem (theoretically, even though implentations are typically very different) LLM's would make excellent (but very slow) lossy or lossless compressors.

Do you have some links/references for someone wanting to learn more about this?

Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook

#149

Post by cuttothechase »

huseyinkeles wrote: post #55
gigel82 wrote: post #2

Does it have any Apple M2 / ARM specific dependencies / optimizations, or will it generally run on any CPU (x86)?

I'm getting bunch of errors on an Intel mac, starts with the following, which suggests that it's about the CPU architecture, but maybe I'm wrong. sysctl: unknown oid 'hw.optional.arm64'

I am getting the same error on an Intel Mac.

Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook

#150

Post by georgelyon »

Y_Y wrote: post #144
radarsat1 wrote: post #136

Earlier quoted context omitted.

I think a really interesting area of further research into LLMs is going to be how to further decouple "intelligence" from "knowledge". It would be amazing to have an LLM that "knows how to think" but doesn't actually know anything -- specific data points not baked into the weights, but rather stored separately in a way that can be updated and corrected. Perhaps a step forward is the "toolformer" router, making the L…

If you want something that "knows how to think" then I don't think you'll find it in an LLM. You can't divorce knowledge from this kind of model, it guesses what word comes next based on its training on a corpus of knowledge.

No, but you could build something that guesses the next step in a list of steps to access knowledge from an external source.
Post reply on HN