Is it possible to download alpaca models with more than 7b parameters?
Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook
221–230 of 303 posts
Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook
#222Earlier quoted context omitted.
I think a really interesting area of further research into LLMs is going to be how to further decouple "intelligence" from "knowledge". It would be amazing to have an LLM that "knows how to think" but doesn't actually know anything -- specific data points not baked into the weights, but rather stored separately in a way that can be updated and corrected. Perhaps a step forward is the "toolformer" router, making the L…
If you want something that "knows how to think" then I don't think you'll find it in an LLM. You can't divorce knowledge from this kind of model, it guesses what word comes next based on its training on a corpus of knowledge.
One could imagine training a much smaller LLM on synthetic data that didn't contain much or any real knowledge, but just examples of in-context reasoning.
E.g. the training data might look like "a fooble is to a tooble as a fabble is to a" with the answer "tabble".
So you feed it tons of these kind of nonsense training data that forces it to only learn the in-context reasoning part of language, and none of the world knowledge.
That said, it is unclear to me how much value such a model would be.
You could imagine a middle ground where it does have basic knowledge, such as dictionary definitions and things derivable from those.
E.g. "A tree has leaves. Leaves are green. Therefore a tree has a part that is green." type stuff.
So you could give it some amount of world-grounding and common sense knowledge, but nothing involving history, proper nouns, etc.
I imagine you could make such a model much smaller than these giant LLMs.
Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook
#223Earlier quoted context omitted.
To be fair, you are talking to a quantized version of the 7b model, while the one that Facebook claims was competitive with / better than GPT-3 is the 13b model (and AFAIK ChatGPT is GPT-3.5, so maybe you should want to focus on 30b+).
I'm trying the 13B one, it's decent, I'd say on par with GPT 3, definitely not 3.5 or 4 yet however. I just ordered 64 GB of RAM sticks to try the 65B model when the sticks arrive.
How well does it work on DRAM? I thought everyone running heavier models are doing it on GPUs.
Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook
#224Earlier quoted context omitted.
Wait, didn't it get the Bible quote correct? "The thief does not come except to steal, and to kill, and to destroy. I have come that they may have life, and that they may have it more abundantly."
Oops, actually you're totally right, go figure! I thought the second sentence was actually verse 11. AI: 1, me: 0.
Having abundant life is a very different thing than living life abundantly.
If it’s not a misquote then it’s a misinterpretation.
Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook
#225I love the Disclaimer at the bottom. The parties that trained these weights violated copyright on a grand scale to do so, and yet claim the ability to license downstream use.
> The parties that trained these weights violated copyright on a grand scale
How exactly did they violate copyright?
Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook
#226Earlier quoted context omitted.
I'm trying the 13B one, it's decent, I'd say on par with GPT 3, definitely not 3.5 or 4 yet however. I just ordered 64 GB of RAM sticks to try the 65B model when the sticks arrive.
How well does it work on DRAM? I thought everyone running heavier models are doing it on GPUs.
Alpaca works entirely off CPU so the models should work fine. I'll try tomorrow and report back.
Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook
#227Earlier quoted context omitted.
If you want something that "knows how to think" then I don't think you'll find it in an LLM. You can't divorce knowledge from this kind of model, it guesses what word comes next based on its training on a corpus of knowledge.
One could imagine training a much smaller LLM on synthetic data that didn't contain much or any real knowledge, but just examples of in-context reasoning. E.g. the training data might look like "a fooble is to a tooble as a fabble is to a" with the answer "tabble". So you feed it tons of these kind of nonsense training data that forces it to only learn the in-context reasoning part of language, and none of the world…
The reason, i suspect, that this method won't work, is because the training dataset has knowledge embedded in it, regardless of whether you think it only contains in-context logic.
The evidence is that there are these "glitched" tokens : https://www.youtube.com/watch?v=WO2X3oZEJOA
if you watched to the end of that video, it explores a very plausible reason why such glitched tokens exist - it is because the training dataset has a bunch of "nonsense" (a large dump of the subreddit that just counts a number, as well as various other "data").
So my hypothesis is that if you attempted the in-context logic training dataset, it would produce large number of glitched tokens (different ones, but equally as "illogical" to the user).
Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook
#228Earlier quoted context omitted.
Oops, actually you're totally right, go figure! I thought the second sentence was actually verse 11. AI: 1, me: 0.
Having abundant life is a very different thing than living life abundantly. If it’s not a misquote then it’s a misinterpretation.
You have to be a little charitable towards an LLM ;). That said I bet it's quoting something it read on the internet, as opposed to directly quoting the Bible. There is even a Wikipedia page on term https://en.wikipedia.org/wiki/Abundant_life#:~:text=The%20te....
Also I'm not sure I understand how they are very different. Could you please help me understand and elaborate?
Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook
#229It's frankly amazing how much information is summarized into those weights. You do need to take whatever it says with a very healthy pinch of salt. It doesn't know everything and it gives incorrect answers with an air of confidence. Here's an example: > who preceded emperor Commodus? The first emperor of Rome was Nerva (98-100 AD). He succeeded Domitian and came to power after a brief civil war in 69AD, which saw the…
“Closed to truth” is actually the worst property of LLM models. Because it hard for human to detect if it is wrong or not.
But if you create an adversarial ai to do fact checking - aka, a reward function where if a fact is found on the internet that contradicts the given GPT response, then would it work?
Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook
#230My first attempt: $ ./chat main: seed = 1678992123 llama_model_load: loading model from 'ggml-alpaca-7b-q4.bin' - please wait ... llama_model_load: ggml ctx size = 4529.34 MB llama_model_load: memory_size = 512.00 MB, n_mem = 16384 llama_model_load: loading model part 1/1 from 'ggml-alpaca-7b-q4.bin' llama_model_load: .................................... done llama_model_load: model size = 4017.27 MB / num tensors =…
From my experimentation I suspect there's some subtle bug in llama.cpp that especially degrades code related prompts- even without quantizing
I noticed there's a couple of open issues on llama.cpp investigating quality issues. It's interesting if a wrong implementation still generates plausible output. It sounds like an objective quality metric would help track down issues.