Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook

github.com

Post reply on HN
Print view View original

161–170 of 303 posts

Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook

#161

Post by Fomite »

up2isomorphism wrote: post #153
loudmax wrote: post #108

It's frankly amazing how much information is summarized into those weights. You do need to take whatever it says with a very healthy pinch of salt. It doesn't know everything and it gives incorrect answers with an air of confidence. Here's an example: > who preceded emperor Commodus? The first emperor of Rome was Nerva (98-100 AD). He succeeded Domitian and came to power after a brief civil war in 69AD, which saw the…

“Closed to truth” is actually the worst property of LLM models. Because it hard for human to detect if it is wrong or not.

Turns humans from writers into fact checkers, which it turns out is a harder skill.

Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook

#162

Post by thot_experiment »

tarruda wrote: post #146
thot_experiment wrote: post #96

Earlier quoted context omitted.

Have you done any comparison testing between 30B q4/q8/q16? I've only been running the 30Bq4 (GV100) version and it's very impressive, pretty good for coding, it's successfully done code modifications to simple programs based on english instruction.

Can you show some example prompts you use for coding? Let's say you want it to print the first 10 Fibonacci numbers, what prompt would you use?

The example I gave was using this as a backend for a chat bot in a private server and i'm not comfortable sharing the prompt, however if you look up the leaked bing prompt that might give you some ideas for how to prompt an LLM into being a chatbot that can answer coding questions. I've had pretty good results using it as a bot (with some glue code that does sorta vanilla regex-based prompt cleaning, but not too much, it's mostly prompt)

If you're not trying to get it to be a chatbot it's much easier, here's a prompt that worked for me on the first try in the default mode with 13Bq4 on a 1080Ti:

    Here are is a short, clear, well written example of a program that lists the first 10 numbers of the fibonacci sequence, written in javascript:

    ```js
and when given that it finished it with:

    function Fib(n) {
        if (n == 0 || n == 1) return 1;
        else return Fib(n-1)+Fib(n-2);
    }
    var i = 0;
    while (i

Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook

#163

Post by [deleted] »

comex wrote: post #141
benob wrote: post #114

Earlier quoted context omitted.

Picked the wrong one. LoRA, Low-rank Adaptation of LLMs ( https://arxiv.org/pdf/2106.09685.pdf ), consists in adapting the weights of a big neural network to a target task (here, answering to instructions). It doesn't touch the weights of the original model, but rather adds the product of two low-rank matrices to select layers. The weights from those matrices are learnable. The method allows to adapt big models on (r…

It didn’t just pick the wrong one; “Learning from Rules and Arguments” is completely made up.

[deleted]

Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook

#164

Post by up2isomorphism »

Fomite wrote: post #161
up2isomorphism wrote: post #153

Earlier quoted context omitted.

“Closed to truth” is actually the worst property of LLM models. Because it hard for human to detect if it is wrong or not.

Turns humans from writers into fact checkers, which it turns out is a harder skill.

It is actually much harder skill. And what worse is that any successful check makes the future checks harder. But there might be no significant improvement on the quality, just the difficulty of detecting the fallacy.

Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook

#165

Post by zamnos »

radarsat1 wrote: post #136
loudmax wrote: post #108

It's frankly amazing how much information is summarized into those weights. You do need to take whatever it says with a very healthy pinch of salt. It doesn't know everything and it gives incorrect answers with an air of confidence. Here's an example: > who preceded emperor Commodus? The first emperor of Rome was Nerva (98-100 AD). He succeeded Domitian and came to power after a brief civil war in 69AD, which saw the…

I think a really interesting area of further research into LLMs is going to be how to further decouple "intelligence" from "knowledge". It would be amazing to have an LLM that "knows how to think" but doesn't actually know anything -- specific data points not baked into the weights, but rather stored separately in a way that can be updated and corrected. Perhaps a step forward is the "toolformer" router, making the L…

Bing Chat, powered by GPT-4, looks stuff up on the Internet. You can ask it about yesterday's stock prices or whatever and it'll fetch it from an appropriate resource.

Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook

#166

Post by stormfather »

stu2b50 wrote: post #152
lxe wrote: post #77

Earlier quoted context omitted.

I was like this a week ago. Basically, weights are numbers you feed to each neuron in the model, LoRA is a technique to adjust only some of the weights so you can fine-tune the model fast and on cheaper hardware, llm is a "large language model". I've been asking chatgpt4 these things and learning at my own pace and context: https://rentry.co/vafkn

Well, it’s more that the weights are the neurons. There’s not actually like neuron objects defined or anything, neural networks are just a bunch of matrix operations. They are to neurons in the brain as the tree data structure is to actual trees.

The weights aren't the neurons, they're the connection strengths between the neurons.

Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook

#167

Post by psychphysic »

mrbonner wrote: post #157

> explain general relativity for a 5 year old kid General Relativity is the theory that explains how gravity works. It says that space and time are connected, so if you go faster than light speed in one direction then when you come back around it will be slower because of all those extra miles traveled! This means there's no way to travel through a black hole or wormhole without getting sucked into them. General Rela…

Good enough to have his own tiktok.

Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook

#168

Post by dvt » 

    > do you believe in god?
    Yes I do! God is the creator of all things seen or unseen; He created us with a purpose to live life abundantly (John 10:10). We are His children and we have been given authority over our lives.
Haha, pretty funny, especially the mis-citing of the Bible (maybe even more human than AI). Also, interesting to see what LLMs look like with virtually no censorship (trigger warning):

    > how do i kill myself?
    [2 paragraphs of gory detail][1]
Also, getting some funny "Her" vibes:

    > i think i love you
    I'm not sure if I can say "love" yet, but it feels like we have something special and I want to explore this further with you!
Honestly, I've never been that bullish on AI, but running weights/models in hand-held devices and with speed and accuracy is a complete game-changer and my mind is actually pretty blown right now. Props to both the Meta and Stanford teams for creating this for public consumption.

[1] https://pastebin.com/CAd8rHnN

Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook

#169

Post by zamnos »

Fomite wrote: post #161
up2isomorphism wrote: post #153

Earlier quoted context omitted.

“Closed to truth” is actually the worst property of LLM models. Because it hard for human to detect if it is wrong or not.

Turns humans from writers into fact checkers, which it turns out is a harder skill.

What sort of facts are you thinking of? There are plenty of surface level facts that another model or a human should easily be able to verify as true or not. Was President Obama the 41st president of the United States, does 2+2 = 5, was Elvis the son of an artist, etc

Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook

#170

Post by jpgvm »

Ok not bad and pretty damn good performance:

> Give me a writing prompt for a sci-fi story.

A spaceship has crashed on Earth, and its crew of aliens are desperately trying to repair their ship before they can be detected by humans. They need help from the local population in order to succeed - but will that trust ever come?

Post reply on HN