Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook

github.com

Post reply on HN
Print view View original

21–30 of 303 posts

Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook

#21

Post by GaggiX »

doctoboggan wrote: post #16

I am hoping the actual Alpaca weights leak at some point. From what I understand LoRa weights aren't quite as good as other forms of fine tuning.

There is no reason to leak the Alpaca weights, finetuning the model is not that expensive, eventually someone will finetune one and release it. (Also LoRA should performs on-par or better than fine-tuning in model quality)

Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook

#22

Post by phren0logy »

zitterbewegung wrote: post #13
phren0logy wrote: post #7

@antimatter15 - how much ram does your M2 MacBook Air have? Performance looks very good. Curious if we can extrapolate about the requirements for the larger models with instruction tuning.

Using the scripts from llama.cpp I was able to run the 65B model on my M1 Max which used ~40 GB of RAM. With 32GB you should be able to run 33B and lower. See https://gist.github.com/zitterbewegung/4787e42617aa0be6019c3...

Thanks!

Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook

#23

Post by antimatter15 »

bogwog wrote: post #12

Does this really need to be a fork of llama.cpp? It seems like there's no real changes to the code.

It reworks the interactive mode into a chat interface, which -if my code was cleaner- could have just been a pull request.

Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook

#24

Post by AndroidKitKat »

zitterbewegung wrote: post #13
phren0logy wrote: post #7

@antimatter15 - how much ram does your M2 MacBook Air have? Performance looks very good. Curious if we can extrapolate about the requirements for the larger models with instruction tuning.

Using the scripts from llama.cpp I was able to run the 65B model on my M1 Max which used ~40 GB of RAM. With 32GB you should be able to run 33B and lower. See https://gist.github.com/zitterbewegung/4787e42617aa0be6019c3...

No chance I'd be able to run this on an 8GB M1 Mac mini then? Looking to hook this up to an IRC bot for a bit of fun with my friends and have this spare Mac lying around.

Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook

#25

Post by ddren »

n8cpdx wrote: post #19

Does anyone know how much context this supports? With chatGPT I can give commands like “summarize the following document: ” and it works. I’d love to be able to do this locally with my obsidian notes.

The llama models were trained with a context size of 2048. By default llama.cpp limits it to 512, but you can use -c 2048 -n 2048 to get the full context window.

Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook

#26

Post by Tiberium »

loxias wrote: post #18
doctoboggan wrote: post #16

I am hoping the actual Alpaca weights leak at some point. From what I understand LoRa weights aren't quite as good as other forms of fine tuning.

Look at the changeset :) It contains a link for "ggml-alpaca-7b-14.bin". Because there's no substantive change to the code, I assume this fork exists (and this HN post exists) purely as a method to distribute the weights.

You're misinterpreting the comment a bit, the poster above is asking for the original Alpaca weights. The code in the repo uses a reproduction of Alpaca with different training code and method of fine-tune (LoRA).

Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook

#27

Post by ggerganov »

antimatter15 wrote: post #23
bogwog wrote: post #12

Does this really need to be a fork of llama.cpp? It seems like there's no real changes to the code.

It reworks the interactive mode into a chat interface, which -if my code was cleaner- could have just been a pull request.

"Clean Code, Horrible Performance" :)

Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook

#28

Post by madmads »

AndroidKitKat wrote: post #24
zitterbewegung wrote: post #13

Earlier quoted context omitted.

Using the scripts from llama.cpp I was able to run the 65B model on my M1 Max which used ~40 GB of RAM. With 32GB you should be able to run 33B and lower. See https://gist.github.com/zitterbewegung/4787e42617aa0be6019c3...

No chance I'd be able to run this on an 8GB M1 Mac mini then? Looking to hook this up to an IRC bot for a bit of fun with my friends and have this spare Mac lying around.

The 7b model loads on a lot less than 8GB RAM, so you should be good to use it on an 8GB M1 Mac Mini. It takes very little time to set up, so it's worth trying to find out.

Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook

#30

Post by doctoboggan »

antimatter15 wrote: post #23
bogwog wrote: post #12

Does this really need to be a fork of llama.cpp? It seems like there's no real changes to the code.

It reworks the interactive mode into a chat interface, which -if my code was cleaner- could have just been a pull request.

There is an interactive mode in llama.cpp

https://github.com/ggerganov/llama.cpp/blob/master/main.cpp#...

EDIT: I see now you are saying you re-worked the existing interactive mode. I still think your changes could be a PR into the original repo

Post reply on HN