Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook • Hacker News Board

Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook

#11

Post by dmw_ng » Thu, Mar 16, 2023, 6:10 PM UTC

siraben wrote: post #8

Are there torrents for 13B and up as well?

The LoRa weights for 13B are on huggingface ( https://huggingface.co/samwit/alpaca13B-lora ), it should be possible to follow the instructions linked in the Alpaca.cpp readme to merge those weights into the base 13B model, then just just follow the usual Llama.cpp conversion and quantization steps

Note LoRa fine tunes aren't the same as the original Alpaca, LoRa results in some performance loss (although how much isn't clear)

Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook

#12

Post by bogwog » Thu, Mar 16, 2023, 6:12 PM UTC

Does this really need to be a fork of llama.cpp? It seems like there's no real changes to the code.

Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook

#13

Post by zitterbewegung » Thu, Mar 16, 2023, 6:14 PM UTC

phren0logy wrote: post #7

@antimatter15 - how much ram does your M2 MacBook Air have? Performance looks very good. Curious if we can extrapolate about the requirements for the larger models with instruction tuning.

Using the scripts from llama.cpp I was able to run the 65B model on my M1 Max which used ~40 GB of RAM. With 32GB you should be able to run 33B and lower.

See https://gist.github.com/zitterbewegung/4787e42617aa0be6019c3...

Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook

#14

Post by gpderetta » Thu, Mar 16, 2023, 6:15 PM UTC

antimatter15 wrote: post #5

gigel82 wrote: post #2

Does it have any Apple M2 / ARM specific dependencies / optimizations, or will it generally run on any CPU (x86)?

It inherits from ARM NEON optimizations in llama.cpp but I believe it should also work on x86.

it built in a couple of seconds with no issues on my x86 machine. Downloading weights now.

Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook

#15

Post by zitterbewegung » Thu, Mar 16, 2023, 6:15 PM UTC

tarruda wrote: post #6

Looking forward to try it, but I don't have a macbook. I wonder if it runs on i7-11800h (8 core 16 thread CPU) with 64gb RAM

Alpaca.cpp is a derivative of llama.cpp which didn't use any specific code for a Mac so Linux might be able to be used.

Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook

#16

Post by doctoboggan » Thu, Mar 16, 2023, 6:15 PM UTC

I am hoping the actual Alpaca weights leak at some point. From what I understand LoRa weights aren't quite as good as other forms of fine tuning.

Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook

#17

Post by mil22 » Thu, Mar 16, 2023, 6:19 PM UTC

gigel82 wrote: post #2

Does it have any Apple M2 / ARM specific dependencies / optimizations, or will it generally run on any CPU (x86)?

Runs fine on WSL2:

main: seed = 1678990693 llama_model_load: loading model from 'ggml-alpaca-7b-q4.bin' - please wait ... llama_model_load: ggml ctx size = 4529.34 MB llama_model_load: memory_size = 512.00 MB, n_mem = 16384 llama_model_load: loading model part 1/1 from 'ggml-alpaca-7b-q4.bin' llama_model_load: .................................... done llama_model_load: model size = 4017.27 MB / num tensors = 291

system_info: n_threads = 4 / 32 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | VSX = 0 | main: interactive mode on. sampling parameters: temp = 0.100000, top_k = 40, top_p = 0.950000, repeat_last_n = 64, repeat_penalty = 1.300000

== Running in chat mode. == - Press Ctrl+C to interject at any time. - Press Return to return control to LLaMa. - If you want to submit another line, end your input in '\'.

> What is an alpaca? An Alpaca (Vicugna pacos) is one of two species in the genus Vicuña, which are both domesticated animals from South America and Central Asia that have been bred for their fine wool. >

Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook

#18

Post by loxias » Thu, Mar 16, 2023, 6:19 PM UTC

doctoboggan wrote: post #16

I am hoping the actual Alpaca weights leak at some point. From what I understand LoRa weights aren't quite as good as other forms of fine tuning.

Look at the changeset :) It contains a link for "ggml-alpaca-7b-14.bin".

Because there's no substantive change to the code, I assume this fork exists (and this HN post exists) purely as a method to distribute the weights.

Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook

#19

Post by n8cpdx » Thu, Mar 16, 2023, 6:20 PM UTC

Does anyone know how much context this supports?

With chatGPT I can give commands like “summarize the following document: ” and it works. I’d love to be able to do this locally with my obsidian notes.

Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook

#20

Post by antimatter15 » Thu, Mar 16, 2023, 6:21 PM UTC

doctoboggan wrote: post #16

I am hoping the actual Alpaca weights leak at some point. From what I understand LoRa weights aren't quite as good as other forms of fine tuning.

From my experimentation it seems like there's some significant loss in accuracy running the tuned LoRa models through llama.cpp (due to bugs/differences in inference or tokenization), even aside from losses due to quantization.