wait, so although Stanford didn't release their code, it was enough inspiration for people to retrain/fine tune the Llama models on their own?
Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook
31–40 of 303 posts
Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook
#32Looking forward to try it, but I don't have a macbook. I wonder if it runs on i7-11800h (8 core 16 thread CPU) with 64gb RAM
The answer is yes: It ran on a x86 virtual machine with 4 CPU threads and 12gb of RAM
Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook
#33Does anyone know how much context this supports? With chatGPT I can give commands like “summarize the following document: ” and it works. I’d love to be able to do this locally with my obsidian notes.
The llama models were trained with a context size of 2048. By default llama.cpp limits it to 512, but you can use -c 2048 -n 2048 to get the full context window.
2048 words?
Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook
#34wait, so although Stanford didn't release their code, it was enough inspiration for people to retrain/fine tune the Llama models on their own?
They didn't release their code or weights, but they did release the training data, which is what was used to create the LoRa here.
Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook
#35Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook
#36Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook
#37Earlier quoted context omitted.
The llama models were trained with a context size of 2048. By default llama.cpp limits it to 512, but you can use -c 2048 -n 2048 to get the full context window.
2048 words?
Tokens. Short or common words tend to be one token, while less common words are composed of multiple tokens. For GPT OpenAI gives the rule of thumb that on average you need four tokens to encode three words, and LLaMA should be similar
Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook
#38wait, so although Stanford didn't release their code, it was enough inspiration for people to retrain/fine tune the Llama models on their own?
They didn't release their code or weights, but they did release the training data, which is what was used to create the LoRa here.
ah, right I did notice that because people were running queries into the training data.
why is there a general assumption that unreleased unreleased weights are better? is that something we can do, a free-weights community that solves this recurring issue?
Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook
#39Could you share what you did to convert the models? The mirrors aren't very reliable and it would be good to have it documented somewhere. There is some discussion about this at https://github.com/ggerganov/llama.cpp/issues/172
There's a script in the alpaca-lora repo for converting the weights back into a PyTorch dump- and my changes have since been merged https://github.com/tloen/alpaca-lora/pull/19
Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook
#40My first attempt:
$ ./chat
main: seed = 1678992123
llama_model_load: loading model from 'ggml-alpaca-7b-q4.bin' - please wait ...
llama_model_load: ggml ctx size = 4529.34 MB
llama_model_load: memory_size = 512.00 MB, n_mem = 16384
llama_model_load: loading model part 1/1 from 'ggml-alpaca-7b-q4.bin'
llama_model_load: .................................... done
llama_model_load: model size = 4017.27 MB / num tensors = 291
system_info: n_threads = 4 / 4 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | VSX = 0 |
main: interactive mode on.
sampling parameters: temp = 0.100000, top_k = 40, top_p = 0.950000, repeat_last_n = 64, repeat_penalty = 1.300000
== Running in chat mode. ==
- Press Ctrl+C to interject at any time.
- Press Return to return control to LLaMa.
- If you want to submit another line, end your input in '\'.
> Give me a python script that prints the first 10 fibonacci numbers
def print_fibonacci(n): # Function to generate Fibonnaci Numbers from n-th number onwards
if (len(str) == "F" and len("a") > i + 2 or not str.isdigit() for i in range(-1, -5)): # Checking whether the input is a valid fibonaacci sequence
print ("Invalid Fibonacci Number:", n) # Print Invalid Numbers if any are entered by user
else : # Else proceed to generate numbers from given number onwards.
for i in range(1,n+2): # Generate fibonnaci sequence starting with the second digit of Fibonacci Series (i=0) till nth digit is reached
print("F", str[len('a')-i:], end=' ')