Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook

github.com

Post reply on HN
Print view View original

51–60 of 303 posts

Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook

#51

Post by thewataccount »

diimdeep wrote: post #46

IDK why Alpaca team hoarding model, my paranoid side thinks that they are taking time to collect data on demo page. xD

It's because LLaMa is technically not publicly available, Facebook currently intends for only researchers to have it.

Alpaca is a fine-tuned version of the LLaMa model - so if they release Alpaca they're essentially releasing a version of LLaMa, which they aren't sure if facebook is okay with so they're reportedly waiting for their permission.

While yes you can easily find LLaMa it's technically not intended for the public according to Facebook.

Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook

#52

Post by unshavedyak »

tarruda wrote: post #6

Looking forward to try it, but I don't have a macbook. I wonder if it runs on i7-11800h (8 core 16 thread CPU) with 64gb RAM

I'm using this exact repo on Linux (NixOS), right now.

Wish i could run the 60B model, this 7B model is a bit weird.

Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook

#53

Post by GaggiX »

thewataccount wrote: post #9
Shank wrote: post #4

Earlier quoted context omitted.

No, this is an open reproduction of Alpaca on the leaked LLaMA 7B weight set.

This isn't actually a direct reproduction. It's a bit missleading because it only says it at the bottom. This actually uses LORAs which train way faster and are much more flexible, but they perform a little worse. There is someone that is doing a direct reproduction of Alpaca in the alpaca repo issues though. > The weights are based on the published fine-tunes from alpaca-lora

>they perform a little worse.

Be aware that LoRA performs on-par or better than fine-tuning in model quality if trained correctly as the paper shows: https://arxiv.org/abs/2106.09685

Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook

#55

Post by huseyinkeles »

gigel82 wrote: post #2

Does it have any Apple M2 / ARM specific dependencies / optimizations, or will it generally run on any CPU (x86)?

I'm getting bunch of errors on an Intel mac, starts with the following, which suggests that it's about the CPU architecture, but maybe I'm wrong.

sysctl: unknown oid 'hw.optional.arm64'

Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook

#57

Post by bogwog »

tarruda wrote: post #40

My first attempt: $ ./chat main: seed = 1678992123 llama_model_load: loading model from 'ggml-alpaca-7b-q4.bin' - please wait ... llama_model_load: ggml ctx size = 4529.34 MB llama_model_load: memory_size = 512.00 MB, n_mem = 16384 llama_model_load: loading model part 1/1 from 'ggml-alpaca-7b-q4.bin' llama_model_load: .................................... done llama_model_load: model size = 4017.27 MB / num tensors =…

It seems to be better at Tweeting than Coding.

prompt: "Recent Tweet by Joe Biden: "

output:

    Recent Tweet by Joe Biden: 3D printing is a great way to make money.
    Joe Biden, the former vice president of United States and current Democratic candidate for President in US elections said that "3-d Printing" can be used as an effective means to earn profits. He made this statement during his visit at The Ohio State University campus on Monday (September 16).
    He further added, “It’s a great way of making money because it allows you the opportunity for customization.” Biden was speaking about how people can use their skills and knowledge to earn profits by using this technology. He also ur
...but not by much

Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook

#58

Post by thewataccount »

GaggiX wrote: post #53
thewataccount wrote: post #9

Earlier quoted context omitted.

This isn't actually a direct reproduction. It's a bit missleading because it only says it at the bottom. This actually uses LORAs which train way faster and are much more flexible, but they perform a little worse. There is someone that is doing a direct reproduction of Alpaca in the alpaca repo issues though. > The weights are based on the published fine-tunes from alpaca-lora

>they perform a little worse. Be aware that LoRA performs on-par or better than fine-tuning in model quality if trained correctly as the paper shows: https://arxiv.org/abs/2106.09685

Ah thank you for the correction this is good to know! I'll admit I'm not super experienced with them directly.

I think the current LoRAs for Alpaca are reportedly performing a little bit worse - I'm guessing from a lack of optimization so I'm sure it'll improve.

Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook

#59

Post by doublepg23 »

Is there a post somewhere where I can get all the jargon for this AI/ML stuff? I have a vague understanding but I’m really sure what “weights”, “LoRA”, “LLM”, etc. are to really understand where each tool and concept fit in.

Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook

#60

Post by d0m »

doublepg23 wrote: post #59

Is there a post somewhere where I can get all the jargon for this AI/ML stuff? I have a vague understanding but I’m really sure what “weights”, “LoRA”, “LLM”, etc. are to really understand where each tool and concept fit in.

Run the Alpaca chat and ask it :)
Post reply on HN