With 8-bit this should fit on a 3060 12GB, which is pretty affordable right now. If this works as well as they state it's going to be amazing.
Alpaca: A strong open-source instruction-following model
251–260 of 313 posts
Re: Alpaca: A strong open-source instruction-following model
#252Earlier quoted context omitted.
Good points, though I would gently encourage not starting a post with "WRONG." in the middle of a nuanced discussion. I remember 'way back when' when there was a .5-2% flat performance drop for UINT8 on some models when it was first introduced (depends upon the modality). Like, 4 bit quantization really is probably enough for a number of usecases and likely beats smaller models with precision enough to make it the eq…
Good points, I didn't mean to come off abrasive but I can see why I would. My attention was to get attention on a thread where my new comment would be buried under the 8 other replies, so I put a big attention grabber at the start. But again good points about the nuances of lower precision. For LLMs at least 'The Case for 4-bit Precision' and 'GPTQ' seem fairly conclusive that over ~10B parameters even 3-bit precisio…
[dead]
Re: Alpaca: A strong open-source instruction-following model
#253This is why I think we're seeing a Stable Diffusion moment for LLMs: https://simonwillison.net/2023/Mar/11/llama/ Look at the timeline: 24th February 2023: LLaMA is announced, starts being shared with academic partners: https://research.facebook.com/publications/llama-open-and-ef... 2nd March: Someone posts a PR with a BitTorrent link to the models: https://github.com/facebookresearch/llama/pull/73 10th March: First…
>Turned this into a blog post: https://simonwillison.net/2023/Mar/13/alpaca/
HN discussion: https://news.ycombinator.com/item?id=35141531
Re: Alpaca: A strong open-source instruction-following model
#254Earlier quoted context omitted.
Wow that was quick, they only started in what, December?
Yeah amazingly fast. I haven't looked at the quality but some comments in the discord seemed to indicate it was decent quality but had a lot of coding-type prompts.
I don’t see any problem with an abundance of coding prompts. The type of person who runs a local LLM is probably using it for coding anyway.
Re: Alpaca: A strong open-source instruction-following model
#255Earlier quoted context omitted.
Sounds like a great Kickstarter option for a truly free model. I'd back it.
Please do it, people shouldn't put up with the apathetic siloed status quo. I'm sure people will find all sorts of beneficial uses for these models they are going to run on their own hardware!
These models are as valuable as code now.
And we know how open source eventually took over pretty much all code. I expect this to be the same, but instead of four decades, it will be four months
Re: Alpaca: A strong open-source instruction-following model
#256Earlier quoted context omitted.
I am not a lawyer, but the consensus (around here anyway) seems to be that weights cannot be copyrighted since they are the output of an automated process, so perhaps the license cannot limit how the weights are used. The code can be copyrighted, but the code can also be reimplemented from scratch and used to load the weights.
The US doesn't have database copyright, but some countries, including the entire European Union, do.[1] [1] https://en.wikipedia.org/wiki/Database_right
Those protections rely on "systematic arrangement", i.e. that some effort has been put into finding, verifying and collection a set of facts / individual pieces of data. It is not obvious that would apply to the set of weights that come out of a training algorithm. (standard I'm not a lawyer, I don't know but I suspect it is unclear enough that it would need to be argued in court).
Re: Alpaca: A strong open-source instruction-following model
#257Earlier quoted context omitted.
Don’t even need human time. Just run it through davinci. Is this correct, yes or no? Or MTurk. Can cost less than $500.
Not sure about this specific example, but the prompts used for training were generated by davinci. The researchers only used 175 human-written prompts, and these were fed into davinci to generate the other ~52k used for fine tuning LLaMA to make it follow instructions. They had humans review the generated prompts to eliminate bad ones before fine tuning, but humans mess up. I wonder how davinci would respond if every…
Funny enough it would.
LLMs have proven to be quite good at evaluating their own output. Think of things like "Think step by step" and how that improves output.
Re: Alpaca: A strong open-source instruction-following model
#258Re: Alpaca: A strong open-source instruction-following model
#259Earlier quoted context omitted.
WRONG. Research shows effectively imperceptible performance difference at 4-bit and even 3-bit with GPTQ quantization. You cannot tell the difference and if you think you do you're wrong, because it barely even registers on any benchmark. (Note: llama.cpp's 4bit is naive, not GPTQ, and sucks but they are refactoring it to use GPTQ quantization) References: https://arxiv.org/abs/2210.17323 - GPTQ: Accurate Post-Traini…
Good points, though I would gently encourage not starting a post with "WRONG." in the middle of a nuanced discussion. I remember 'way back when' when there was a .5-2% flat performance drop for UINT8 on some models when it was first introduced (depends upon the modality). Like, 4 bit quantization really is probably enough for a number of usecases and likely beats smaller models with precision enough to make it the eq…
Tak that WRONG! as a reference to the Two Stupid Dogs, then it may be a lot easier to stomach. :)
Re: Alpaca: A strong open-source instruction-following model
#260Earlier quoted context omitted.
For 10 billion+ parameter models, the effects of quantization are relatively small, for smaller models like Llama 7B the effect becomes more dramatic, but there is ongoing research on new quantization methods (like GPTQ) that preserve significant performance even on the lower end. Quantization isn't the only technique available for downsizing a model, Llama itself is already the result of sizing the model and input d…
Curious about the google traffic comment. Are you saying people are visiting sites less because they can stay on Bing/OpenAI?
I don't know the data but as an anecdote for most searches that would have returned blogspam (i.e. "what's the best birthday gift for a groom") in relying more and more on chatgpt.
I used to use it even more, but some of the recent changes reduced its ability at complex, creative tasks.