11–20 of 436 posts
The pace of collaborative OSS development on these projects is amazing, but the rate of optimisations being achieved is almost unbelievable. What has everyone been doing wrong all these years cough sorry, I mean to say weeks? Ok I answered my own question.
I have predicted that LLaMA will be available on mobile phones before the end of this year. We are very close.
The pace of collaborative OSS development on these projects is amazing, but the rate of optimisations being achieved is almost unbelievable. What has everyone been doing wrong all these years cough sorry, I mean to say weeks? Ok I answered my own question.
I have predicted that LLaMA will be available on mobile phones before the end of this year. We are very close.
Has anyone done any comprehensive analysis on exactly how much quantization affects the quality of model output? I haven't seen any more than people running it and being impressed (or not) by a few sample outputs. I would be very curious about some contrastive benchmarks between a quantized and non-quantized version of the same model.
tl;dr quantizing the 13B model gives up about 30% of the improvement you get from moving from 7B to 13B - so quantized 13B is still much better than unquantized 7B. Similar results for the larger models.
Has anyone done any comprehensive analysis on exactly how much quantization affects the quality of model output? I haven't seen any more than people running it and being impressed (or not) by a few sample outputs. I would be very curious about some contrastive benchmarks between a quantized and non-quantized version of the same model.
Perplexity - model options
5.5985 - 13B, q4_0
5.9565 - 7B, f16
6.3001 - 7B, q4_1
6.5949 - 7B, q4_0
6.5995 - 7B, q4_0, --memory_f16
According to this repo[1] difference is about 3% in their implementation with right group size. If you'd like to know more, I think you should read GPTQ paper[2].
An anonymous HN user named L pledged $200k for llama-dl’s legal defense: https://twitter.com/theshawwn/status/1641804013791215619?s=6...
This may not seem like much vs Meta, but it’s enough to get the issue into the court system where it can be settled. The tweet chain has the details.
The takeaway for you is that you’ll soon be able to use LLaMA without worrying that Facebook will knock you offline for it. (I wouldn’t push your luck by trying to use it for commercial purposes though.)
Past discussion: https://news.ycombinator.com/item?id=35288415
I’d also like to take this opportunity to thank all of the researchers at MetaAI for their tremendous work. It’s because of them that we have access to such a wonderful model in the first place. They have no say over the legal side of things. One day we’ll all come together again, and this will just be a small speedbump in the rear view mirror.
EDIT: Please do me a favor and skip ahead to this comment: https://news.ycombinator.com/item?id=35393615
It's from jart, the author of the PR the submission points to. I really had no idea that this was a de facto Show HN, and it's terribly rude to post my comment in that context. I only meant to reassure everyone that they can freely hack on llama, not make a huge splash and detract from their moment on HN. (I feel awful about that; it's wonderful to be featured on HN, and no one should have to share their spotlight when it's a Show HN. Apologies.)
On the legal front, I’ve been working with counsel to draft a counterclaim to Meta’s DMCA against llama-dl. (GPT-4 is surprisingly capable, but I’m talking to a few attorneys: https://twitter.com/theshawwn/status/1641841064800600070?s=6... ) An anonymous HN user named L pledged $200k for llama-dl’s legal defense: https://twitter.com/theshawwn/status/1641804013791215619?s=6... This may not seem like much vs Meta, but…
What is lama? What can it do?
On the legal front, I’ve been working with counsel to draft a counterclaim to Meta’s DMCA against llama-dl. (GPT-4 is surprisingly capable, but I’m talking to a few attorneys: https://twitter.com/theshawwn/status/1641841064800600070?s=6... ) An anonymous HN user named L pledged $200k for llama-dl’s legal defense: https://twitter.com/theshawwn/status/1641804013791215619?s=6... This may not seem like much vs Meta, but…
Wish you all luck in the world. We need much more clarity in legal status of these models.
I’m grateful for the opportunity to help protect open source projects such as this one. It will at least give Huggingface a basis to resist DMCAs in the short term.
Does that also mean 6GB VRAM? And does that include Alpaca models like this? https://huggingface.co/elinas/alpaca-30b-lora-int4
According to https://mobile.twitter.com/JustineTunney/status/164190201019... you can probably use the conversion tools from the repo on Alpaca and get the same result. If you want to run larger Alpaca models on a low VRAM GPU, try FlexGen. I think https://github.com/oobabooga/text-generation-webui/ is one of the easier ways to get that going.
I dunno why I thought llama.cpp would support gpus. shrug