211–220 of 313 posts
Earlier quoted context omitted.
I am curious how you are expected to prove in a court of law that the weights you have are derived from some licensed weights if I take just a few gradient steps from the original weights.
Same way that court determines whether my book is a paraphrase of yours. Substantial similarity.
This is why I think we're seeing a Stable Diffusion moment for LLMs: https://simonwillison.net/2023/Mar/11/llama/ Look at the timeline: 24th February 2023: LLaMA is announced, starts being shared with academic partners: https://research.facebook.com/publications/llama-open-and-ef... 2nd March: Someone posts a PR with a BitTorrent link to the models: https://github.com/facebookresearch/llama/pull/73 10th March: First…
I'm in my 40s, so I'm acutely aware of the risk of turning into a "get off my lawn" curmudgeon, so I will only say this: I am glad that I saved a dump of Wikipedia a few months ago, because soon it will probably be impossible to distinguish human-written content from AI.
[flagged]
Earlier quoted context omitted.
Here is one question I have not seen answered yet: All the magic of "7B LLaMA running on a potato" seems to involve lowering precision down to f16 and then further quantizing to int4. Clearly this quantized model still outputs something resembling human language, at the very least. But I haven't seen anyone show what effect this quantizing has on the quality of the output. If the quality of the output is bad, it's un…
> All the magic of "7B LLaMA running on a potato" seems to involve lowering precision down to f16 LLaMa weights are f16s to start out with, no lowering necessary to get to there. You can stream weights from RAM to the GPU pretty efficiently. If you have >= 32GB ram and >=2GB vram my code here should work for you: https://github.com/gmorenz/llama/tree/gpu_offload There's probably a cleaner version of it somewhere else…
This is the kind of thing that the stable diffusion community optimized the shit out
Earlier quoted context omitted.
For 10 billion+ parameter models, the effects of quantization are relatively small, for smaller models like Llama 7B the effect becomes more dramatic, but there is ongoing research on new quantization methods (like GPTQ) that preserve significant performance even on the lower end. Quantization isn't the only technique available for downsizing a model, Llama itself is already the result of sizing the model and input d…
The difference is small, UNTIL you get to 4 bit quantization, where the model is noticeably dumber. 8 bits, imo, is the minimum.
(Note: llama.cpp's 4bit is naive, not GPTQ, and sucks but they are refactoring it to use GPTQ quantization)
References:
https://arxiv.org/abs/2210.17323 - GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers [Oct, 2022]
https://arxiv.org/abs/2212.09720 - The case for 4-bit precision: k-bit Inference Scaling Laws [Dec, 2022]
https://github.com/ggerganov/llama.cpp/issues/9 - llama.cpp: GPTQ Quantization (3-bit and 4-bit) #9
https://github.com/qwopqwop200/GPTQ-for-LLaMa/ - 4 bits quantization of LLaMa using GPTQ
Earlier quoted context omitted.
Here is one question I have not seen answered yet: All the magic of "7B LLaMA running on a potato" seems to involve lowering precision down to f16 and then further quantizing to int4. Clearly this quantized model still outputs something resembling human language, at the very least. But I haven't seen anyone show what effect this quantizing has on the quality of the output. If the quality of the output is bad, it's un…
For 10 billion+ parameter models, the effects of quantization are relatively small, for smaller models like Llama 7B the effect becomes more dramatic, but there is ongoing research on new quantization methods (like GPTQ) that preserve significant performance even on the lower end. Quantization isn't the only technique available for downsizing a model, Llama itself is already the result of sizing the model and input d…
I'm in my 40s, so I'm acutely aware of the risk of turning into a "get off my lawn" curmudgeon, so I will only say this: I am glad that I saved a dump of Wikipedia a few months ago, because soon it will probably be impossible to distinguish human-written content from AI.
[flagged]
We've already seen that, without very restrictive lockdowns by researchers, LLMs will display all the worst kinds of human traits. Why would this be different?
>We are waiting for huggingface to officially support the llama models (i.e. this PR to be merged) before we release a stable version of the finetuning code. Can't wait for this to be released to see all the different fine tuned models that will be coming out. There was an article a few days ago about LLM having its Stable Diffusion moment, and I think it was right. We went from LLama release > llama.cpp > Dalai and…
>We are waiting for huggingface to officially support the llama models (i.e. this PR to be merged) before we release a stable version of the finetuning code. Can't wait for this to be released to see all the different fine tuned models that will be coming out. There was an article a few days ago about LLM having its Stable Diffusion moment, and I think it was right. We went from LLama release > llama.cpp > Dalai and…
I need a chat bot baked into a news app that can summarize the latest developments for me. It's too hard to keep up on my own!