101–110 of 311 posts
Earlier quoted context omitted.
Quantization is highly effective at reducing memory and storage requirements, and it barely has any impact on quality (unless you take it to the extreme). Approximately no one should ever be running the full fat fp16 models during inference of any of these LLMs. That would be incredibly inefficient. I run 33B parameter models on my RTX 3090 (24GB VRAM) no problem. 70B should easily fit into 64GB of RAM.
I'm aware but is it still LLaMA 70B at that point?
Given how good some of the smaller code models are (such as Deepseek Coder at 6.7B), I'll be curious to see what this 70B model is capable of!
AlphaCodium is the newest kid on the block that's SoTA pass@5 on coding tasks (authors claim at least 2x better than GPT4): https://github.com/Codium-ai/AlphaCodium As for small models, Microsoft has been making noise with the unreleased WaveCoder-Ultra-6.7b ( https://arxiv.org/abs/2312.14187 ).
Earlier quoted context omitted.
AlphaCodium is the newest kid on the block that's SoTA pass@5 on coding tasks (authors claim at least 2x better than GPT4): https://github.com/Codium-ai/AlphaCodium As for small models, Microsoft has been making noise with the unreleased WaveCoder-Ultra-6.7b ( https://arxiv.org/abs/2312.14187 ).
Are weights available?
Llama is getting better and better, I heard this and Llama 3 will start to be good as GPT-4. Who would have thought that Meta, that has been chucking billions on the metaverse is on the forefront of Open Source AI. Not to mention their stock is up and they are worth $1TN, again. Not sure how I feel about this given the fact of all the scandals that have plagued them and the massive 1BN fine from the EU, Cambridge Ana…
Earlier quoted context omitted.
Quantization is highly effective at reducing memory and storage requirements, and it barely has any impact on quality (unless you take it to the extreme). Approximately no one should ever be running the full fat fp16 models during inference of any of these LLMs. That would be incredibly inefficient. I run 33B parameter models on my RTX 3090 (24GB VRAM) no problem. 70B should easily fit into 64GB of RAM.
I'm aware but is it still LLaMA 70B at that point?
Can someone explain Meta's strategy with the open source models here? Genuine question, I don't fully undestand. (Please don't say "commoditize your complement" without explaining what exactly they're commoditizing...)
Total speculation: Yann LeCun is there and he is really passionate about the technology and openness
Earlier quoted context omitted.
We actually run already in-house ollama server prototype for coding assistance with deepseek coder and it is pretty good. Now if we would get a model for this, that is on chatgpt 4 level, I would be super happy.
Did you finetune a model?
How come a company as big as Meta still uses bit.ly ?
Earlier quoted context omitted.
How? It's larger than 64GB.
Quantization is highly effective at reducing memory and storage requirements, and it barely has any impact on quality (unless you take it to the extreme). Approximately no one should ever be running the full fat fp16 models during inference of any of these LLMs. That would be incredibly inefficient. I run 33B parameter models on my RTX 3090 (24GB VRAM) no problem. 70B should easily fit into 64GB of RAM.
Earlier quoted context omitted.
Quantization is highly effective at reducing memory and storage requirements, and it barely has any impact on quality (unless you take it to the extreme). Approximately no one should ever be running the full fat fp16 models during inference of any of these LLMs. That would be incredibly inefficient. I run 33B parameter models on my RTX 3090 (24GB VRAM) no problem. 70B should easily fit into 64GB of RAM.
I'm aware but is it still LLaMA 70B at that point?