How come a company as big as Meta still uses bit.ly ?
Because this is a marketing channel. They handle tracking of FB/IG messages by other means, intended for engineers.
Meta AI releases Code Llama 70B
161–170 of 311 posts
Re: Meta AI releases Code Llama 70B
#162Earlier quoted context omitted.
It's a legit question, the model will be worse in some way... I've seen it discussed that all things being equal more parameters is better (meaning it's better to take a big model and quantized it to fit in memory than use a smaller unquantized model that fits), but a quantized model wouldn't be expected to run identically to or as well as the full model.
You don’t stop being andy99 just because you’re a little tired, do you? Being tired makes everyone a little less capable at most things. Sometimes, a lot less capable. In traditional software, the same program compiled for 32-bit and 64-bit architectures won’t be able to handle all of the same inputs, because the 32-bit version is limited by the available address space. It’s still the same program. If we’re not willi…
I don't like the metaphor: when I'm tired, I will be alert again later. Quantization is lossy compression: the human equivalent would be more like a traumatic brain injury affecting recall, especially of fine details.
The question of whether I am still me after a traumatic brain injury is philosophically unclear, and likely depends on specifics about the extent of the deficits.
Re: Meta AI releases Code Llama 70B
#163I'm not very plugged into how to use these models, but I do love and pay for both ChatGPT and GitHub Copilot. How does one take a model like this (or a smaller version) and leverage it in VS Code? There's a dizzying array of GPT wrapper extensions for VS Code, many of which either seem like kind of junk (10 d/ls, no updates in a year), or just lead to another paid plan, at which point I might as well just keep my GH Copilot. Curious what others are doing here for Copilot-esque code completion without Copilot.
Re: Meta AI releases Code Llama 70B
#164Earlier quoted context omitted.
You don’t stop being andy99 just because you’re a little tired, do you? Being tired makes everyone a little less capable at most things. Sometimes, a lot less capable. In traditional software, the same program compiled for 32-bit and 64-bit architectures won’t be able to handle all of the same inputs, because the 32-bit version is limited by the available address space. It’s still the same program. If we’re not willi…
I don't like the metaphor: when I'm tired, I will be alert again later. Quantization is lossy compression: the human equivalent would be more like a traumatic brain injury affecting recall, especially of fine details. The question of whether I am still me after a traumatic brain injury is philosophically unclear, and likely depends on specifics about the extent of the deficits.
The impact on accuracy is somewhere in the single-digit percentages at 4-bit quantization, from what I’ve been able to gather. Very small impact. To draw the analogy out further, if the model was able to get an A on a test before quantization, it would likely still get a B at worst afterwards, given a drop in the score of less than 10%. Depending on the task, the measured impact could even be negligible.
It’s far more similar to the model being perpetually tired than it is to a TBI.
You may nitpick the analogy, but analogies are never exact. You also ignore the other piece that I pointed out, which is how we treat other software that comes in multiple slightly different forms.
Re: Meta AI releases Code Llama 70B
#165Can you explain why big tech company make a race to release an open source model? If model is free and open source then how will they earn and how will they compete with others?
they want to incentivize dependency
Re: Meta AI releases Code Llama 70B
#166Earlier quoted context omitted.
It's aimed at OpenAI's moat. Making sure they don't accumulate too much of one. No one actually has to use this, it just needs to be clear that LLM as a service won't be super high margin because competition can simply start building on Meta's open source releases.
The moat is all but guaranteed to be the scale of the GPUs required to operate these for a lot of users as they get ever larger, specifically the extreme cost that is going along with that. Anybody have $10 billion sitting around to deploy that gigantic open source set-up for millions of users? There's your moat and only a relatively few companies will be able to do it. One of Google's moats is, has been, and will al…
>The moat is all but guaranteed to be the scale of the GPUs required to operate these
You don't have to run them locally.
Re: Meta AI releases Code Llama 70B
#167Earlier quoted context omitted.
Just make sure you're comfortable with manually compiling the bitsandbytes and generally combine a software stack of almost out of date libraries
That's a good point. Are you referring to the out of date cuda libraries?
I don't remember exactly (either cuda directly or the cudnn version used by the flashattention)... Anyway, /r/localLlama has few instances of such builds. Might be really worthwhile looking that up before buying
Re: Meta AI releases Code Llama 70B
#168I'm not very plugged into how to use these models, but I do love and pay for both ChatGPT and GitHub Copilot. How does one take a model like this (or a smaller version) and leverage it in VS Code? There's a dizzying array of GPT wrapper extensions for VS Code, many of which either seem like kind of junk (10 d/ls, no updates in a year), or just lead to another paid plan, at which point I might as well just keep my GH…
https://continue.dev/ is a good place to start.
Re: Meta AI releases Code Llama 70B
#169Curious what's the current SOTA local copilot model? Are there any extensions in vscode that give you a similar experience? I'd love something more powerful than copilot for local use (I have a 4090, so I should be able to run a decent number of models).
When this 70b model gets quantized you should be able to run it fine on your 4090. Check out 'TheBloke' on huggingface and the llamacpp to run the gguf files.
I think your take is a bit optimistic. I like quantization as much as the next person, but even the 2-bit model won’t fit entirely on a 4090: https://huggingface.co/TheBloke/Llama-2-70B-GGUF
I would be uncomfortable recommending less than 4-bit quantization on a non-MoE model, which is ~40GB on a 70B model.
Re: Meta AI releases Code Llama 70B
#170Earlier quoted context omitted.
It's aimed at OpenAI's moat. Making sure they don't accumulate too much of one. No one actually has to use this, it just needs to be clear that LLM as a service won't be super high margin because competition can simply start building on Meta's open source releases.
The moat is all but guaranteed to be the scale of the GPUs required to operate these for a lot of users as they get ever larger, specifically the extreme cost that is going along with that. Anybody have $10 billion sitting around to deploy that gigantic open source set-up for millions of users? There's your moat and only a relatively few companies will be able to do it. One of Google's moats is, has been, and will al…
> VRAM isn't moving nearly as fast as the models are progressing in size. And it's never going to. The cost will get ever greater to operate these at scale.
It is in at least 2025. AMD (and Intel, maybe) will have M-Pro-Esque APUs that can run a 70B model at very reasonable speeds.
I am pretty sure Intel is going to rock the VRAM boat on desktops as well. They literally have no market to lose, unlike AMD which infuriatingly still artificially segments their high VRAM cards.