61–70 of 311 posts
Not sure who this is aimed at? The avg programmer probably doesn’t have the gear on hand to run this at the required pace Cool nonetheless
Llama is getting better and better, I heard this and Llama 3 will start to be good as GPT-4. Who would have thought that Meta, that has been chucking billions on the metaverse is on the forefront of Open Source AI. Not to mention their stock is up and they are worth $1TN, again. Not sure how I feel about this given the fact of all the scandals that have plagued them and the massive 1BN fine from the EU, Cambridge Ana…
But I do wonder in the back of my mind why. And I should be suspicious of their angle and I will keep thinking about it. Is it paranoid to think that maybe their angle is putting almost some kind of metadata by style of code being unique to different machines that they can trace generated code to different people? Is that their angle or am I biased in remembering who they have been for the past decade?
Given how good some of the smaller code models are (such as Deepseek Coder at 6.7B), I'll be curious to see what this 70B model is capable of!
Not sure who this is aimed at? The avg programmer probably doesn’t have the gear on hand to run this at the required pace Cool nonetheless
It's aimed at OpenAI's moat. Making sure they don't accumulate too much of one. No one actually has to use this, it just needs to be clear that LLM as a service won't be super high margin because competition can simply start building on Meta's open source releases.
There is a bait and switch going on, and sam altman or mark zuckerberg are the first to tell you. “No one can compete with us, but it’s cute to try! Make applications though” —almost direct quote from Sam Altman. I have 64gb and an RTX 3090 and a macbook M3, and I already can’t run a lot of the newest models even in their quantized form. The business model requires this to be a subscription service. At least as of to…
A 70B model is quite accessible; just rent a data center GPU hourly. There are easy deployment services that are getting better all the time. Smaller models can be derived from the big ones to run on a MacBook running Apple Silicon. While the compute won’t be a match for Nvidia hardware, a MacBook can pack 128GB of RAM and run enormous models - albeit slowly.
So. Maybe we could help other people figure out why VRAM is maxing out. I think it has to do with various new platforms leaking memory.
In my case, I suspect ollama and diffusers are not actually evicting VRAM. nvidia-smi shows it in one case, but I haven’t figured it out yet.
Hey, my point remains. The models are going to get too expensive for me, personally, to run locally. I suspect we’ll default into subscriptions to APIs because the upgrade slope is too steep.
Earlier quoted context omitted.
You can run it on a Macbook M1/M2 with 64GB of RAM.
How? It's larger than 64GB.
I run 33B parameter models on my RTX 3090 (24GB VRAM) no problem. 70B should easily fit into 64GB of RAM.
Llama is getting better and better, I heard this and Llama 3 will start to be good as GPT-4. Who would have thought that Meta, that has been chucking billions on the metaverse is on the forefront of Open Source AI. Not to mention their stock is up and they are worth $1TN, again. Not sure how I feel about this given the fact of all the scandals that have plagued them and the massive 1BN fine from the EU, Cambridge Ana…
I'm really curious what their goal is
Given how good some of the smaller code models are (such as Deepseek Coder at 6.7B), I'll be curious to see what this 70B model is capable of!
AlphaCodium is the newest kid on the block that's SoTA pass@5 on coding tasks (authors claim at least 2x better than GPT4): https://github.com/Codium-ai/AlphaCodium As for small models, Microsoft has been making noise with the unreleased WaveCoder-Ultra-6.7b ( https://arxiv.org/abs/2312.14187 ).
(Please don't say "commoditize your complement" without explaining what exactly they're commoditizing...)