~220 GB :O That's quite big!
Needs ~200GB of graphics ram to run... Not many people will get this running!
I need just two more 24gb Tesla's and I can do it!
Facebook LLAMA is being openly distributed via torrents
131–140 of 719 posts
Re: Facebook LLAMA is being openly distributed via torrents
#132Earlier quoted context omitted.
For anyone curious, it took 2048 A100 GPUs to train LLaMa, each GPU costs roughly $15k, facebook probably gets some sort of discount. That's a $30Mil if you want to train at that scale. Also IIRC it took 23 days to train the biggest model. Someone else can do the power consumption cost calculations.
$30m training cost is too high. Amazon's p4d.24xlarge is $32.77 an hour for 8 A100 GPUs. 2048 A100 GPUs for 23 days costs $4.6m at that rate. You might even get a discount.
At the same time I guarantee you they didn’t get it right the first time. I’m sure there were multiple (both serially and in parallel) runs as they worked out kinks and tuned hyper parameters.
Re: Facebook LLAMA is being openly distributed via torrents
#133Earlier quoted context omitted.
Great, now how do I run it? Do I need a GPU with over 65GB RAM?
Try this, it's for running llms that won't fit in the gpu: https://github.com/FMInference/FlexGen
Currently that looks like it only supports facebook's opt and galactica models. Though they do appear to plan to add support for more models.
Re: Facebook LLAMA is being openly distributed via torrents
#134Earlier quoted context omitted.
The most time/cost optimal solution is probably to buy 32 or 64 gigs of ram. That'll still be slow but most people are already half way there.
That will be very VERY slow. Pcie bandwidth is way too slow.
Should be like an order of magnitude faster than trying to run it from a NVMe still, no? I've ran some small flan models from RAM and it was fine, but yeah it's not exactly realtime.
Re: Facebook LLAMA is being openly distributed via torrents
#135old school opensource, which is a bit surprising from meta. I wonder how they managed to square that with legal. Someone must have been very good friends with Zuck.
> old school opensource, which is a bit surprising from meta Aren't you a cheeky lad? Metea turned out lots of open-source database systems: * RocksDB * Hive * Presto * Cassandra * Velox LFP
and, you know... React.
Re: Facebook LLAMA is being openly distributed via torrents
#136In case it's not clear what's happening here (and from the comments it doesn't seem like it is), someone (not Meta) leaked the models and had the brilliant idea of advertising the magnet link through a GitHub pull request. The part about saving bandwidth is a joke. Meta employees may have not noticed or are still figuring out how to react, so the PR is still up. (Disclaimer: I work at Meta, but have no relationship w…
someone (not Meta)
Did you mean to write "someone (possibly Meta)" ?
Re: Facebook LLAMA is being openly distributed via torrents
#137Earlier quoted context omitted.
My understanding is that weights are normally stored as pickled python blobs, which means arbitrary code execution as they are unpickled.
"They turned the model into a pickle? Funniest shit I've ever seen." But seriously, why not something more human readable and text-based if it's just weights?
... why not CBOR or other efficient binary format?
Re: Facebook LLAMA is being openly distributed via torrents
#138Re: Facebook LLAMA is being openly distributed via torrents
#139Earlier quoted context omitted.
-1: as an ex-googler, I can say it was hard enough for Google itself to get its code to run, given gonzo infrastructure assumptions, proprietary libraries/languages, etc.
That speaks volumes of the code quality @ Google.
You ought to be able to arrive at the same conclusion, then, with these LLMs. Without arrays of GPUs, it would take thousands of years to train one. Without a corpus of billions or trillions of words, one would produce output of very limited utility.
I think you have to consider that some things are systems, and it is the assembly of their components that imparts the true quality.
Re: Facebook LLAMA is being openly distributed via torrents
#140I give it a week before we see tools for subtly watermarking your secret LLM's weights, so you can trace leaks like this later.
Watermarking the weights is trivial. Watermarking the output is also possible, but more complex and with a statistical success rate Vs performance tradeoff.
I love the idea that LLMs will get watermarked in a way where you can ask them who they were built for and they just tell you.