Are we celebrating theft from tech companies now?
Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model
101–110 of 143 posts
Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model
#102Earlier quoted context omitted.
MD5 signatures don't mean much now that hash collisions could be created instantly on consumer hardware. MD5 is only good for checking for unintentional data corruption.
>hash collisions could be created instantly on consumer hardware Collisions can be created, but MD5 is still preimage resistant. As long as someone with the actual model made the hash and Meta didn't generate colliding models themselves. You can trust it.
[deleted]
Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model
#103Earlier quoted context omitted.
> That definition would apply to almost anything software produces Not really. The reason software can be copyrighted at all is because the actual code (and resulting object code) is creative. Courts have named this threshold the "Structure, sequence and organization" of the work. ML models don't follow any creative SSO the way actual code does. > Also the output of ML seems to belong to whoever pressed the generate…
I think that copyright law works differently. Source code is copyright; the expression as compiled code from that source enjoys the same protections. If the model can be copyrighted, the expression of the model in the form of its weights is probably also protected.
You're correct, but that doesn't disprove my point. I'm saying the model itself is uncopyrightable.
Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model
#104Earlier quoted context omitted.
The compute necessary to run 65B naively was only available on AWS (and perhaps Azure, I don't work with them) and the required instance types have been unavailable to the public recently (it seems everyone had the same idea to hop on this and try to run it). In my other post here [1], the memory requirements have been lowered through other work, and it should now be possible to run the 65B on a provider like CoreWea…
Are you sure about that? I can't remember where I saw the table of memory requirements, but I'm sure some of the larger instances here [1] will surely be able to cope (assuming they're available!) Oracle gives you a $300 free trial, which equates to running BM.GPU4.8 for over 10 hours - enough for a focused day of prompting [1] https://www.oracle.com/cloud/compute/gpu/
If you actually try and do this, the sales people will stop you due to some internal rule. No GPUs on free credit. Unless the situation has changed of course..
Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model
#105Are we celebrating theft from tech companies now?
I mean highseas, adblockers, bypassing paywalls, each one of them is theft.
But on the flipside, companies are constantly trying to keep the ownership of data we paid for full price, scooping up personal data, selling low quality work behind paywall.
Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model
#106Earlier quoted context omitted.
I don't understand, the Ampere they refer to in their free tier are cpu's not gpu's. How did you manage to do that
Custom PyTorch with on-chip acceleration: https://cloudmarketplace.oracle.com/marketplace/en_US/listin... Not as fast as a GPU, but less than 5 seconds for a 250 token response is good enough for a Discord bot.
This is the most interesting thing I've read in this thread. How have I never heard of this accelerator?!
Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model
#107Earlier quoted context omitted.
Seconded. Do write it up. I see vast.ai listing interruptible instance with a single A100 80GB at $1/hour which is pretty reasonable. ChatGPT plus is $20/month which would be roughly 20 hours of use and I wont't be lectured like I'm in a kindergarten or something. A bonus point would be to make the writeup accessible for AI challenged developers. Asking for a friend.
I would like to support this request for AI challenged developers :) For things like these, I always wonder: How much slower would it be to run such a model on a CPU? I mean, clearly a lot less interactive, but is it possible at all? Could it be chopped up and "streamed" to a GPU with less memory halfway efficiently? What is the bottleneck currently on GPUs, memory bw or compute?
On a CPU I'd estimate it would get a maximum of around 5 tokens per second (a token being a sub-word token, so generally a couple of letters). I suspect it'd be more like 1 token per second on the large model without additional optimisation.
Yes models can be split up. See eg Hugging Face Accelerate.
Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model
#108Earlier quoted context omitted.
Seconded. Do write it up. I see vast.ai listing interruptible instance with a single A100 80GB at $1/hour which is pretty reasonable. ChatGPT plus is $20/month which would be roughly 20 hours of use and I wont't be lectured like I'm in a kindergarten or something. A bonus point would be to make the writeup accessible for AI challenged developers. Asking for a friend.
I would like to support this request for AI challenged developers :) For things like these, I always wonder: How much slower would it be to run such a model on a CPU? I mean, clearly a lot less interactive, but is it possible at all? Could it be chopped up and "streamed" to a GPU with less memory halfway efficiently? What is the bottleneck currently on GPUs, memory bw or compute?
[deleted]
Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model
#109If anyone is interested in running this at home, please follow the llama-int8 project [1]. LLM.int8() is a recent development allowing LLMs to run in half the memory without loss of performance [2]. Note that at the end of [2]'s abstract, the authors state "This result makes such models much more accessible, for example making it possible to use OPT-175B/BLOOM on a single server with consumer GPUs. We open-source our…
I feel like we're less than a decade away from being able to hook LLMs into gaming. How incredible would it be to have NPCs driven by LLM?
Honestly I don't think it would be completely impossible now in a limited fashion.
Imagine playing a level and doing some particular feats in it. They get presented to GPT with a prompt and the story gets send to a AI voice model in game where the NPC asks/tells the player character about it.
Re: Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model
#110Are we celebrating theft from tech companies now?
Copying isn't theft. If you bought the ssd it belongs to you in its entirety regardless what state you decide to configure it into.