Can this be downloaded like stable diffusion? How big are these models? How long does it take to run a "query" against them on a decent gaming PC?
No one seems to be running it locally at the moment.
LLaMA: A foundational, 65B-parameter large language model
131–140 of 209 posts
Re: LLaMA: A foundational, 65B-parameter large language model
#132Why not a 1 Billion parameter model that can run on a single GPU ?
worse is better will come to LLM’s.
Re: LLaMA: A foundational, 65B-parameter large language model
#133So what hardware do we need to run this model?
7 billion can run on 16+ gb GPUs as fp16, 14 billion can be run on 16+ gb if quantized to int8. 14G @ fp16 and 30G at int8 will require one of the 48 gb cards (less, but hardware mostly goes 24 -> 48).
Requirements could be reduced with something like DeepSpeed or ColossalAI (or even just simple hacks to move bits to RAM more aggressively)
Re: LLaMA: A foundational, 65B-parameter large language model
#134Earlier quoted context omitted.
I don't know what my bias is supposed to be but I called them dorks affectionately for one. The other replies are literally arguing that they are very extremely supervised where as I am speculating they are just eager to share their work for the right reasons and the eye of Sauron has yet to turn upon them. Inside knowledge I never claimed. Anything else I can help you with today? :)
> I called them dorks affectionately Never in my life have I seen that word used affectionately
I think Noah Kagan's blog is literally called "OK Dork", and I always assumed the title was self-deprecating (in a fun/positive way) rather than negative.
Re: LLaMA: A foundational, 65B-parameter large language model
#135Earlier quoted context omitted.
Today we're releasing a new state-of-the-art AI large language model called LLaMA designed to help researchers advance their work. LLMs have shown a lot of promise in generating text, having conversations, summarizing written material, and more complicated tasks like solving math theorems or predicting protein structures. Meta is committed to this open model of research and we'll make our new model available to the A…
It is to please the shareholders. Now shareholder can know that Meta can compete with GPT3.
100%. This kind of announcement is for the street. Investors need to be reassured that Meta is keeping up with the cool AI stuff over at Google and Microsoft. A release like this will cause a flurry of downstream news coverage. I’m sure they are hoping that the model will be picked up by researchers doing interesting things who might further generate good coverage.
And, yes, it fills the moat.
Re: LLaMA: A foundational, 65B-parameter large language model
#136Quick notes from first glance at paper https://research.facebook.com/publications/llama-open-and-ef... : * All variants were trained on 1T - 1.4T tokens; which is a good compared to their sizes based on the Chinchilla-metric. Code is 4.5% of the training data (similar to others). [Table 2] * They note the GPU hours as 82,432 (7B model) to 1,022,362 (65B model). [Table 15] GPU hour rates will vary, but let's give a ra…
> we used 2048 A100-80GB for a period of approximately 5 months Do we know how much total energy a human consumes from birth to 20 yo? Something like 2000 calories integrated over 20 years. How does it compare to the GPUs above? Wolfram Alpha: - human - 17 MW/h ((2000 calories per day) over 20 years in MWh) - GPUs - 3000 MW/h ((2048 * 400) W over 5 months in MWh) We still have the edge. LOL, I'm being downvoted, I wo…
You have to include our evolutionary history too. A considerable amount of our sophisticated behavior doesn't require specific training, as it is encoded in our genetic and epigenetic systems. We aren't starting from zero.
Re: LLaMA: A foundational, 65B-parameter large language model
#137How big is one of these models once it is trained and tuned? Could it be run on a single reasonably-large EC2? Or does it need any special architecture?
Re: LLaMA: A foundational, 65B-parameter large language model
#138Quick notes from first glance at paper https://research.facebook.com/publications/llama-open-and-ef... : * All variants were trained on 1T - 1.4T tokens; which is a good compared to their sizes based on the Chinchilla-metric. Code is 4.5% of the training data (similar to others). [Table 2] * They note the GPU hours as 82,432 (7B model) to 1,022,362 (65B model). [Table 15] GPU hour rates will vary, but let's give a ra…
(1022362 + 82432) gpu-hours / 2048gpus / 5 months ~= 15% uptime.
That's only 0.08 nines of availability!
I remember in one of their old guidebooks a lot of struggle to keep their 64 machine (512 gpu) cluster running this was probably 4x the machines and 4x the number of cluster dropouts.
Re: LLaMA: A foundational, 65B-parameter large language model
#139Earlier quoted context omitted.
For a specific example some variation of "fast inverse square root" will usually get you the exact GPL licensed code from Quake III, comments included.
Do you mean the same code that has its own Wikipedia page where the exact code is written, comments included, and has probably been copy pasted into 100’s of other projects? https://en.m.wikipedia.org/wiki/Fast_inverse_square_root
You mean this code?
https://archive.softwareheritage.org/browse/content/sha1_git...
Do you see that notice at the top of the file? It says:
==
This file is part of Quake III Arena source code.
Quake III Arena source code is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.
===
but because it's been laundered by Microsoft, you think it's okay to steal free software and make it proprietary?
Re: LLaMA: A foundational, 65B-parameter large language model
#140Earlier quoted context omitted.
> we used 2048 A100-80GB for a period of approximately 5 months Do we know how much total energy a human consumes from birth to 20 yo? Something like 2000 calories integrated over 20 years. How does it compare to the GPUs above? Wolfram Alpha: - human - 17 MW/h ((2000 calories per day) over 20 years in MWh) - GPUs - 3000 MW/h ((2048 * 400) W over 5 months in MWh) We still have the edge. LOL, I'm being downvoted, I wo…
You have to include our evolutionary history too. A considerable amount of our sophisticated behavior doesn't require specific training, as it is encoded in our genetic and epigenetic systems. We aren't starting from zero.
Then you would need to include the our history in the GPU calculation. GPUs require evolutionary bootstrapping - they didn't materialize alongside the first few hydrogen atoms post BB.