"The recent change also means you can run multiple LLaMA ./main processes at the same time, and they'll all share the same memory resources."
So this could have a main and multiple sub-worker llm processes possibly collaborating while sharing same memory footprint?
Llama.cpp 30B runs with only 6GB of RAM now
211–220 of 436 posts
Re: Llama.cpp 30B runs with only 6GB of RAM now
#212Earlier quoted context omitted.
Didn't expect to see two titans today: ggerganov AND jart. Can ya'll slow down you make us mortals look bad :') Seeing such clever use of mmap makes me dread to imagine how much Python spaghetti probably tanks OpenAI's and other "big ML" shops' infra when they should've trusted in zero copy solutions. Perhaps SWE is dead after all, but LLMs didn't kill it...
You can mmap from python.
The CPython mmap module docs:
https://docs.python.org/3/library/mmap.html
zero_buffer (CFFI, 2013) https://github.com/alex/zero_buffer/blob/master/zero_buffer....
"Buffers on the edge: Python and Rust" (2022) https://alexgaynor.net/2022/oct/23/buffers-on-the-edge/ :
> If you have a Python object and want to obtain its buffer, you can do so with memoryview in Python or PyObject_GetBuffer in C. If you’re defining a class and want to expose a buffer, you can do so in Python by… actually you can’t, only classes implemented in C can implement the buffer protocol. To implement the buffer protocol in C, you provide the bf_getbuffer and bf_releasebuffer functions which are called to obtain a buffer from an object and when that buffer is being released, respectively.
iocursor (CPython C API, ~Rust std::io::Cursor) https://github.com/althonos/iocursor
Arrow Python (C++) > On disk and MemoryMappedFile s: https://arrow.apache.org/docs/python/memory.html#on-disk-and...
"Apache Arrow: Read DataFrame With Zero Memory" (2020) https://towardsdatascience.com/apache-arrow-read-dataframe-w...
pyarrow.Tensor: https://arrow.apache.org/docs/python/generated/pyarrow.Tenso...
ONNX is built on protocolbuffers/protobufs (google/protobufs), while Arrow is built on google/flatbuffers.
FlatBuffers https://en.wikipedia.org/wiki/FlatBuffers :
> It supports “zero-copy” deserialization, so that accessing the serialized data does not require first copying it into a separate part of memory. This makes accessing data in these formats much faster than data in formats requiring more extensive processing, such as JSON, CSV, and in many cases Protocol Buffers. Compared to other serialization formats however, the handling of FlatBuffers requires usually more code, and some operations are not possible (like some mutation operations).
Re: Llama.cpp 30B runs with only 6GB of RAM now
#213Earlier quoted context omitted.
Are the weights on NVME? Old SSD? HDD?
It’s interesting how NVMe will be even more critically important if this lazy weights loading approach works out. PCIe 5 has arrived just in time for LLM interference it seems.
Well in this case it does not have to do with SSDs, quite the opposite here a performance gain seems to happen by caching the file in RAM in the beginning.
Re: Llama.cpp 30B runs with only 6GB of RAM now
#214I don't understand. I thought each parameter was 16 bit (two bytes) which would predict minimally 60GB of RAM for a 30 billion parameter model. Not 6GB.
I was thinking something similar. Turns out that you don't need all the weights for any given prompt. > LLaMA 30B appears to be a sparse model. While there's 20GB of weights, depending on your prompt I suppose only a small portion of that needs to be used at evaluation time [...] Found the answer from the author of this amazing pull request: https://github.com/ggerganov/llama.cpp/discussions/638#discu...
Does this mean LLaMA only uses 10% of it's brain? An urban legend come to life!
Re: Llama.cpp 30B runs with only 6GB of RAM now
#215Earlier quoted context omitted.
>> I'm convinced entire power plants could be retired if the world stopped using python unfortunately. On the other hand, many business and professionals wouldn't exist :)
I can't find a single good argument for Python based on merit that's not at least 15+ years dated and stems from "But Google is using it". It's not the easiest syntax, not the best compiler support, performance and threading is a joke. The entire language is based on hype back from the time when the only two mainstream languages were C++ and Java.
There were plenty of other languages competing with python for the same niche such as perl, ruby, js, php etc... Python is superior to all of those just for syntax alone, it is easier and cleaner to both read and write.
Re: Llama.cpp 30B runs with only 6GB of RAM now
#216Earlier quoted context omitted.
How so?
There are two distinct kinds of jobs: ML researchers and software engineers. A lot of ML researchers write pretty bad code by software engineering standards but that's okay; it's not their job to produce clean code. They string together libraries in Python and do a lot of experimentation and analysis. When they produced something ready to be productionized, software engineers then come in and optimize things. This is…
> ML researchers and software engineers. A lot of ML researchers write pretty bad code by software engineering standards but that's okay
Yes. When you have to try out dozens of research ideas, most of which won't pan out, then you stop writing engineering-style code and switch to hacker mode. Why make it nice when you could be trying 2 more ideas in the meantime. Most of research code it is going to the trash anyway.
Re: Llama.cpp 30B runs with only 6GB of RAM now
#217Earlier quoted context omitted.
Tragedy of the commons only work for things you don't directly pay for.
well in a way - open source software something that you don’t directly pay for
More saliently, the overwhelming majority of the Linux kernel's direct and extended userbase has contributed nothing at all directly to the Linux kernel, as just one example.
Re: Llama.cpp 30B runs with only 6GB of RAM now
#218"How much RAM did you shave off last week?" "Oh, you know, like 12-18GB" "Haha shut the fuck up, how much RAM did you shave off last week" "12-18GB" "Let me tell you what - you show me your commits right now, if you shaved off 12-18GB of RAM last week I quit my job right now and come work for you" https://www.youtube.com/watch?v=TxHITqC5rxE
Maybe not so fast. Other users are reporting that it’s not actually running properly in environments with limited RAM. The reduced memory usage might be more of a reporting misunderstanding, not an actual reduction in memory usage.
Re: Llama.cpp 30B runs with only 6GB of RAM now
#219Earlier quoted context omitted.
IANYL - This is not legal advice. As you may be aware, a counter-notice that meets the statutory requirements will result in reinstatement unless Meta sues over it. So the question isn't so much whether your counter-notice covers all the potential defenses as whether Meta is willing to sue. The primary hurdle you're going to face is your argument that weights are not creative works, and not copyrightable. That argume…
I think the counter on those arguments is that LLM owners want to avoid arguing that the model is a derivative work of the training data. If the LLM is a specific arrangement of the copyrighted works, it's very clearly a derivative work of them
I was not suggesting that an LLM itself consists of an arrangement of the copyrighted works comprising the training data, but that the specific selection of the copyrighted works comprising the training data is part of what differentiates one LLM from another. A strained but useful analogy might be to think of the styles of painting an artist is trained in and/or exposed to prior to creating their own art. Obvious or subtle, the art style an artist has studied would likely impact the style they develop for themself.
However, to address your point about derivative works directly, the consensus among copyright law experts appears to be that whether a particular model output is infringing depends on the standard copyright infringement analysis (and that’s regardless of the minor and correctable issue represented by memorization/overfitting of duplicate data in training sets). Only in the most unserious legal complaint (the class action filed against Midjourney, Stability AI, etc.) is the argument being made and that the models actually contain copies of the training data.
Re: Llama.cpp 30B runs with only 6GB of RAM now
#220Earlier quoted context omitted.
Tragedy of the commons. If you want to do something that benefits everyone a little bit, and you can't productize it like OpenAI's $20/month subscription, then there's no rational economic reason to do it, and you have to wait for someone like me who has an irrational love of coding. It's not a lifestyle that makes you rich, but it does help you see the opportunities to fix problems that the well-resourced folks who…
Tragedy of the commons only work for things you don't directly pay for.
Exactly, the software supplier isn't paying for RAM.