91–100 of 436 posts
Is the 30B model clearly better than the 7B? I played with Pi3141/alpaca-lora-7B-ggml two days ago and it was super disappointing. In percentage between 0% = alpaca-lora-7B-ggml and 100% GPT-3.5, where would LLaMA 30B be positioned?
Earlier quoted context omitted.
AI training has very high minimum requirements to get in the door. If your GPU has 12GB of VRAM and your model and gradients require 13GB, you can't train the model. CPUs don't have this limitation but they are ridiculously inefficient for any training task. There are techniques like ZeRO to give pagefile-like state partitioning to GPU training, but that requires additional engineering. You can't if you have one 12gb…
Maybe a good candidate for the SETI@home treatment?
Does this mean that we can also run the 60B model on a 16GB ram computer now? I have the M2 air and can't wait until further optimisation with the Neural Engine / multicore gpu + shared ram etc. I find it absolutely mind boggling that GPT-3.5(4?) level quality may be within reach locally on my $1500 laptop / $800 m2 mini.
I doubt it: text size and text pattern size don't scale linearly.
Earlier quoted context omitted.
I don't think it's actually trading away inference speed. You can pass an --mlock flag, which calls mlock() on the entire 20GB model (you need root to do it), then htop still reports only like 4GB of RAM is in use. My change helps inference go faster. For instance, I've been getting inference speeds of 30ms per token after my recent change on the 7B model, and I normally get 200ms per eval on the 30B model.
Very cool! Are you testing after a reboot / with an empty page cache?
Does anyone know how/why this change decreases memory consumption (and isn't a bug in the inference code)? From my understanding of the issue, mmap'ing the file is showing that inference is only accessing a fraction of the weight data. Doesn't the forward pass necessitate accessing all the weights and not a fraction of them?
Great to see this advancing! I’m curious if anyone knows what the best repo is for running this stuff on an Nvidia GPU with 16GB vram. I ran the official repo with the leaked weights and the best I could run was the 7B parameter model. I’m curious if people have found ways to fit the larger models on such a system.
The pace of collaborative OSS development on these projects is amazing, but the rate of optimisations being achieved is almost unbelievable. What has everyone been doing wrong all these years cough sorry, I mean to say weeks? Ok I answered my own question.
It’s several things:
* Cutting-edge code, not overly concerned with optimization
* Code written by scientists, who aren’t known for being the world’s greatest programmers
* The obsession the research world has with using Python
Not surprising that there’s a lot of low-hanging fruit that can be optimized.
Earlier quoted context omitted.
> But we don't have a compelling enough theory yet to explain the RAM usage miracle. My guess would be that the model is faulted into memory lazily page by page (4K or 16K chunks) as the model is used, so only the actual parts that are needed are loaded. The kernel also removes old pages from the page cache to make room for new ones, and especially so if the computer is using a lot of its RAM. As with all performance…
I don't think it's actually trading away inference speed. You can pass an --mlock flag, which calls mlock() on the entire 20GB model (you need root to do it), then htop still reports only like 4GB of RAM is in use. My change helps inference go faster. For instance, I've been getting inference speeds of 30ms per token after my recent change on the 7B model, and I normally get 200ms per eval on the 30B model.
> Someone mentioning "32-bit systems"
Um no, you're not mapping 6GB on RAM on a 32-bit system. The address space simply doesn't exist.