Where do you download the tokenizer.model that is needed to convert the GPT4ALL model to the appropriate format?
lesmo provides a magnet uri to all models here. it's inside (499kb): https://github.com/facebookresearch/llama/pull/73/files/016a...
Llama.cpp 30B runs with only 6GB of RAM now
191–200 of 436 posts
Re: Llama.cpp 30B runs with only 6GB of RAM now
#192Does anyone know how/why this change decreases memory consumption (and isn't a bug in the inference code)? From my understanding of the issue, mmap'ing the file is showing that inference is only accessing a fraction of the weight data. Doesn't the forward pass necessitate accessing all the weights and not a fraction of them?
yeah, I believe some readers are misinterpreting the report. The OS manages mmap, it won't show up as "regular" memory utilization because it's lazy-loaded and automatically managed. If the OS can keep the whole file in memory, it will, and it will also magically swap to disk prioritizing explicit memory allocation (malloc).
Sounds like the big win is load time from the optimizations. Also, maybe llama.cpp now supports low-memory systems through mmap swapping? ... at the end of the day, 30B quantized is still 19GB...
Re: Llama.cpp 30B runs with only 6GB of RAM now
#193I might be missing something but I actually couldn't reproduce. I purposefully chose a computer with 16GiB RAM to run the 30B model. Performance was extremely slow, and the process was clearly not CPU-limited, unlike when it's running the 13B model. It's clearly swapping a lot.
Same, performance of the quantised 30B model on my m1 16GB air is absolutely terrible. A couple of things I noticed on activity monitor:
1. "memory used" + "cached files" == 16GB (while swap is zero)
2. Disk reading is 500-600MB/s
3. it seems that every token is computed exactly _after every ~20GB read from disk_ which actually points that for calculating each token it actually re-reads the weights file again (instead of caching it). I actually suspect that swapping may have been more efficient.
The last part (3) that it rereads the whole file again is an assumption and it could just be a coincidence that the new token is computed at every ~20GB read from disk, but it makes sense, as I do not think swapping would have been that inefficient.
Re: Llama.cpp 30B runs with only 6GB of RAM now
#194I'm curious if someone will have to port these enhancements elsewhere, ie: https://github.com/rustformers/llama-rs
Re: Llama.cpp 30B runs with only 6GB of RAM now
#195Earlier quoted context omitted.
AI training has very high minimum requirements to get in the door. If your GPU has 12GB of VRAM and your model and gradients require 13GB, you can't train the model. CPUs don't have this limitation but they are ridiculously inefficient for any training task. There are techniques like ZeRO to give pagefile-like state partitioning to GPU training, but that requires additional engineering. You can't if you have one 12gb…
One thing I don't understand: If it's possible to chunk and parallelize it, is it not relatively straightforward to do these chunks sequentially on a single GPU with a roughly linear increase in runtime? Or are the parallelized computations actually interdependent and involving message-passing, making this unfeasible?
Data moving back and forth from CPU to to bus to GPU and back again for however many chunked model parts you have would increase training time far beyond what you would be willing to invest, not to mention how inefficient and power intensive it is, far more power needed than doing just CPU only or GPU only training. Back to the time part - it's not linear at all. IMO its easily quadratic.
It's not unfeasible, in fact that's how things were done before lots of improvements to the various libraries in essence, many corps still have poorly built pipelines that spend a lot of time in CPU land and not enough in GPU land.
Just an FYI as well - intermediate outputs of models are used in quite a bit of ML, you may see them in some form being used for hyperparameter optimization and searching.
Re: Llama.cpp 30B runs with only 6GB of RAM now
#196Earlier quoted context omitted.
This doesn't even seem that clever, just regular ol' use of mmap where there was none before. Wonder what other performance is being left on the floor. I'm convinced entire power plants could be retired if the world stopped using python unfortunately.
>> I'm convinced entire power plants could be retired if the world stopped using python unfortunately. On the other hand, many business and professionals wouldn't exist :)
I can't find a single good argument for Python based on merit that's not at least 15+ years dated and stems from "But Google is using it".
It's not the easiest syntax, not the best compiler support, performance and threading is a joke. The entire language is based on hype back from the time when the only two mainstream languages were C++ and Java.
Re: Llama.cpp 30B runs with only 6GB of RAM now
#197Earlier quoted context omitted.
> we don't really have a way for the FOSS community to pool together that much money There must be open source projects with enough money to pool into such a project. I wonder whether wikimedia or apache are considering anything.
Maybe we can repurpose the SETI@home infrastructure :)
BOINC might be usable but the existing distributed training setups assume all nodes have very high speed I/O so they can trade gradients and model updates around quickly. The kind of setup that's feasible for BOINC is "here's a dataset shard, here's the last epoch, send me back gradients and I'll average them with the other ones I get to make the next epoch". This is quite a bit different from, say, the single-node case which is entirely serial and model updates happen every step rather than epoch.
Re: Llama.cpp 30B runs with only 6GB of RAM now
#198[dead]
Re: Llama.cpp 30B runs with only 6GB of RAM now
#199Earlier quoted context omitted.
Tragedy of the commons. If you want to do something that benefits everyone a little bit, and you can't productize it like OpenAI's $20/month subscription, then there's no rational economic reason to do it, and you have to wait for someone like me who has an irrational love of coding. It's not a lifestyle that makes you rich, but it does help you see the opportunities to fix problems that the well-resourced folks who…
Tragedy of the commons only work for things you don't directly pay for.
well in a way - open source software something that you don’t directly pay for
Re: Llama.cpp 30B runs with only 6GB of RAM now
#200Does that also mean 6GB VRAM? And does that include Alpaca models like this? https://huggingface.co/elinas/alpaca-30b-lora-int4
According to https://mobile.twitter.com/JustineTunney/status/164190201019... you can probably use the conversion tools from the repo on Alpaca and get the same result. If you want to run larger Alpaca models on a low VRAM GPU, try FlexGen. I think https://github.com/oobabooga/text-generation-webui/ is one of the easier ways to get that going.
I think those specific Alpaca models are all in safetensor now and there isn't simple converter to ggml.