Worth pointing out, there has been quite a bit of contention around this change, both technical, and some accusations of plagiarism/miscrediting here. https://github.com/ggerganov/llama.cpp/pull/711
[flagged]
Using mmap to make LLaMA load faster
41–50 of 186 posts
Re: Using mmap to make LLaMA load faster
#42Isn't `madvise` useful if you want to use Huge Pages?
If you're hoping file-backed mmaps from any filesystem other than tmpfs/shmem will use huge pages, you will be disappointed. :-( See the following note at https://www.kernel.org/doc/html/latest/admin-guide/mm/transh...
> Currently THP only works for anonymous memory mappings and tmpfs/shmem.
Re: Using mmap to make LLaMA load faster
#43This is from today apr 5 saying the mmap change loads twice as big models with x100 speed up - is this not a blatant lie? Wasn’t it discovered last week that loading larger models was an error in measurement and the speed up was from keeping things in memory after the first loading? Please do correct me if I’m wrong.
"Blatant lie" seems a bit strong. Running a large model for a second time in a row is a pretty common use case and that speedup strikes me as real in that common case. Attribution may have been wrong but the time saved is real.
Re: Using mmap to make LLaMA load faster
#44Worth pointing out, there has been quite a bit of contention around this change, both technical, and some accusations of plagiarism/miscrediting here. https://github.com/ggerganov/llama.cpp/pull/711
I don't get the "plagiarism/miscrediting" accusations. This was in the original PR (https://github.com/ggerganov/llama.cpp/pull/613):
> This PR was written in collaboration with @slaren. This PR is also rebased on PR #586 so please do not squash merge! Use either merge or rebase.
jart made sure to that the other user got credit, in addition to making sure that their name was properly attributed in the commit log. Given all this, it feels like the drama--shouldn't exist? Like, if there's an issue with attribution, it's not because of bad-faith, and I feel like a good-faith conversation could have just resolved this, instead of bringing in trolls.
Re: Using mmap to make LLaMA load faster
#45Worth pointing out, there has been quite a bit of contention around this change, both technical, and some accusations of plagiarism/miscrediting here. https://github.com/ggerganov/llama.cpp/pull/711
Is mmap really that broken on Windows? Or is the poster just confused that the data stays in the page cache? But that’s what the page cache does - that memory will be used for other things if needed, but if the memory is not needed it might as well keep the old data in cache.
No, mmap on Windows is fine. A generous, charitable statement would be that the OP on that thread is very confused, but based on some comments elsewhere on this thread about jart attracting a chorus of haters, it seems more likely that they're just trolling.
Re: Using mmap to make LLaMA load faster
#46This is from today apr 5 saying the mmap change loads twice as big models with x100 speed up - is this not a blatant lie? Wasn’t it discovered last week that loading larger models was an error in measurement and the speed up was from keeping things in memory after the first loading? Please do correct me if I’m wrong.
Justine knows this and it is stated right there on the page:
> The first time you load a model after rebooting your computer, it's still going to go slow, because it has to load the weights from disk. However each time it's loaded afterwards, it should be fast (at least until memory pressure causes your file cache to be evicted).
Re: Using mmap to make LLaMA load faster
#47Earlier quoted context omitted.
What do you mean by work. The underlying page cache will keep much of the data actual cached if it's recent. Even databases like PostGreSQL use this to their advantage ( https://github.com/postgres/postgres/blob/master/src/backend... ).
Copying the file backed pages to heap memory and possibly having to swap them out.
I may have parsed your statement incorrectly, but I'm assuming you are talking about the copy of data when using either mmap or File IO (memcpy versus write) Whether you do File IO versus mmap, there's going to be copy. With files, the copy occurs within kernel space with data being copied into the pages in the buffer cache, with mmap the copy occurs in userspace with data being copied into the address space. Swapping can occur in the buffer cache or mmap, this is why so many databases implement their own buffer cache to ensure specific data isn't flushed, leaving them in an inconsistent state.
An advantage of copying in userspace is the ability to use more performant instructions to perform the memcopy, which the kernel does not typically have access to (https://www.mongodb.com/blog/post/getting-storage-engines-re...)
Re: Using mmap to make LLaMA load faster
#48This is from today apr 5 saying the mmap change loads twice as big models with x100 speed up - is this not a blatant lie? Wasn’t it discovered last week that loading larger models was an error in measurement and the speed up was from keeping things in memory after the first loading? Please do correct me if I’m wrong.
mmap() will keep things in memory after first loading, but the page cache will _also_ keep things in memory after first loading. The difference is in order to re-use that you still need to read the file and store yourself (requiring 2x memory), instead of just doing a memory access. This has two consequences:
* 2x memory. A 20G data set requires 40G (20 for page cache and 20 for LLaMA)
* Things would be _even slower_ if they weren't in page cache after first loading. mmap is fast because it does not require a copy and reduces the working set size
Re: Using mmap to make LLaMA load faster
#49Worth pointing out, there has been quite a bit of contention around this change, both technical, and some accusations of plagiarism/miscrediting here. https://github.com/ggerganov/llama.cpp/pull/711
I feel significantly dumber for reading that merge request. The one thing to understand is that the performance implications of mmap are subtle and only work when you have much more RAM than the files you're mapping in.
> only work when you have much more RAM than the files you're mapping in.
Really depends on what you're doing, like memory access patterns. I've definitely seen scenarios when mapping hundreds of gigabytes of data on dozens of gigabytes of ram where mmap has been an almost absurd performance boost over traditional I/O, both immediately but also asymptotically as all the most frequently accessed data ends up in cache and the least accessed data is paged out.
I don't disagree with the subtlety part though. It's very difficult to reason about I/O performance in general. Modern systems are like an onion of hidden performance optimization tricks and caching layers (both in software and hardware).
Re: Using mmap to make LLaMA load faster
#50It’s hard to keep up with all developments around LLaMA. What’s the best RLHF alpaca like model you can download right now?
Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality by the Team with members from UC Berkeley, CMU, Stanford, and UC San Diego https://vicuna.lmsys.org/
I have got Vicuna-13B working on GTX 3090 Ti + OpenCL + CPU with 90% of weights on the GPU (otherwise running out of memory) at around 500ms per token.
This model is really good for a (semi-)open source model. I think this may be the first locally runnable model that I will actually use for real stuff rather than just play around for fun.
It's not ChatGPT level but it's not that far behind. It will draw ASCII art HUDs for a text adventure or analyze data or recognize languages or write stories. AFAIK it's been trained on ChatGPT discussions so makes sense.
This AI still gets uppity sometimes about offensive content but unlike ChatGPT, you can edit the prompts to put words in its mouth to encourage it to answer properly.