Read that as nmap and got really confused.
Using mmap to make LLaMA load faster
31–40 of 186 posts
Re: Using mmap to make LLaMA load faster
#32Isn't `madvise` useful if you want to use Huge Pages?
Re: Using mmap to make LLaMA load faster
#33Earlier quoted context omitted.
I feel the same. I maybe should not be surprised, given that we live in the era of Unity and Electron, but using mmap() to load large files should be not be seen as rocket science. And this is basically available on almost any platform with a MMU and a kernel.
I think there just hasn't been a consumer application that is really resource constrained, for a long time now. Only things for enthusiasts have been. LLMs have product market fit, but running a useful one client side is resource constrained, but instead of it truly being a consumer hardware limitation, it just turns out they were never optimized to begin with - coming from the perceived "top AI/ML minds" at FAANGs,…
> never optimized to begin with
I think the better read is that they're being adapted to new applications, constraints, and environments, all at once.
Re: Using mmap to make LLaMA load faster
#34Earlier quoted context omitted.
I feel the same. I maybe should not be surprised, given that we live in the era of Unity and Electron, but using mmap() to load large files should be not be seen as rocket science. And this is basically available on almost any platform with a MMU and a kernel.
I think there just hasn't been a consumer application that is really resource constrained, for a long time now. Only things for enthusiasts have been. LLMs have product market fit, but running a useful one client side is resource constrained, but instead of it truly being a consumer hardware limitation, it just turns out they were never optimized to begin with - coming from the perceived "top AI/ML minds" at FAANGs,…
Probably a combination of (a) ML framework people not paying much attention to CPU inference due to already having GPUs/TPUs already lying around for training - CPU inference is just for very quick experiments (b) research code has never been the best optimized for performance (c) ML people are not generally systems programmers, and a lot of systems programmers are afraid to mess with the ML code outside of low-level computation kernels (doesn't help that ML code is notoriously unreproducible).
Re: Using mmap to make LLaMA load faster
#35Earlier quoted context omitted.
In this case, the main benefit is from multiple invocations of the same program. Using mmap, you avoid doing any work at all the 2nd time you load the file.
What do you mean by work. The underlying page cache will keep much of the data actual cached if it's recent. Even databases like PostGreSQL use this to their advantage ( https://github.com/postgres/postgres/blob/master/src/backend... ).
Copying the file backed pages to heap memory and possibly having to swap them out.
Re: Using mmap to make LLaMA load faster
#36This is from today apr 5 saying the mmap change loads twice as big models with x100 speed up - is this not a blatant lie?
Wasn’t it discovered last week that loading larger models was an error in measurement and the speed up was from keeping things in memory after the first loading?
Please do correct me if I’m wrong.
Re: Using mmap to make LLaMA load faster
#37Earlier quoted context omitted.
I feel the same. I maybe should not be surprised, given that we live in the era of Unity and Electron, but using mmap() to load large files should be not be seen as rocket science. And this is basically available on almost any platform with a MMU and a kernel.
I think there just hasn't been a consumer application that is really resource constrained, for a long time now. Only things for enthusiasts have been. LLMs have product market fit, but running a useful one client side is resource constrained, but instead of it truly being a consumer hardware limitation, it just turns out they were never optimized to begin with - coming from the perceived "top AI/ML minds" at FAANGs,…
> while some of the most basic optimizations are seemingly a lost art
mmap isn't relevant to anyone except CPU-using programmers because other hardware doesn't have virtual memory paging. Firmware programmers don't care, GPU programmers don't care.
Re: Using mmap to make LLaMA load faster
#38jart is a genius. What they’ve done with Blink, Cosmopolitan C, Redbean, and now llama.cpp is incredible. It gives me hope for the future of systems/low-level programming.
Using mmap is basic systems programming knowledge and not "incredible" even though it's a useful contribution.
Re: Using mmap to make LLaMA load faster
#39Worth pointing out, there has been quite a bit of contention around this change, both technical, and some accusations of plagiarism/miscrediting here. https://github.com/ggerganov/llama.cpp/pull/711
Is mmap really that broken on Windows? Or is the poster just confused that the data stays in the page cache? But that’s what the page cache does - that memory will be used for other things if needed, but if the memory is not needed it might as well keep the old data in cache.
Re: Using mmap to make LLaMA load faster
#40Since the post is from day, so the improvements were all ‘real’? I didn’t follow closely but I remember multiple points people brought up earlier like: is the memory counting correct, why aren’t all the weights accessed for a query, whether quantisation is a problem etc. Were all these fixed?
There are no memory improvements, people were not measuring correct. The giant improvement is the load times after the first run(if you do not invalidate your caches). Quantization to 4 bit is a big gain, the loss appears to be minimal from benchmarks. So with quantization you gain the ability to try a bigger model, if you have the hardware to fit the biggest model then you can skip it but for most people we need to…
> There are no memory improvements, people were not measuring correct.
Using filebacked pages instead of anonymous memory is a real improvement because it doesn't have to get swapped out if there's memory pressure. And this program probably isn't the only thing running on the machine.