Using mmap to make LLaMA load faster • Hacker News Board

Re: Using mmap to make LLaMA load faster

#21

Post by detrites » Wed, Apr 05, 2023, 4:44 PM UTC

isoprophlex wrote: post #13

menzoic wrote: post #8

Earlier quoted context omitted.

Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality by the Team with members from UC Berkeley, CMU, Stanford, and UC San Diego https://vicuna.lmsys.org/

Are the Vicuna weights available for download, and are they llama.cpp compatible? I can't grok that by skimming the page...

Vicuna's fine-tune of the LLaMA weights are available for download, called "deltas".

So you get the LLaMA weights (somewhere), then apply the Vicuna deltas to them to end up with the Vicuna model.

Re: Using mmap to make LLaMA load faster

#22

Post by londons_explore » Wed, Apr 05, 2023, 4:47 PM UTC

dekhn wrote: post #10

NickGerleman wrote: post #6

Worth pointing out, there has been quite a bit of contention around this change, both technical, and some accusations of plagiarism/miscrediting here. https://github.com/ggerganov/llama.cpp/pull/711

I feel significantly dumber for reading that merge request. The one thing to understand is that the performance implications of mmap are subtle and only work when you have much more RAM than the files you're mapping in.

In this case, the main benefit is from multiple invocations of the same program.

Using mmap, you avoid doing any work at all the 2nd time you load the file.

Re: Using mmap to make LLaMA load faster

#23

Post by bigodanktime » Wed, Apr 05, 2023, 4:48 PM UTC

NickGerleman wrote: post #6

Worth pointing out, there has been quite a bit of contention around this change, both technical, and some accusations of plagiarism/miscrediting here. https://github.com/ggerganov/llama.cpp/pull/711

I can understand these folks struggling with what mmap is actually doing. But this isn't a new discussion about the qualities of MMAP versus file based IO etc. Although, many of the comments stated are quite wrong.

Related Work on this problem: 1. https://www.mongodb.com/blog/post/getting-storage-engines-re... - talks about developments on MongoDB's backend to use mmap. 2. https://www.pdl.cmu.edu/PDL-FTP/Database/p13-crotty.pdf - Talks about some of the cons of mmap, some I think are not as prevalent due to the existence of low latency, high throughput storage devices. 3. https://www.cs.cit.tum.de/fileadmin/w00cfj/dis/_my_direct_up... - less relevant but related.

Re: Using mmap to make LLaMA load faster

#24

Post by super256 » Wed, Apr 05, 2023, 4:48 PM UTC

yieldcrv wrote: post #11

NickGerleman wrote: post #6

Worth pointing out, there has been quite a bit of contention around this change, both technical, and some accusations of plagiarism/miscrediting here. https://github.com/ggerganov/llama.cpp/pull/711

only thing this discussion has showed me is that more people need Computer Science degrees again like, wow, mmap and paging. really guys?

> only thing this discussion has showed me is that more people need Computer Science degrees again

You have too much faith in unis. Mine did not teach me about mmap at all.

Re: Using mmap to make LLaMA load faster

#25

Post by stephc_int13 » Wed, Apr 05, 2023, 4:53 PM UTC

yieldcrv wrote: post #11

NickGerleman wrote: post #6

Worth pointing out, there has been quite a bit of contention around this change, both technical, and some accusations of plagiarism/miscrediting here. https://github.com/ggerganov/llama.cpp/pull/711

only thing this discussion has showed me is that more people need Computer Science degrees again like, wow, mmap and paging. really guys?

I feel the same.

I maybe should not be surprised, given that we live in the era of Unity and Electron, but using mmap() to load large files should be not be seen as rocket science.

And this is basically available on almost any platform with a MMU and a kernel.

Re: Using mmap to make LLaMA load faster

#26

Post by bigodanktime » Wed, Apr 05, 2023, 4:54 PM UTC

londons_explore wrote: post #22

dekhn wrote: post #10

Earlier quoted context omitted.

I feel significantly dumber for reading that merge request. The one thing to understand is that the performance implications of mmap are subtle and only work when you have much more RAM than the files you're mapping in.

In this case, the main benefit is from multiple invocations of the same program. Using mmap, you avoid doing any work at all the 2nd time you load the file.

What do you mean by work. The underlying page cache will keep much of the data actual cached if it's recent. Even databases like PostGreSQL use this to their advantage (https://github.com/postgres/postgres/blob/master/src/backend...).

Re: Using mmap to make LLaMA load faster

#27

Post by dekhn » Wed, Apr 05, 2023, 4:55 PM UTC

londons_explore wrote: post #22

dekhn wrote: post #10

Earlier quoted context omitted.

I feel significantly dumber for reading that merge request. The one thing to understand is that the performance implications of mmap are subtle and only work when you have much more RAM than the files you're mapping in.

In this case, the main benefit is from multiple invocations of the same program. Using mmap, you avoid doing any work at all the 2nd time you load the file.

Yes- I have 35 years experience with UNIX and used to use mmapping with BLAST, a sequence search tool, as well as my own codes.

I'll repeat myself: mmap is subtle. If what you mmap is larger than your host RAM, only some of the pages will be loaded at any time, and depending on access patterns, can lead to significant paging.

Re: Using mmap to make LLaMA load faster

#28

Post by stephc_int13 » Wed, Apr 05, 2023, 4:57 PM UTC

candrewlee14 wrote: post #14

jart is a genius. What they’ve done with Blink, Cosmopolitan C, Redbean, and now llama.cpp is incredible. It gives me hope for the future of systems/low-level programming.

She is very good indeed, but this mmap thing is not comparable to her previous work.

Hopefully we should publicize this kind of achievement as a way to teach more devs about mmap... (this really should be common knowledge)

Re: Using mmap to make LLaMA load faster

#29

Post by bestcoder69 » Wed, Apr 05, 2023, 4:58 PM UTC

NickGerleman wrote: post #6

Worth pointing out, there has been quite a bit of contention around this change, both technical, and some accusations of plagiarism/miscrediting here. https://github.com/ggerganov/llama.cpp/pull/711

Also worth pointing out that you can follow the thread’s link to Rentry, which links to a 4chan (?) archived thread, where you can see anons getting worked up over jart being a trans internet celebrity. And unless you’re playing dumb, you have to admit they were looking for an excuse to troll jart. Unless you seriously want me to believe they were all that mad about… mmap

Re: Using mmap to make LLaMA load faster

#30

Post by yieldcrv » Wed, Apr 05, 2023, 4:59 PM UTC

stephc_int13 wrote: post #25

yieldcrv wrote: post #11

Earlier quoted context omitted.

only thing this discussion has showed me is that more people need Computer Science degrees again like, wow, mmap and paging. really guys?

I feel the same. I maybe should not be surprised, given that we live in the era of Unity and Electron, but using mmap() to load large files should be not be seen as rocket science. And this is basically available on almost any platform with a MMU and a kernel.

I think there just hasn't been a consumer application that is really resource constrained, for a long time now. Only things for enthusiasts have been. LLMs have product market fit, but running a useful one client side is resource constrained, but instead of it truly being a consumer hardware limitation, it just turns out they were never optimized to begin with - coming from the perceived "top AI/ML minds" at FAANGs, while some of the most basic optimizations are seemingly a lost art.

On the other hand, its only been a few weeks, so maybe I should ignore this absurdity and just wait.