Using mmap to make LLaMA load faster
justine.lol
Using mmap to make LLaMA load faster
1–10 of 186 posts
Re: Using mmap to make LLaMA load faster
#2related hacker news discussion https://news.ycombinator.com/item?id=35411909
Re: Using mmap to make LLaMA load faster
#3It’s hard to keep up with all developments around LLaMA. What’s the best RLHF alpaca like model you can download right now?
Re: Using mmap to make LLaMA load faster
#4Since the post is from day, so the improvements were all ‘real’?
I didn’t follow closely but I remember multiple points people brought up earlier like: is the memory counting correct, why aren’t all the weights accessed for a query, whether quantisation is a problem etc.
Were all these fixed?
Re: Using mmap to make LLaMA load faster
#5related hacker news discussion https://news.ycombinator.com/item?id=35411909
Should we be linking flagged items? That discussion was very ugly and unproductive.
Re: Using mmap to make LLaMA load faster
#6Worth pointing out, there has been quite a bit of contention around this change, both technical, and some accusations of plagiarism/miscrediting here. https://github.com/ggerganov/llama.cpp/pull/711
Re: Using mmap to make LLaMA load faster
#7It’s hard to keep up with all developments around LLaMA. What’s the best RLHF alpaca like model you can download right now?
Vicune
Re: Using mmap to make LLaMA load faster
#8It’s hard to keep up with all developments around LLaMA. What’s the best RLHF alpaca like model you can download right now?
Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality
by the Team with members from UC Berkeley, CMU, Stanford, and UC San Diego
Re: Using mmap to make LLaMA load faster
#9[flagged]
Re: Using mmap to make LLaMA load faster
#10Worth pointing out, there has been quite a bit of contention around this change, both technical, and some accusations of plagiarism/miscrediting here. https://github.com/ggerganov/llama.cpp/pull/711
I feel significantly dumber for reading that merge request.
The one thing to understand is that the performance implications of mmap are subtle and only work when you have much more RAM than the files you're mapping in.