Using mmap to make LLaMA load faster • Hacker News Board

Re: Using mmap to make LLaMA load faster

#11

Post by yieldcrv » Wed, Apr 05, 2023, 4:30 PM UTC

NickGerleman wrote: post #6

Worth pointing out, there has been quite a bit of contention around this change, both technical, and some accusations of plagiarism/miscrediting here. https://github.com/ggerganov/llama.cpp/pull/711

only thing this discussion has showed me is that more people need Computer Science degrees again

like, wow, mmap and paging. really guys?

Re: Using mmap to make LLaMA load faster

#12

Post by a5huynh » Wed, Apr 05, 2023, 4:32 PM UTC

valine wrote: post #3

It’s hard to keep up with all developments around LLaMA. What’s the best RLHF alpaca like model you can download right now?

I recently found this list of models that works with llama.cpp: https://rentry.org/nur779 (with dl links, albeit given llama's licensing gray area, use at your own risk)

The latest so far would be Vicuna, whose weights were just recently release.

Re: Using mmap to make LLaMA load faster

#13

Post by isoprophlex » Wed, Apr 05, 2023, 4:32 PM UTC

menzoic wrote: post #8

valine wrote: post #3

It’s hard to keep up with all developments around LLaMA. What’s the best RLHF alpaca like model you can download right now?

Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality by the Team with members from UC Berkeley, CMU, Stanford, and UC San Diego https://vicuna.lmsys.org/

Are the Vicuna weights available for download, and are they llama.cpp compatible? I can't grok that by skimming the page...

Re: Using mmap to make LLaMA load faster

#14

Post by candrewlee14 » Wed, Apr 05, 2023, 4:34 PM UTC

jart is a genius. What they’ve done with Blink, Cosmopolitan C, Redbean, and now llama.cpp is incredible. It gives me hope for the future of systems/low-level programming.

Re: Using mmap to make LLaMA load faster

#15

Post by simion314 » Wed, Apr 05, 2023, 4:38 PM UTC

radicality wrote: post #4

Since the post is from day, so the improvements were all ‘real’? I didn’t follow closely but I remember multiple points people brought up earlier like: is the memory counting correct, why aren’t all the weights accessed for a query, whether quantisation is a problem etc. Were all these fixed?

There are no memory improvements, people were not measuring correct. The giant improvement is the load times after the first run(if you do not invalidate your caches). Quantization to 4 bit is a big gain, the loss appears to be minimal from benchmarks. So with quantization you gain the ability to try a bigger model, if you have the hardware to fit the biggest model then you can skip it but for most people we need to try to fit the biggest model possible in our VRAM or RAM.

Re: Using mmap to make LLaMA load faster

#16

Post by HopenHeyHi » Wed, Apr 05, 2023, 4:40 PM UTC

NickGerleman wrote: post #6

Worth pointing out, there has been quite a bit of contention around this change, both technical, and some accusations of plagiarism/miscrediting here. https://github.com/ggerganov/llama.cpp/pull/711

Unfortunately Justine has attracted a peculiar fanbase+haterbase. As their numbers swell the collective intelligence and technical understanding diminishes.

So the discussions end up gravitating towards weird drama. I wish you wouldn't have linked this thread. Theres going to be a bunch of stupid comments here as well about how great/awful jart is.

Re: Using mmap to make LLaMA load faster

#17

Post by danielbln » Wed, Apr 05, 2023, 4:43 PM UTC

isoprophlex wrote: post #13

menzoic wrote: post #8

Earlier quoted context omitted.

Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality by the Team with members from UC Berkeley, CMU, Stanford, and UC San Diego https://vicuna.lmsys.org/

Are the Vicuna weights available for download, and are they llama.cpp compatible? I can't grok that by skimming the page...

The weight deltas are available: https://github.com/lm-sys/FastChat#vicuna-weights

Re: Using mmap to make LLaMA load faster

#18

Post by [deleted] » Wed, Apr 05, 2023, 4:43 PM UTC

yieldcrv wrote: post #11

NickGerleman wrote: post #6

Worth pointing out, there has been quite a bit of contention around this change, both technical, and some accusations of plagiarism/miscrediting here. https://github.com/ggerganov/llama.cpp/pull/711

only thing this discussion has showed me is that more people need Computer Science degrees again like, wow, mmap and paging. really guys?

[deleted]

Re: Using mmap to make LLaMA load faster

#19

Post by TehCorwiz » Wed, Apr 05, 2023, 4:44 PM UTC

isoprophlex wrote: post #13

menzoic wrote: post #8

Earlier quoted context omitted.

Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality by the Team with members from UC Berkeley, CMU, Stanford, and UC San Diego https://vicuna.lmsys.org/

Are the Vicuna weights available for download, and are they llama.cpp compatible? I can't grok that by skimming the page...

The github page (https://github.com/lm-sys/FastChat#vicuna-weights) is better:

> We release Vicuna weights as delta weights to comply with the LLaMA model license. You can add our delta to the original LLaMA weights to obtain the Vicuna weights.

Re: Using mmap to make LLaMA load faster

#20

Post by popip » Wed, Apr 05, 2023, 4:44 PM UTC

HopenHeyHi wrote: post #16

NickGerleman wrote: post #6

Worth pointing out, there has been quite a bit of contention around this change, both technical, and some accusations of plagiarism/miscrediting here. https://github.com/ggerganov/llama.cpp/pull/711

Unfortunately Justine has attracted a peculiar fanbase+haterbase. As their numbers swell the collective intelligence and technical understanding diminishes. So the discussions end up gravitating towards weird drama. I wish you wouldn't have linked this thread. Theres going to be a bunch of stupid comments here as well about how great/awful jart is.

[flagged]