Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Using mmap to make LLaMA load faster

justine.lol

Post reply on HN
Print view View original

71–80 of 186 posts

Re: Using mmap to make LLaMA load faster

#71

Post by simion314 »

scottlamb wrote: post #59
simion314 wrote: post #15

Earlier quoted context omitted.

There are no memory improvements, people were not measuring correct. The giant improvement is the load times after the first run(if you do not invalidate your caches). Quantization to 4 bit is a big gain, the loss appears to be minimal from benchmarks. So with quantization you gain the ability to try a bigger model, if you have the hardware to fit the biggest model then you can skip it but for most people we need to…

Unless the prior code was using O_DIRECT, the data was getting loaded into the kernel's page cache, and then the application was copying it into its own anonymous memory. Now the copy isn't happening. There are some subtleties involved [1] but it's not crazy to claim approximately half the RAM usage, even before bringing multiple processes into the picture. [1] The kernel doesn't necessarily load the whole thing into…

Thanks for the extra clarifications, but the claims were something impossible like a 23 Gb model only using 6Gb with this change. So maybe before this change it would have used a lot more of 23 Gb. I was referring to those miracle memory reductions, unfortunetly not possible, I would like to try 3 bit qunatizations when models and software will be ready(found none in my searches today)

Re: Using mmap to make LLaMA load faster

#72

Post by bigodanktime »

astrange wrote: post #57
bigodanktime wrote: post #47

Earlier quoted context omitted.

I may have parsed your statement incorrectly, but I'm assuming you are talking about the copy of data when using either mmap or File IO (memcpy versus write) Whether you do File IO versus mmap, there's going to be copy. With files, the copy occurs within kernel space with data being copied into the pages in the buffer cache, with mmap the copy occurs in userspace with data being copied into the address space. Swappin…

> With files, the copy occurs within kernel space with data being copied into the pages in the buffer cache, with mmap the copy occurs in userspace with data being copied into the address space. There is no copy with mmap, the page is either unwritable or CoW. There's always a copy with read(). (But read() can still be faster and more memory efficient nevertheless.) > An advantage of copying in userspace is the abili…

> There is no copy with mmap

You are right, if you are directly modifying the mmaped region. I always internally model my data as staging my changes to be synchronized to the mmaped region, so thats my mistake there.

> the page is either unwritable or CoW.

This is not universally true, or maybe I'm confused on this statement. MAP_SHARED exists, but maybe you are referencing a specific kernels' implementation on how they achieve coherence between file backed shared memory regions in two processes? Im not sure.

> Darwin kernel does though.

Sure we can always point to a kernel that has has implemented some feature or another, which is why I said typically you don't see it.

Re: Using mmap to make LLaMA load faster

#73

Post by 514723553 »

chad1n wrote: post #61
514723553 wrote: post #55

Earlier quoted context omitted.

If people were mad at Greg for changing or not changing a magic number--then they can just make a PR to fix it! That'd be so easy!

Greg didn't change it, it was changed in Jart's pull request. Also they can't just make a PR to fix it, because the models were already converted to that magic string that was changed for no reason.

Well if it wasn't Greg, then I feel like I need to ask--why the heck would you use he/him to refer to Justine? I'm glad you've changed your mind and referred to her with they/them above, though!

> they can't just make a PR to fix it, because the models were already converted to that magic string that was changed for no reason.

I _think_ the magic string was changed _because_ of versioning issues--it sounds like you're arguing that the magic version number _instead_ of the magic string should've been changed...but it sounds like Justine _was_ concerned with versioning, even if the versioning wasn't done in what you're saying is the best possible way. I just don't think that "one magic number was changed instead of another magic number" really warrants this level of animosity.

Re: Using mmap to make LLaMA load faster

#74

Post by candrewlee14 »

stephc_int13 wrote: post #28
candrewlee14 wrote: post #14

jart is a genius. What they’ve done with Blink, Cosmopolitan C, Redbean, and now llama.cpp is incredible. It gives me hope for the future of systems/low-level programming.

She is very good indeed, but this mmap thing is not comparable to her previous work. Hopefully we should publicize this kind of achievement as a way to teach more devs about mmap... (this really should be common knowledge)

Of course its simpler, I just wanted it to make sure the comment still related to her work on this :) In summation, what she’s done is awesome.

Re: Using mmap to make LLaMA load faster

#75

Post by simion314 »

astrange wrote: post #40
simion314 wrote: post #15

Earlier quoted context omitted.

There are no memory improvements, people were not measuring correct. The giant improvement is the load times after the first run(if you do not invalidate your caches). Quantization to 4 bit is a big gain, the loss appears to be minimal from benchmarks. So with quantization you gain the ability to try a bigger model, if you have the hardware to fit the biggest model then you can skip it but for most people we need to…

> There are no memory improvements, people were not measuring correct. Using filebacked pages instead of anonymous memory is a real improvement because it doesn't have to get swapped out if there's memory pressure. And this program probably isn't the only thing running on the machine.

I was referring that you would not gain any memory, there was no magic compression so you could use a bigger model on the same hardware. There were some wild claims made but it was some people meassuring memory usage wrong, but you are correct there might be some small memory improvements and soem speed improvements.

Re: Using mmap to make LLaMA load faster

#77

Post by HopenHeyHi »

oceanplexian wrote: post #68
HopenHeyHi wrote: post #16

Earlier quoted context omitted.

Unfortunately Justine has attracted a peculiar fanbase+haterbase. As their numbers swell the collective intelligence and technical understanding diminishes. So the discussions end up gravitating towards weird drama. I wish you wouldn't have linked this thread. Theres going to be a bunch of stupid comments here as well about how great/awful jart is.

I'm not a fan or a hater, I didn't even know who this person was until this thread. Does the change deserve a blog post or wild claims like "llama.cpp is 100x faster and uses half the memory!"? No. The original PR looks like a decent addition but the blog posts reads as incredibly narcissistic (i.e. lots of language like "We spent several weeks volunteering" and "our project") uh whatever. It also breaks a backwards…

Yes, exactly.

The claim that it uses half the memory was probably a honest mistake. The ensuing disappointment that it did not in fact halve memory usage and drama attracted trolls and white knights and is icky. The discussion around nmap I suppose is subtle and when emotion abounds can no longer be had. :/

Re: Using mmap to make LLaMA load faster

#78

Post by patrakov »

isoprophlex wrote: post #13
menzoic wrote: post #8

Earlier quoted context omitted.

Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality by the Team with members from UC Berkeley, CMU, Stanford, and UC San Diego https://vicuna.lmsys.org/

Are the Vicuna weights available for download, and are they llama.cpp compatible? I can't grok that by skimming the page...

Officially, only as deltas against LLaMa weights, and needing a complicated and resource-intensive conversion procedure. Unofficially, yes, a pre-converted llama.cpp compatible ggml file is available, but obviously I won't publish the link here to avoid violating the Y Combinator's terms of use.

Re: Using mmap to make LLaMA load faster

#79

Post by jjoonathan »

marginalia_nu wrote: post #49
dekhn wrote: post #10

Earlier quoted context omitted.

I feel significantly dumber for reading that merge request. The one thing to understand is that the performance implications of mmap are subtle and only work when you have much more RAM than the files you're mapping in.

> only work when you have much more RAM than the files you're mapping in. Really depends on what you're doing, like memory access patterns. I've definitely seen scenarios when mapping hundreds of gigabytes of data on dozens of gigabytes of ram where mmap has been an almost absurd performance boost over traditional I/O, both immediately but also asymptotically as all the most frequently accessed data ends up in cache…

> least accessed data is paged out

Aren't all the weights touched in every pass?

Post reply on HN