Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Using mmap to make LLaMA load faster

justine.lol

Post reply on HN
Print view View original

131–140 of 186 posts

Re: Using mmap to make LLaMA load faster

#131

Post by the8472 »

scottlamb wrote: post #42
secondcoming wrote: post #32

Isn't `madvise` useful if you want to use Huge Pages?

If you're hoping file-backed mmaps from any filesystem other than tmpfs/shmem will use huge pages, you will be disappointed. :-( See the following note at https://www.kernel.org/doc/html/latest/admin-guide/mm/transh... > Currently THP only works for anonymous memory mappings and tmpfs/shmem.

Not sure if that's true any longer on recent kernels. There have been some changes with folios that I think enable huge pages in the page cache.

  $ cat /proc//smaps
  [...]
  7efca62e3000-7efcaa13d000 r-xp 00ae3000 00:18 75354786                   /usr/lib/libLLVM-15.so
  Size:              63848 kB
  KernelPageSize:        4 kB
  MMUPageSize:           4 kB
  Rss:               58420 kB
  Pss:               19213 kB
  Pss_Dirty:             0 kB
  Shared_Clean:      56372 kB
  Shared_Dirty:          0 kB
  Private_Clean:      2048 kB
  Private_Dirty:         0 kB
  Referenced:        58420 kB
  Anonymous:             0 kB
  LazyFree:              0 kB
  AnonHugePages:         0 kB
  ShmemPmdMapped:        0 kB
  FilePmdMapped:     57344 kB

Re: Using mmap to make LLaMA load faster

#132

Post by astrange »

simion314 wrote: post #75
astrange wrote: post #40

Earlier quoted context omitted.

> There are no memory improvements, people were not measuring correct. Using filebacked pages instead of anonymous memory is a real improvement because it doesn't have to get swapped out if there's memory pressure. And this program probably isn't the only thing running on the machine.

I was referring that you would not gain any memory, there was no magic compression so you could use a bigger model on the same hardware. There were some wild claims made but it was some people meassuring memory usage wrong, but you are correct there might be some small memory improvements and soem speed improvements.

Well, you can use a bigger model now, it will "just" be really slow. This is different from GPUs, which would just fail to load larger models than VRAM because they don't support paging (unless you build that yourself.)

Re: Using mmap to make LLaMA load faster

#133

Post by stuaxo »

The original change made intuitive sense, some of the arguments against seem a bit weird - asserting that MMAPing the file could mean the memory sticks around after the program stops.. no.

Suggesting that MMAP limits things to the size of the ram, well - no, as well - paging may happen, but then we are just back out to the file.

Honestly, some of the weird assertions wouldn't take long for people to double check an verify (or falsify).

Re: Using mmap to make LLaMA load faster

#134

Post by chad1n »

yieldcrv wrote: post #30
stephc_int13 wrote: post #25

Earlier quoted context omitted.

I feel the same. I maybe should not be surprised, given that we live in the era of Unity and Electron, but using mmap() to load large files should be not be seen as rocket science. And this is basically available on almost any platform with a MMU and a kernel.

I think there just hasn't been a consumer application that is really resource constrained, for a long time now. Only things for enthusiasts have been. LLMs have product market fit, but running a useful one client side is resource constrained, but instead of it truly being a consumer hardware limitation, it just turns out they were never optimized to begin with - coming from the perceived "top AI/ML minds" at FAANGs,…

Why would Facebook care about running LLAMA on a cpu with optimizing for 1-2% more latency when it has a lot of A100s laying around?

Re: Using mmap to make LLaMA load faster

#135

Post by astrange »

saagarjha wrote: post #114
astrange wrote: post #57

Earlier quoted context omitted.

> With files, the copy occurs within kernel space with data being copied into the pages in the buffer cache, with mmap the copy occurs in userspace with data being copied into the address space. There is no copy with mmap, the page is either unwritable or CoW. There's always a copy with read(). (But read() can still be faster and more memory efficient nevertheless.) > An advantage of copying in userspace is the abili…

> Darwin kernel does though. It does not. Compare the implementation of _bcopyout against _platform_memmove, you'll see the difference :)

Huh, maybe I was thinking of "you can use floating point in the kernel".

That doesn't work in every kernel because they don't want to bother saving/restoring the extra registers.

Re: Using mmap to make LLaMA load faster

#136

Post by IshKebab »

antimatter15 wrote: post #76

Note that as a result of some llama drama associated with this change, @jart (author of this post) and @anzz1 are apparently no longer welcome as collaborators on llama.cpp: https://github.com/ggerganov/llama.cpp/pull/711#issuecomment...

[flagged]

Re: Using mmap to make LLaMA load faster

#137

Post by Operyl »

ncann wrote: post #126
version_five wrote: post #83

Earlier quoted context omitted.

I can't really understand what's going on after a few minutes poking around (best I can tell an argument over attribution?), but I certainly sympathize with the project author having to deal with some stupid internet drama that takes away from a cool project. This seems to be one of the less-discussed challenges with popular open-source.

I found this summary from that thread: https://rentry.org/Jarted

Which Jart contests to some degree: https://news.ycombinator.com/item?id=35413289

Re: Using mmap to make LLaMA load faster

#138

Post by 10000truths »

dundarious wrote: post #64
astrange wrote: post #51

> I don't think I've ever seen a high-level library that's able to do what mmap() does, because it defies attempts at abstraction. I'm not sure what this means but I'm pretty sure I can name several "high level libraries" that mmap things. None of those are the STL, but it's not exactly perfect design.

I read it to mean mmap is irreplaceable. There is no other sophisticated dance of system calls or userspace trickery that can achieve what mmap can achieve. She's saying that everything up and down the stack, including high level libraries, do just call mmap, because there would be no DIY alternative with similar cost-benefit.

Except it’s not irreplaceable, at least on Linux. userfaultfd allows you to define custom page fault handling. With it, you can even do crazy things like “mmap” a remote resource by making HTTP range requests on a read fault.

Re: Using mmap to make LLaMA load faster

#139

Post by bestcoder69 »

ncann wrote: post #126
version_five wrote: post #83

Earlier quoted context omitted.

I can't really understand what's going on after a few minutes poking around (best I can tell an argument over attribution?), but I certainly sympathize with the project author having to deal with some stupid internet drama that takes away from a cool project. This seems to be one of the less-discussed challenges with popular open-source.

I found this summary from that thread: https://rentry.org/Jarted

Click the 4chan links and you’ll see in their own words what this was really about (trans maintainer, channers worried about their waifu bot getting cucked.) Most seem to not know what they’re talking about- some admit to being retarded. Highly suspicious of this being tech related.

Also note the stats on GH subscribers and stuff. This is a lolcow dossier…

Events like this make me glad I don’t contribute OSS. I’ll keep my coombots proprietary.

Re: Using mmap to make LLaMA load faster

#140

Post by chad1n »

eternalban wrote: post #117
514723553 wrote: post #44

Earlier quoted context omitted.

I don't get the "plagiarism/miscrediting" accusations. This was in the original PR ( https://github.com/ggerganov/llama.cpp/pull/613 ): > This PR was written in collaboration with @slaren. This PR is also rebased on PR #586 so please do not squash merge! Use either merge or rebase. jart made sure to that the other user got credit, in addition to making sure that their name was properly attributed in the commit log. G…

> > This PR was written in collaboration with @slaren. This PR is also rebased on PR #586 so please do not squash merge! Use either merge or rebase. I read that PR (didn't click any links) and here on HN posted a "Great work" to jart. The reason I did that is precisely because those final lines in the PR came across as an upright acknowledgement that some people helped out. I also got the impression that jart was a c…

I think that Georgi regrets making the project so openly to PR, he was probably happier with running it on his own.
Post reply on HN