Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Using mmap to make LLaMA load faster

justine.lol

Post reply on HN
Print view View original

141–150 of 186 posts

Re: Using mmap to make LLaMA load faster

#141

Post by pradn »

astrange wrote: post #51

> I don't think I've ever seen a high-level library that's able to do what mmap() does, because it defies attempts at abstraction. I'm not sure what this means but I'm pretty sure I can name several "high level libraries" that mmap things. None of those are the STL, but it's not exactly perfect design.

It might be the non-obvious problem with mmap : failures are surfaced through signals. So, a failure signal handler may run at any time. Your program needs to be resilient to this, and it's not trivial to do so. It's not a local change to one class.

Re: Using mmap to make LLaMA load faster

#142

Post by detrites »

antimatter15 wrote: post #76

Note that as a result of some llama drama associated with this change, @jart (author of this post) and @anzz1 are apparently no longer welcome as collaborators on llama.cpp: https://github.com/ggerganov/llama.cpp/pull/711#issuecomment...

Probably - if sadly - the right decision. Despite being a great feature and improvement, something simply "happened" to the project vibe and so the community, apparently as a direct response to that PR.

Similarly, this GH issue response to that occurrence, despite having made valid points in a reasoned manner, also held some of the same kind of "happen", in response - which is understandable, but not diffusive.

It ultimately doesn't matter who contributes, if someone truly believes in the project, they'll be just as happy to step away from it if they're affecting its momentum, even through no fault of their own or just a misunderstanding.

Momentum is important - and at an early stage like this, when vibe is building and community is forming, it can be very. I hope ggerganov continues to make these difficult decisions characteristic of clear leadership.

Re: Using mmap to make LLaMA load faster

#143

Post by leni536 »

astrange wrote: post #37
yieldcrv wrote: post #30

Earlier quoted context omitted.

I think there just hasn't been a consumer application that is really resource constrained, for a long time now. Only things for enthusiasts have been. LLMs have product market fit, but running a useful one client side is resource constrained, but instead of it truly being a consumer hardware limitation, it just turns out they were never optimized to begin with - coming from the perceived "top AI/ML minds" at FAANGs,…

> while some of the most basic optimizations are seemingly a lost art mmap isn't relevant to anyone except CPU-using programmers because other hardware doesn't have virtual memory paging. Firmware programmers don't care, GPU programmers don't care.

AFAIK CUDA offers unified memory which basically works with virtual address space and page faulting in data from main memory. There is also IOMMU in general.

Re: Using mmap to make LLaMA load faster

#144

Post by [deleted] »

dundarious wrote: post #64
astrange wrote: post #51

> I don't think I've ever seen a high-level library that's able to do what mmap() does, because it defies attempts at abstraction. I'm not sure what this means but I'm pretty sure I can name several "high level libraries" that mmap things. None of those are the STL, but it's not exactly perfect design.

I read it to mean mmap is irreplaceable. There is no other sophisticated dance of system calls or userspace trickery that can achieve what mmap can achieve. She's saying that everything up and down the stack, including high level libraries, do just call mmap, because there would be no DIY alternative with similar cost-benefit.

[deleted]

Re: Using mmap to make LLaMA load faster

#145

Post by marwis »

Dwedit wrote: post #113
stephc_int13 wrote: post #25

Earlier quoted context omitted.

I feel the same. I maybe should not be surprised, given that we live in the era of Unity and Electron, but using mmap() to load large files should be not be seen as rocket science. And this is basically available on almost any platform with a MMU and a kernel.

Using memory mapped files is not always the right answer. Memory mapped files have their disadvantages. The biggest disadvantage is that any disk read error (or yanking the USB drive) becomes an access violation exception (also known as a crash), just like you read from a bad pointer. You need to have robust exception handling, which is a taller order than just checking a return value. Another disadvantage is that ev…

Wouldn't huge pages and readahead make number of page faults and context switches potentially smaller than with read()?

Re: Using mmap to make LLaMA load faster

#146

Post by leni536 »

scottlamb wrote: post #42
secondcoming wrote: post #32

Isn't `madvise` useful if you want to use Huge Pages?

If you're hoping file-backed mmaps from any filesystem other than tmpfs/shmem will use huge pages, you will be disappointed. :-( See the following note at https://www.kernel.org/doc/html/latest/admin-guide/mm/transh... > Currently THP only works for anonymous memory mappings and tmpfs/shmem.

You don't actually need transparent huge pages, but maybe you can work with explicit huge pages.

Re: Using mmap to make LLaMA load faster

#147

Post by bobbylarrybobby »

version_five wrote: post #83
antimatter15 wrote: post #76

Note that as a result of some llama drama associated with this change, @jart (author of this post) and @anzz1 are apparently no longer welcome as collaborators on llama.cpp: https://github.com/ggerganov/llama.cpp/pull/711#issuecomment...

I can't really understand what's going on after a few minutes poking around (best I can tell an argument over attribution?), but I certainly sympathize with the project author having to deal with some stupid internet drama that takes away from a cool project. This seems to be one of the less-discussed challenges with popular open-source.

The issue is that mmap was unilaterally (or close to unilaterally) implemented and made the only way of loading files. Users do not have an option to continue to not use mmap.

Re: Using mmap to make LLaMA load faster

#148

Post by [deleted] »

jchw wrote: post #120
baffo32 wrote: post #105

Earlier quoted context omitted.

justine worked hard on sorting out these changes and was attacked and experienced character assassination via orchestrated trolling, it’s so sad

There's something extremely odd about the drama that occurs around justine in particular. It makes me feel there's some iceberg of things I don't know going on and I genuinely don't have the faintest clue what it is. I just know they make some cool software, like APE, cosmopolitan libc, and I believe landlock-make, which are inspirational projects to people who love clever yet practical hacks. As for these LLaMA chan…

[deleted]

Re: Using mmap to make LLaMA load faster

#149

Post by gmork13 »

jerf wrote: post #43
gmork13 wrote: post #36

This is from today apr 5 saying the mmap change loads twice as big models with x100 speed up - is this not a blatant lie? Wasn’t it discovered last week that loading larger models was an error in measurement and the speed up was from keeping things in memory after the first loading? Please do correct me if I’m wrong.

"Blatant lie" seems a bit strong. Running a large model for a second time in a row is a pretty common use case and that speedup strikes me as real in that common case. Attribution may have been wrong but the time saved is real.

So it can load twice as large models somehow?

Re: Using mmap to make LLaMA load faster

#150

Post by pantalaimon »

ncann wrote: post #126
version_five wrote: post #83

Earlier quoted context omitted.

I can't really understand what's going on after a few minutes poking around (best I can tell an argument over attribution?), but I certainly sympathize with the project author having to deal with some stupid internet drama that takes away from a cool project. This seems to be one of the less-discussed challenges with popular open-source.

I found this summary from that thread: https://rentry.org/Jarted

> @slaren made 7 commits in his fork, which @jart then squashed down into one

Good lord, it's terrible when the peanut gallery feels like they have to comment on development practice. Why would numbers of commits be a relevant metric in an Open Source project? Of course squashed commits are easier to handle during reabses and such, and when that work can be squashed to a single "initial mmap support" commit, then that's fine.

> @jart rewrote @slaren's code, which slaren wrote first

now this is just kindergarten level of arguments

Post reply on HN