Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Revert for jart’s llama.cpp MMAP miracles

github.com

Post reply on HN
Print view View original

51–60 of 96 posts

Re: Revert for jart’s llama.cpp MMAP miracles

#51

Post by sitkack »

chpatrick wrote: post #47
sitkack wrote: post #44

Earlier quoted context omitted.

Swap is a system level property not a program level property. They are similar, use similar mechanisms, but the experience that a user would see are very different.

I'm not sure it would be that different but mmap has the benefit that it can swap directly to the model file on the disk instead of making a copy in swap space.

mmap is like tightly scoped, targeted swap. If the PoR is on disk, the OS is free to reclaim that memory for something else at anytime. It really is a beautiful hack, but if you turn on swap for the system in that way, it has to balance memory usage across all the running programs.

In this use of mmap, there is nothing to swap out as the source of truth is always on disk. During general swap usage, memory has to travel in both directions.

Re: Revert for jart’s llama.cpp MMAP miracles

#52

Post by 4bpp »

ukuina wrote: post #2

Can someone in the know describe what the hullaballoo is about? Seems like ego-driven optimization that breaks compatibility?

A project that has been generating a lot of buzz lately (CPU-based inference for Facebook's LLaMa model that works on commodity hardware) has attracted contributions from a tech/activist celebrity (https://en.wikipedia.org/wiki/Justine_Tunney). Their somewhat overly self-assured/-aggrandizing style (e.g. Github posts written in a tone like they run the place, changing the file format magic number to include their own initials along those of the project's originator) rubbed many people the wrong way, and a sweeping change they introduced may have resulted in performance regressions for several users (while also being hugely oversold: a fantastical and quickly disproven claim about significantly reduced memory usage sat at >1k upvotes on HN yesterday). Then another long-standing contributor made a PR just seeking to flat out revert the patch in question. The discussion quickly turned toxic, with a (thankfully) not-yet-quite-vocalised US culture war undercurrent and people bandwagoning based on their personal disposition towards the person at the core of it.

Re: Revert for jart’s llama.cpp MMAP miracles

#54

Post by jcranmer »

AceJohnny2 wrote: post #12

> > > memory mapping means that the model will stay behind and eat your memory even after the process is closed > > I don't think I'm unterstanding this right: You're saying that memory will not be freed by the OS after the process terminates? > You're understanding it perfectly. The whole raison d'etre for mmap() is the ability to leave stuff in RAM (or swap, albeit if that happens it's completely detrimental to thi…

The fundamental operation of mmap is to add new entries to the page table of a process, and the precise properties of those entries are heavily dependent on what the arguments to mmap are.

When you mmap a regular file, you're essentially adding an entry to the page table that shares the data with the kernel's filesystem cache. I think he was trying to explain the implications of this fact, but doing so in an incredibly garbled manner, and getting his conclusions wrong.

There are performance implications to using mmap (not always good, not always bad), but both sides of the discussion here immediately dug their heels in on their conclusion without anyone trying to do any analysis to see what the actual implications were, and why.

Re: Revert for jart’s llama.cpp MMAP miracles

#55

Post by _gabe_ »

This is the part of Open Source I really despise. It looks like the top contributors in this repository have contributed a few hundred lines of code (as opposed to the 20Kloc by the author). I understand that lines of code is not comparable to level of effort, but there is at least some level of correlation there.

The predominant attitude I have seen with my open source projects is entitlement and anger at decisions I have made, (whether that's because my license isn't MIT or because I don't want to use the latest and greatest features of language X, or because I use 2 spaces instead of 4). I just want to share my code, but some people make this unnecessarily difficult and want to cause drama where there never needed to be any.

Now, with that said, I have also met amazing people who have offered invaluable insights. These people have made contributions, to code and discussions, and on the other side provided amazing libraries and support. I really love Open Source, but there is a certain aspect of the community that can be downright hostile, and I hate that. I never understood why some developers feel the need to belittle others or to scoff at what other people want to share. I hope that if people know my name it's because I encouraged them and gave them help and/or praise for a cool project, and not because I was a dick and made them feel like crap for an inconsequential action.

Re: Revert for jart’s llama.cpp MMAP miracles

#56

Post by jart »

xiphias2 wrote: post #41
qwertox wrote: post #35

Earlier quoted context omitted.

But you should quote it completely, else it's manipulation: > You are no longer welcome as collaborators to the project. I know you really care about it and only doing it because you really want to make it better - I'm 100% sure about this. But in fact, you are doing the opposite. And if you fail to see this - I'm sorry

It's really bad reply from Greg. He's the owner of the project, he has the power to accept / not accept changes, and he didn't object to the version change, now he pushes responsibility to the contributors. It's ugly way of dealing with other people. The way to solve this situation is to set up a video call between them to deescalate the emotional part of the situation (which is not a big deal anyways, we can wait a…

Then he'd have to do a video call between him, myself, and 4chan.

Re: Revert for jart’s llama.cpp MMAP miracles

#57

Post by fastball »

xiphias2 wrote: post #41
qwertox wrote: post #35

Earlier quoted context omitted.

But you should quote it completely, else it's manipulation: > You are no longer welcome as collaborators to the project. I know you really care about it and only doing it because you really want to make it better - I'm 100% sure about this. But in fact, you are doing the opposite. And if you fail to see this - I'm sorry

It's really bad reply from Greg. He's the owner of the project, he has the power to accept / not accept changes, and he didn't object to the version change, now he pushes responsibility to the contributors. It's ugly way of dealing with other people. The way to solve this situation is to set up a video call between them to deescalate the emotional part of the situation (which is not a big deal anyways, we can wait a…

Not super surprising. These projects have (understandably given the current environment) blown up in the last few weeks. The level of exposure / pace / etc is something very few people in general are probably equipped to deal with.

Re: Revert for jart’s llama.cpp MMAP miracles

#58

Post by fastball »

4bpp wrote: post #52
ukuina wrote: post #2

Can someone in the know describe what the hullaballoo is about? Seems like ego-driven optimization that breaks compatibility?

A project that has been generating a lot of buzz lately (CPU-based inference for Facebook's LLaMa model that works on commodity hardware) has attracted contributions from a tech/activist celebrity ( https://en.wikipedia.org/wiki/Justine_Tunney ). Their somewhat overly self-assured/-aggrandizing style (e.g. Github posts written in a tone like they run the place, changing the file format magic number to include their o…

Also worth mentioning that there is some level of controversy over how much of this work involving mmap should be attributed to jart vs slaren. Slaren originally authored a PR using mmap which some people are claiming was the much better implementation of the feature (including not needing to change the model format) and jart basically re-wrote it so that she could take credit for it.

Re: Revert for jart’s llama.cpp MMAP miracles

#59

Post by fastball »

trsohmers wrote: post #29
simonw wrote: post #23

Earlier quoted context omitted.

The 20GB to 6GB confusion appears to have come from the title of the Hacker News post the other day: https://news.ycombinator.com/item?id=35393284 The PR it linked to said nothing of the sort: https://github.com/ggerganov/llama.cpp/pull/613

Sadly I’m on my phone at the moment and can’t find the specific post, but in that PR or related discussion there was talk of only a few GB of the weights actually being used during the computation, which anyone who understands how a multi headed attention transformer works would know is impossible… your QKV matmuls need to touch all of the weights once you go through all the layers. Since that post yesterday getting…

The comment you're looking for is here[1], where @jart does indeed seem to be echoing the claim that memory usage is reduced to surprising levels given the LLM architecture. Justine's original guess was that this is because the model is sparse (?) but in actuality it seems like memory usage is just being reported strangely by the system and that a model that takes up 20GB on disk still effectively needs 20GB of RAM in the end.

[1] https://github.com/ggerganov/llama.cpp/discussions/638#discu...

Re: Revert for jart’s llama.cpp MMAP miracles

#60

Post by azeirah »

jart wrote: post #56
xiphias2 wrote: post #41

Earlier quoted context omitted.

It's really bad reply from Greg. He's the owner of the project, he has the power to accept / not accept changes, and he didn't object to the version change, now he pushes responsibility to the contributors. It's ugly way of dealing with other people. The way to solve this situation is to set up a video call between them to deescalate the emotional part of the situation (which is not a big deal anyways, we can wait a…

Then he'd have to do a video call between him, myself, and 4chan.

huh? Of all things what does 4chan have to do with anything here?
Post reply on HN