Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Revert for jart’s llama.cpp MMAP miracles

github.com

Post reply on HN
Print view View original

71–80 of 96 posts

Re: Revert for jart’s llama.cpp MMAP miracles

#71

Post by detaro »

fastball wrote: post #58
4bpp wrote: post #52

Earlier quoted context omitted.

A project that has been generating a lot of buzz lately (CPU-based inference for Facebook's LLaMa model that works on commodity hardware) has attracted contributions from a tech/activist celebrity ( https://en.wikipedia.org/wiki/Justine_Tunney ). Their somewhat overly self-assured/-aggrandizing style (e.g. Github posts written in a tone like they run the place, changing the file format magic number to include their o…

Also worth mentioning that there is some level of controversy over how much of this work involving mmap should be attributed to jart vs slaren. Slaren originally authored a PR using mmap which some people are claiming was the much better implementation of the feature (including not needing to change the model format) and jart basically re-wrote it so that she could take credit for it.

> basically re-wrote it so that she could take credit for it.

compare the actual PR: https://github.com/ggerganov/llama.cpp/pull/613

You'll note that it includes the original commits from Slaren, explicitly mentions the collaboration with Slaren and explicitly requests to preserve these commits on merge.

Re: Revert for jart’s llama.cpp MMAP miracles

#72

Post by tremon »

AceJohnny2 wrote: post #12

> > > memory mapping means that the model will stay behind and eat your memory even after the process is closed > > I don't think I'm unterstanding this right: You're saying that memory will not be freed by the OS after the process terminates? > You're understanding it perfectly. The whole raison d'etre for mmap() is the ability to leave stuff in RAM (or swap, albeit if that happens it's completely detrimental to thi…

Rather than just pointing and laughing, your post would have more substance if you could correct the flawed assumptions.

Re: Revert for jart’s llama.cpp MMAP miracles

#73

Post by stevenhuang »

jart wrote: post #62
azeirah wrote: post #60

Earlier quoted context omitted.

huh? Of all things what does 4chan have to do with anything here?

See https://rentry.org/Jarted I'm not sure if it's 4chan but it appears to be some chan. When I was working on solving the mmap() problem in https://github.com/ggerganov/llama.cpp/issues/91 (that issue tracks the full history) I originally wrote a malloc() hack to prove zero latency load times could work. But I thought we should hold off and wait until we could fix the file format before merging anything into master.…

Hey Jart, I read that rentry and suspected there was more to it, so I'm relieved to see your input and the full picture here.

Such is the way with online drama. Though I think only working with known people might not be it, as the same could still happen (not being communicative on their part, silently brooding, then doing something regrettable). Perhaps it's less likely.

I would venture those in the field understand that unfortunate misunderstandings of this sort happen all the time and find nuance in the situation. Please don't let this stop you from contributing, your work is amazing. Cheers.

Re: Revert for jart’s llama.cpp MMAP miracles

#74

Post by chuddington »

jart wrote: post #70
xiphias2 wrote: post #69

Earlier quoted context omitted.

I see, I'm sorry about that.

No reason to be sorry. GG has had more success than he can handle right now. It's a good problem to have. It takes quite a stressful toll the first time one of your community projects skyrockets into the big leagues. I'm sure things will be fine if we just wait a little bit for this to blow over.

You're Wondering Now, But All Results Well :)

Re: Revert for jart’s llama.cpp MMAP miracles

#75

Post by qwertox »

jart wrote: post #62
azeirah wrote: post #60

Earlier quoted context omitted.

huh? Of all things what does 4chan have to do with anything here?

See https://rentry.org/Jarted I'm not sure if it's 4chan but it appears to be some chan. When I was working on solving the mmap() problem in https://github.com/ggerganov/llama.cpp/issues/91 (that issue tracks the full history) I originally wrote a malloc() hack to prove zero latency load times could work. But I thought we should hold off and wait until we could fix the file format before merging anything into master.…

Wow... I feel sorry for you.

Having an entire troll brigade going after you, after you acted with the best intentions... not nice. I hope you get over it, soon.

Re: Revert for jart’s llama.cpp MMAP miracles

#76

Post by AceJohnny2 »

jcranmer wrote: post #54
AceJohnny2 wrote: post #12

> > > memory mapping means that the model will stay behind and eat your memory even after the process is closed > > I don't think I'm unterstanding this right: You're saying that memory will not be freed by the OS after the process terminates? > You're understanding it perfectly. The whole raison d'etre for mmap() is the ability to leave stuff in RAM (or swap, albeit if that happens it's completely detrimental to thi…

The fundamental operation of mmap is to add new entries to the page table of a process, and the precise properties of those entries are heavily dependent on what the arguments to mmap are. When you mmap a regular file, you're essentially adding an entry to the page table that shares the data with the kernel's filesystem cache. I think he was trying to explain the implications of this fact, but doing so in an incredib…

> I think he was trying to explain the implications of this fact, but doing so in an incredibly garbled manner, and getting his conclusions wrong.

Yeah. I like using the expression "knows just enough to be dangerous" (usually applied in humility to myself), and this situation is such a perfect example. Someone who seems to know just enough about the advanced workings under mmap() to completely misunderstand the implications.

Re: Revert for jart’s llama.cpp MMAP miracles

#77

Post by xiphias2 »

tempaccount420 wrote: post #31
augusto-moura wrote: post #8

Earlier quoted context omitted.

Is not ego-driven, the requirements for running any model dropped by more than half, you can know run the largest model, 30b, on domestic over the shelf computers. The change is very welcome in the community

There were no improvements to memory use, the earlier GBs were a measurement error. If you couldn't run a model before (or were swapping, so running very slow), then you still have the same problem. You will actually "swap" a lot more than before if you have barely enough memory, but this is fixable with the --mlock flag. Edit: For everyone downvoting, please tell me what is wrong with my comment. I don't have a bone…

4chan came here to chime in a bit. You were downvoted by 10 year old children, good luck explaining to them how memory locking works :)

Re: Revert for jart’s llama.cpp MMAP miracles

#78

Post by [deleted] »

qwertox wrote: post #75
jart wrote: post #62

Earlier quoted context omitted.

See https://rentry.org/Jarted I'm not sure if it's 4chan but it appears to be some chan. When I was working on solving the mmap() problem in https://github.com/ggerganov/llama.cpp/issues/91 (that issue tracks the full history) I originally wrote a malloc() hack to prove zero latency load times could work. But I thought we should hold off and wait until we could fix the file format before merging anything into master.…

Wow... I feel sorry for you. Having an entire troll brigade going after you, after you acted with the best intentions... not nice. I hope you get over it, soon.

[deleted]

Re: Revert for jart’s llama.cpp MMAP miracles

#79

Post by turmeric_root »

jart wrote: post #62
azeirah wrote: post #60

Earlier quoted context omitted.

huh? Of all things what does 4chan have to do with anything here?

See https://rentry.org/Jarted I'm not sure if it's 4chan but it appears to be some chan. When I was working on solving the mmap() problem in https://github.com/ggerganov/llama.cpp/issues/91 (that issue tracks the full history) I originally wrote a malloc() hack to prove zero latency load times could work. But I thought we should hold off and wait until we could fix the file format before merging anything into master.…

ugh, that's so shitty. so many people in this space seem to be absurdly demanding and angry at devs, but one thing I've noticed is that every text AI project discord I've hung out in has this sleazy, obsessive 4chan /g/ vibe hiding somewhere in it.

Re: Revert for jart’s llama.cpp MMAP miracles

#80

Post by patrakov »

Jasper_ wrote: post #11

This contributor doesn't appear to know how mmap works if they're claiming the only benefit is sharing data between processes (what? MAP_PRIVATE mappings aren't shared), and that memory is leaked after the process exits. There are a lot of thorny issues with mmap, and I'm sure there are legitimate regressions with the approach and things to be fixed, but it sure would be nice to see an analysis from someone who actua…

The code is related to MAP_PRIVATE mappings of the same file that are not written to. Such mappings are effectively deduplicated and thus occupy the RAM once no matter how many processes map the file.
Post reply on HN