Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Revert for jart’s llama.cpp MMAP miracles

github.com

Post reply on HN
Print view View original

31–40 of 96 posts

Re: Revert for jart’s llama.cpp MMAP miracles

#31

Post by tempaccount420 »

augusto-moura wrote: post #8
ukuina wrote: post #2

Can someone in the know describe what the hullaballoo is about? Seems like ego-driven optimization that breaks compatibility?

Is not ego-driven, the requirements for running any model dropped by more than half, you can know run the largest model, 30b, on domestic over the shelf computers. The change is very welcome in the community

There were no improvements to memory use, the earlier GBs were a measurement error. If you couldn't run a model before (or were swapping, so running very slow), then you still have the same problem. You will actually "swap" a lot more than before if you have barely enough memory, but this is fixable with the --mlock flag.

Edit: For everyone downvoting, please tell me what is wrong with my comment. I don't have a bone in this fight.

Re: Revert for jart’s llama.cpp MMAP miracles

#32

Post by fastball »

surteen wrote: post #25

ggerganov commented Apr 2, 2023 So this is pretty stupid - I just lost my Sunday trying to figure out how to salvage this stupid drama @anzz1 and @jart You are no longer welcome as collaborators to the project.

What is the purpose of re-posting a comment in the linked thread here verbatim?

Re: Revert for jart’s llama.cpp MMAP miracles

#33

Post by shakow »

d23 wrote: post #27
shakow wrote: post #5

I really dislike giving HN exposure to this kind of issue; it only brings us the forbidden pleasure of voyeurism while not helping the maintainers & contributors in the slightest – and can even crystallize conflicts while we eat popcorn. Let us let them take their time, wash their dirty laundry among themselves, and take the time they need to go forward on the project.

There's an opportunity for a wider discussion about how these situations should be handled. We can all learn and grow from that.

Such discussions are dime a dozen on any major GH repo, no need to follow the latest, most fashionable, scandal.

Re: Revert for jart’s llama.cpp MMAP miracles

#34

Post by blondin »

aww_dang wrote: post #22
shakow wrote: post #5

I really dislike giving HN exposure to this kind of issue; it only brings us the forbidden pleasure of voyeurism while not helping the maintainers & contributors in the slightest – and can even crystallize conflicts while we eat popcorn. Let us let them take their time, wash their dirty laundry among themselves, and take the time they need to go forward on the project.

Ideally, this will be resolved diplomatically by someone who knows how to handle these personalities. From that point we might learn from their example.

i vote for someone like dang to handle these open-source drama episodes.

i have limited knowledge on what is being discussed but i echo the "ego-driven" comment below. because it's Sunday. and the replies are coming in so fast that it's certain, they are not giving enough thought to the other side's argument. and already talking about forking, not accepting collaborators, etc.

Re: Revert for jart’s llama.cpp MMAP miracles

#35

Post by qwertox »

surteen wrote: post #25

ggerganov commented Apr 2, 2023 So this is pretty stupid - I just lost my Sunday trying to figure out how to salvage this stupid drama @anzz1 and @jart You are no longer welcome as collaborators to the project.

But you should quote it completely, else it's manipulation:

> You are no longer welcome as collaborators to the project. I know you really care about it and only doing it because you really want to make it better - I'm 100% sure about this. But in fact, you are doing the opposite. And if you fail to see this - I'm sorry

Re: Revert for jart’s llama.cpp MMAP miracles

#37

Post by vore »

AceJohnny2 wrote: post #12

> > > memory mapping means that the model will stay behind and eat your memory even after the process is closed > > I don't think I'm unterstanding this right: You're saying that memory will not be freed by the OS after the process terminates? > You're understanding it perfectly. The whole raison d'etre for mmap() is the ability to leave stuff in RAM (or swap, albeit if that happens it's completely detrimental to thi…

If you squint maybe you can argue that mmap will leave things in the page cache. But, you know, it doesn’t matter and not even munmap will save you there so I have no idea what they’re getting at.

Re: Revert for jart’s llama.cpp MMAP miracles

#38

Post by Aeolun »

cuuupid wrote: post #19

I don’t understand the controversy in this issue. It seems they could have saved a lot of time spent throwing shade back and forth by just implementing a feature flag. The argument against the feature flag is ultimately more egregious; it’s an experimental feature, breaks compatibility, decreases memory usage for a fair portion of the population while 10x’ing load speed for the rest so very YMMV for an optimization.…

In my experience, there will always be a population of developers/users that prefer for things to always stay the same (and therefore never break). Unfortunately that means never improving.

Re: Revert for jart’s llama.cpp MMAP miracles

#39

Post by Operyl »

fastball wrote: post #32
surteen wrote: post #25

ggerganov commented Apr 2, 2023 So this is pretty stupid - I just lost my Sunday trying to figure out how to salvage this stupid drama @anzz1 and @jart You are no longer welcome as collaborators to the project.

What is the purpose of re-posting a comment in the linked thread here verbatim?

Updated context for those who read the issue before it was posted, that wasn’t there during the time of the original submission.

Re: Revert for jart’s llama.cpp MMAP miracles

#40

Post by chpatrick »

blibble wrote: post #16
trsohmers wrote: post #7

Earlier quoted context omitted.

The tl;dr as I understand it is that jart had a misunderstanding of how what was actually happening and the benefits of the map optimization… the claims of actually being able to shrink the model size from 20GB > 6GB were just completely false, and while there was a model loading time improvement, actual memory required and used did not change. A number of people saw this and said that making a breaking change to the…

here mmap is being used for essentially lazy loading that's it

But then it's pretty much the same as using the old version and turning on swap, so I don't really see the point. As far as I understand the whole model needs to be read constantly so there's no benefit from the random access mmap provides.
Post reply on HN