Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Revert for jart’s llama.cpp MMAP miracles

github.com

Post reply on HN
Print view View original

21–30 of 96 posts

Re: Revert for jart’s llama.cpp MMAP miracles

#21

Post by shakow »

ftxbro wrote: post #17
shakow wrote: post #5

I really dislike giving HN exposure to this kind of issue; it only brings us the forbidden pleasure of voyeurism while not helping the maintainers & contributors in the slightest – and can even crystallize conflicts while we eat popcorn. Let us let them take their time, wash their dirty laundry among themselves, and take the time they need to go forward on the project.

[flagged]

Please tell me what people who would discover this thread on HN could constructively add to this conversation.

> Real discussion and transparency can involve multiple viewpoints that conflict with each other while each having their plusses and minuses.

Which is exactly what's currently happening between the concerned people in the GH issue.

Re: Revert for jart’s llama.cpp MMAP miracles

#22

Post by aww_dang »

shakow wrote: post #5

I really dislike giving HN exposure to this kind of issue; it only brings us the forbidden pleasure of voyeurism while not helping the maintainers & contributors in the slightest – and can even crystallize conflicts while we eat popcorn. Let us let them take their time, wash their dirty laundry among themselves, and take the time they need to go forward on the project.

Ideally, this will be resolved diplomatically by someone who knows how to handle these personalities. From that point we might learn from their example.

Re: Revert for jart’s llama.cpp MMAP miracles

#23

Post by simonw »

trsohmers wrote: post #7
ukuina wrote: post #2

Can someone in the know describe what the hullaballoo is about? Seems like ego-driven optimization that breaks compatibility?

The tl;dr as I understand it is that jart had a misunderstanding of how what was actually happening and the benefits of the map optimization… the claims of actually being able to shrink the model size from 20GB > 6GB were just completely false, and while there was a model loading time improvement, actual memory required and used did not change. A number of people saw this and said that making a breaking change to the…

The 20GB to 6GB confusion appears to have come from the title of the Hacker News post the other day: https://news.ycombinator.com/item?id=35393284

The PR it linked to said nothing of the sort: https://github.com/ggerganov/llama.cpp/pull/613

Re: Revert for jart’s llama.cpp MMAP miracles

#24

Post by [deleted] »

AceJohnny2 wrote: post #12

> > > memory mapping means that the model will stay behind and eat your memory even after the process is closed > > I don't think I'm unterstanding this right: You're saying that memory will not be freed by the OS after the process terminates? > You're understanding it perfectly. The whole raison d'etre for mmap() is the ability to leave stuff in RAM (or swap, albeit if that happens it's completely detrimental to thi…

[deleted]

Re: Revert for jart’s llama.cpp MMAP miracles

#26

Post by ftxbro »

shakow wrote: post #21
ftxbro wrote: post #17

Earlier quoted context omitted.

[flagged]

Please tell me what people who would discover this thread on HN could constructively add to this conversation. > Real discussion and transparency can involve multiple viewpoints that conflict with each other while each having their plusses and minuses. Which is exactly what's currently happening between the concerned people in the GH issue.

> Please tell me what people who would discover this thread on HN could constructively add to this conversation.

I mean is that really the bar for posting hacker news articles?

> > Real discussion and transparency can involve multiple viewpoints that conflict with each other while each having their plusses and minuses.

> Which is exactly what's currently happening between the concerned people in the GH issue.

Yeah that was my point.

Re: Revert for jart’s llama.cpp MMAP miracles

#27

Post by d23 »

shakow wrote: post #5

I really dislike giving HN exposure to this kind of issue; it only brings us the forbidden pleasure of voyeurism while not helping the maintainers & contributors in the slightest – and can even crystallize conflicts while we eat popcorn. Let us let them take their time, wash their dirty laundry among themselves, and take the time they need to go forward on the project.

There's an opportunity for a wider discussion about how these situations should be handled. We can all learn and grow from that.

Re: Revert for jart’s llama.cpp MMAP miracles

#29

Post by trsohmers »

simonw wrote: post #23
trsohmers wrote: post #7

Earlier quoted context omitted.

The tl;dr as I understand it is that jart had a misunderstanding of how what was actually happening and the benefits of the map optimization… the claims of actually being able to shrink the model size from 20GB > 6GB were just completely false, and while there was a model loading time improvement, actual memory required and used did not change. A number of people saw this and said that making a breaking change to the…

The 20GB to 6GB confusion appears to have come from the title of the Hacker News post the other day: https://news.ycombinator.com/item?id=35393284 The PR it linked to said nothing of the sort: https://github.com/ggerganov/llama.cpp/pull/613

Sadly I’m on my phone at the moment and can’t find the specific post, but in that PR or related discussion there was talk of only a few GB of the weights actually being used during the computation, which anyone who understands how a multi headed attention transformer works would know is impossible… your QKV matmuls need to touch all of the weights once you go through all the layers. Since that post yesterday getting 1200+ upvotes resulted in multiple conversations in my social circle that took that untrue statement as fact.

Re: Revert for jart’s llama.cpp MMAP miracles

#30

Post by d23 »

tempaccount420 wrote: post #20

Editorialized title. Please rename it to "Bring back the ggml model format and revert breaking mmap change (#613)" @dang

The only editorialized bit is "miracles," in my opinion. The current title gets closer to including the relevant context than does your updated title.
Post reply on HN