Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Revert for jart’s llama.cpp MMAP miracles

github.com

Post reply on HN
Print view View original

41–50 of 96 posts

Re: Revert for jart’s llama.cpp MMAP miracles

#41

Post by xiphias2 »

qwertox wrote: post #35
surteen wrote: post #25

ggerganov commented Apr 2, 2023 So this is pretty stupid - I just lost my Sunday trying to figure out how to salvage this stupid drama @anzz1 and @jart You are no longer welcome as collaborators to the project.

But you should quote it completely, else it's manipulation: > You are no longer welcome as collaborators to the project. I know you really care about it and only doing it because you really want to make it better - I'm 100% sure about this. But in fact, you are doing the opposite. And if you fail to see this - I'm sorry

It's really bad reply from Greg.

He's the owner of the project, he has the power to accept / not accept changes, and he didn't object to the version change, now he pushes responsibility to the contributors. It's ugly way of dealing with other people.

The way to solve this situation is to set up a video call between them to deescalate the emotional part of the situation (which is not a big deal anyways, we can wait a few days for the technical details to get resolved).

Re: Revert for jart’s llama.cpp MMAP miracles

#43

Post by michaelmrose »

ftxbro wrote: post #26
shakow wrote: post #21

Earlier quoted context omitted.

Please tell me what people who would discover this thread on HN could constructively add to this conversation. > Real discussion and transparency can involve multiple viewpoints that conflict with each other while each having their plusses and minuses. Which is exactly what's currently happening between the concerned people in the GH issue.

> Please tell me what people who would discover this thread on HN could constructively add to this conversation. I mean is that really the bar for posting hacker news articles? > > Real discussion and transparency can involve multiple viewpoints that conflict with each other while each having their plusses and minuses. > Which is exactly what's currently happening between the concerned people in the GH issue. Yeah th…

At least one reasonable bar is not effectively vandalizing other communities by airlifting in a bunch of uninvolved commenters into an already drama filled situation.

Re: Revert for jart’s llama.cpp MMAP miracles

#44

Post by sitkack »

chpatrick wrote: post #40
blibble wrote: post #16

Earlier quoted context omitted.

here mmap is being used for essentially lazy loading that's it

But then it's pretty much the same as using the old version and turning on swap, so I don't really see the point. As far as I understand the whole model needs to be read constantly so there's no benefit from the random access mmap provides.

Swap is a system level property not a program level property. They are similar, use similar mechanisms, but the experience that a user would see are very different.

Re: Revert for jart’s llama.cpp MMAP miracles

#47

Post by chpatrick »

sitkack wrote: post #44
chpatrick wrote: post #40

Earlier quoted context omitted.

But then it's pretty much the same as using the old version and turning on swap, so I don't really see the point. As far as I understand the whole model needs to be read constantly so there's no benefit from the random access mmap provides.

Swap is a system level property not a program level property. They are similar, use similar mechanisms, but the experience that a user would see are very different.

I'm not sure it would be that different but mmap has the benefit that it can swap directly to the model file on the disk instead of making a copy in swap space.

Re: Revert for jart’s llama.cpp MMAP miracles

#48

Post by michaelmrose »

So this matches a pattern of submissions that eventually is or should be flagged. Can we not post GitHub issues that represent inter-project drama? It's not the discussion here that is the problem its effectively bringing commenters FROM here to stir up drama there where tensions are already high.

Re: Revert for jart’s llama.cpp MMAP miracles

#50

Post by sitkack »

trsohmers wrote: post #29
simonw wrote: post #23

Earlier quoted context omitted.

The 20GB to 6GB confusion appears to have come from the title of the Hacker News post the other day: https://news.ycombinator.com/item?id=35393284 The PR it linked to said nothing of the sort: https://github.com/ggerganov/llama.cpp/pull/613

Sadly I’m on my phone at the moment and can’t find the specific post, but in that PR or related discussion there was talk of only a few GB of the weights actually being used during the computation, which anyone who understands how a multi headed attention transformer works would know is impossible… your QKV matmuls need to touch all of the weights once you go through all the layers. Since that post yesterday getting…

> [which anyone who understands] how a multi headed attention transformer works [would know is impossible]

How is that productive? The author already claimed in their changes that they aren't an ML expert and asked for advice.

It could be written like

multi-headed attention transformers frobnicate all the weights in their QKV matmuls.

You have an opportunity to teach rather than admonish.

Post reply on HN