Live data from Hacker News

Revert for jart’s llama.cpp MMAP miracles

github.com

61–70 of 96 posts

Re: Revert for jart’s llama.cpp MMAP miracles

#61

This is someone angrily filing a revert-all PR due to a performance regression, rather than helping diagnose the issue or make it configurable. Don't bother reading. It sounds like one person experienced a performance regression as a result of the llama.cpp MMAP changes, and decided to create a pull request to revert all of those changes. While they propose wrapping the mmap changes behind a feature flag / command-li…

That’s not what’s happening. File format compatibility is broken while performance degrades by 10x for some people.

Re: Revert for jart’s llama.cpp MMAP miracles

#62
post #60
post #56

Earlier quoted context omitted.

Then he'd have to do a video call between him, myself, and 4chan.

huh? Of all things what does 4chan have to do with anything here?

See https://rentry.org/Jarted I'm not sure if it's 4chan but it appears to be some chan.

When I was working on solving the mmap() problem in https://github.com/ggerganov/llama.cpp/issues/91 (that issue tracks the full history) I originally wrote a malloc() hack to prove zero latency load times could work. But I thought we should hold off and wait until we could fix the file format before merging anything into master.

A week or so later, an anonymous GitHub user called Slarin pointed out something really smart. He said the 7B model only has 1D tensors, so we could bring the benefits of mmap() to 7B users right away. My file format change would have solved it for all models. But it wasn't ready yet. So when Slarin asked me to abandon my mmap branch, start over, and use his change as the starting point, I said sure. Since that would have ensured users of the project get some benefit as soon as possible. And we could iterate.

So Slarin and I collaborated on Discord. I had a really positive impression. As far as I knew, everything was fine. Slarin made a pull request, which he kept in draft mode. The work he was doing was basically to (1) call mmap() to load the full weights into memory, and (2) update the tensor->data instead of calling read() if the tensor was 1D. The issue is he Slarin got blocked on getting the WIN32 support to work. He had difficulties getting it to not crash. And by the time he was blocked, I had already finished rewriting the file format and data conversion tools. So I said, since I'm done with the full change, just push whatever you did to your PR branch, and I'll just rebase on that, I'll fix your WIN32 code, and then we'll merge our commits as part of the same PR. That PR was https://github.com/ggerganov/llama.cpp/pull/613 where I was very careful to recognize and document all of Slarin's contributions, which are now reflected in the master branch.

Slarin agreed, and told me "it looks like you've got this situation handled" and then left. We haven't seen him on the Discord since. I honestly had no idea he was so unhappy about the way things went. It really bums me out, because I have no way of ensuring people stay happy if they don't tell me they're unhappy. It appears, based on the link above, that Slarin felt I had put him in charge of the mmap() contribution and that the change belonged to him because his work was going to be its genesis once merged in the master branch. After the contribution went viral, he probably felt I didn't do enough to give him credit (I'm not exactly sure how to credit an anonymous developer more than I did?) and as a result he went to an online forum to ask a troll brigade to give justice.

The fallout of that "chan" troll brigade, is what you see here. They've also been going after my Wikipedia page among other things. I'm probably never going to work with an anonymous person again. I'm a public figure. I live with the sword of damocles hanging over my head. The power that gives anonymous people over me is really quite huge. The consequences here have damaged my reputation and career. And I don't even know who I'd reach out to if I wanted to mediate this. Quite a shame.

Re: Revert for jart’s llama.cpp MMAP miracles

#63
post #60
post #56

Earlier quoted context omitted.

Then he'd have to do a video call between him, myself, and 4chan.

huh? Of all things what does 4chan have to do with anything here?

That's where the brunt of the coordination, outreach, and testing around these models is happening.

Re: Revert for jart’s llama.cpp MMAP miracles

#64

Earlier quoted context omitted.

It's really bad reply from Greg. He's the owner of the project, he has the power to accept / not accept changes, and he didn't object to the version change, now he pushes responsibility to the contributors. It's ugly way of dealing with other people. The way to solve this situation is to set up a video call between them to deescalate the emotional part of the situation (which is not a big deal anyways, we can wait a…

Not super surprising. These projects have (understandably given the current environment) blown up in the last few weeks. The level of exposure / pace / etc is something very few people in general are probably equipped to deal with.

I agree. Also as a person who tried to contribute to open source projects, I understand the pressure Jart had to put all his changes into 1 big change instead of splitting it up to pieces: there's always a power asymmetry that favors the project maintainer by default as he's getting more famous (which happened to Greg).

If Jart would have tried to do everything the ,,proper way'', this change wouldn't have had this visibility, and he would have had much harder time to change the memory layout in the file.

I think things are going great, I just wish people would be able to not care about the negative feedback part that comes with it naturally.

Re: Revert for jart’s llama.cpp MMAP miracles

#65
post #11

This contributor doesn't appear to know how mmap works if they're claiming the only benefit is sharing data between processes (what? MAP_PRIVATE mappings aren't shared), and that memory is leaked after the process exits. There are a lot of thorny issues with mmap, and I'm sure there are legitimate regressions with the approach and things to be fixed, but it sure would be nice to see an analysis from someone who actua…

[deleted]

Re: Revert for jart’s llama.cpp MMAP miracles

#66
post #50

Earlier quoted context omitted.

Sadly I’m on my phone at the moment and can’t find the specific post, but in that PR or related discussion there was talk of only a few GB of the weights actually being used during the computation, which anyone who understands how a multi headed attention transformer works would know is impossible… your QKV matmuls need to touch all of the weights once you go through all the layers. Since that post yesterday getting…

> [which anyone who understands] how a multi headed attention transformer works [would know is impossible] How is that productive? The author already claimed in their changes that they aren't an ML expert and asked for advice. It could be written like multi-headed attention transformers frobnicate all the weights in their QKV matmuls. You have an opportunity to teach rather than admonish.

I appreciate your feedback, and I agree I could have better worded and taken it in a more constructive direction. Thank you, and I hope myself and others will try to have more productive discourse.

Re: Revert for jart’s llama.cpp MMAP miracles

#67
post #62
post #60

Earlier quoted context omitted.

huh? Of all things what does 4chan have to do with anything here?

See https://rentry.org/Jarted I'm not sure if it's 4chan but it appears to be some chan. When I was working on solving the mmap() problem in https://github.com/ggerganov/llama.cpp/issues/91 (that issue tracks the full history) I originally wrote a malloc() hack to prove zero latency load times could work. But I thought we should hold off and wait until we could fix the file format before merging anything into master.…

Thanks, that clears it up somewhat (especially the trollings and downvotes).

The cool thing is nobody cares about your wikipedia page, people care about your contributions.

I still think you should hop onto a video chat with Greg and be humble (I hope he can be more understanding and humble as well). File formats don't matter, that's just technical detail (although important) that will be resolved. People relations do matter.

Re: Revert for jart’s llama.cpp MMAP miracles

#68
post #62

Earlier quoted context omitted.

See https://rentry.org/Jarted I'm not sure if it's 4chan but it appears to be some chan. When I was working on solving the mmap() problem in https://github.com/ggerganov/llama.cpp/issues/91 (that issue tracks the full history) I originally wrote a malloc() hack to prove zero latency load times could work. But I thought we should hold off and wait until we could fix the file format before merging anything into master.…

Thanks, that clears it up somewhat (especially the trollings and downvotes). The cool thing is nobody cares about your wikipedia page, people care about your contributions. I still think you should hop onto a video chat with Greg and be humble (I hope he can be more understanding and humble as well). File formats don't matter, that's just technical detail (although important) that will be resolved. People relations d…

I tried getting him on video chat before this troll brigade even happened. He hasn't responded to any of my messages over the past few days.

Re: Revert for jart’s llama.cpp MMAP miracles

#69
post #68

Earlier quoted context omitted.

Thanks, that clears it up somewhat (especially the trollings and downvotes). The cool thing is nobody cares about your wikipedia page, people care about your contributions. I still think you should hop onto a video chat with Greg and be humble (I hope he can be more understanding and humble as well). File formats don't matter, that's just technical detail (although important) that will be resolved. People relations d…

I tried getting him on video chat before this troll brigade even happened. He hasn't responded to any of my messages over the past few days.

I see, I'm sorry about that.

Re: Revert for jart’s llama.cpp MMAP miracles

#70
post #68

Earlier quoted context omitted.

I tried getting him on video chat before this troll brigade even happened. He hasn't responded to any of my messages over the past few days.

I see, I'm sorry about that.

No reason to be sorry. GG has had more success than he can handle right now. It's a good problem to have. It takes quite a stressful toll the first time one of your community projects skyrockets into the big leagues. I'm sure things will be fine if we just wait a little bit for this to blow over.
Post reply on HN