Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Using mmap to make LLaMA load faster

justine.lol

Post reply on HN
Print view View original

151–160 of 186 posts

Re: Using mmap to make LLaMA load faster

#151

Post by IshKebab »

blitzkrieg3 wrote: post #48
gmork13 wrote: post #36

This is from today apr 5 saying the mmap change loads twice as big models with x100 speed up - is this not a blatant lie? Wasn’t it discovered last week that loading larger models was an error in measurement and the speed up was from keeping things in memory after the first loading? Please do correct me if I’m wrong.

mmap() will keep things in memory after first loading, but the page cache will _also_ keep things in memory after first loading. The difference is in order to re-use that you still need to read the file and store yourself (requiring 2x memory), instead of just doing a memory access. This has two consequences: * 2x memory. A 20G data set requires 40G (20 for page cache and 20 for LLaMA) * Things would be _even slower_…

Why would you need to keep the entire file in cache though?

Re: Using mmap to make LLaMA load faster

#152

Post by ingenieroariel »

The post is a technical article that describes a very cool systems engineering approach to a problem that is usually coded away in proprietary code from nvidia.

No wonder a lot of people see it as "mmap, nothing new". But this is not the case in a lot of libraries where the norm is to just budget for a lot of time moving things to/from gpu and just relying on someone else's code.

Instead of accumulating technical debt the owner of the repo decided to merge this, have a breaking change and move on. When there was some community backlash to the breaking changes there was a pull request trying to revert all changes instead working through the issues (it was a net win for several users but not all, some configurations with slower drives were better served by the older approach). There was an ugly back and forth and the repo owner decided to ban both the person who did the pull request and the author of this post.

This article brings the conversation back to the technical merits, the roadmap, credit the the original authors and tones down the ownership tone that may have pissed off some community members.

That pull request has now been closed by the owner of the repo. They are trying to move on and be productive, let's do the same.

Re: Using mmap to make LLaMA load faster

#153

Post by Karrot_Kream »

marginalia_nu wrote: post #49
dekhn wrote: post #10

Earlier quoted context omitted.

I feel significantly dumber for reading that merge request. The one thing to understand is that the performance implications of mmap are subtle and only work when you have much more RAM than the files you're mapping in.

> only work when you have much more RAM than the files you're mapping in. Really depends on what you're doing, like memory access patterns. I've definitely seen scenarios when mapping hundreds of gigabytes of data on dozens of gigabytes of ram where mmap has been an almost absurd performance boost over traditional I/O, both immediately but also asymptotically as all the most frequently accessed data ends up in cache…

Yeah and on top of that, different systems (software and hardware combos) are different, so I can see the performance of this depending on the implementation of mmap on the system and the implementation of caches and virtual memory on the architecture. When I've debugged stuff like this, it's either been for myself in which case I know what combo I'm running on or it's been for work where we know which combinations we target and we run regression tests to observe perf implications.

Re: Using mmap to make LLaMA load faster

#154

Post by lostmsu »

bobbylarrybobby wrote: post #147
version_five wrote: post #83

Earlier quoted context omitted.

I can't really understand what's going on after a few minutes poking around (best I can tell an argument over attribution?), but I certainly sympathize with the project author having to deal with some stupid internet drama that takes away from a cool project. This seems to be one of the less-discussed challenges with popular open-source.

The issue is that mmap was unilaterally (or close to unilaterally) implemented and made the only way of loading files. Users do not have an option to continue to not use mmap.

Is there a reason not to?

Re: Using mmap to make LLaMA load faster

#155

Post by smoldesu »

empathy_m wrote: post #130
IAmNotACellist wrote: post #84

Earlier quoted context omitted.

That's not the original PR. jart was working on a malloc() approach that didn't work and slaren wrote all the code actually doing mmap, which jart then rebased in a random new PR, changed to support an unnecessary version change, magic numbers, a conversion tool, and WIN32 support when that was already working in the draft PR. https://archive.ph/Uva8c This is the original PR: https://github.com/ggerganov/llama.cpp/pu…

Hmm, based on what you've quoted here and knowing nothing else but a few messages on AI Twitter I would invest in jart. This is BillG-style product skill -- there is a ton of work that goes into representing a piece of software as something important and valuable that people should buy into.

Jart is a pretty exceptional engineer, even if she wrote this patch single-handedly it would hardly be a footnote in her list of professional accomplishments. This is the author of Cosmopolitan libc, redbean and APE we're talking about, after all.

That being said, it's important to attribute work properly. It can be easy to mix things up (eg. "my patch" is excusable) but repeatedly insisting authorship when you're not the author of the change just seems disingenuous. I'm sure it was in good faith, but since they didn't address the issue or clear anything up, it's come to this.

Dramatic, and hardly the conclusion people wanted to the story of a free performance improvement. It's not entirely contrived though, and I think the maintainer handled this exceptionally well given the circumstances.

Re: Using mmap to make LLaMA load faster

#156

Post by lostmsu »

10000truths wrote: post #138
dundarious wrote: post #64

Earlier quoted context omitted.

I read it to mean mmap is irreplaceable. There is no other sophisticated dance of system calls or userspace trickery that can achieve what mmap can achieve. She's saying that everything up and down the stack, including high level libraries, do just call mmap, because there would be no DIY alternative with similar cost-benefit.

Except it’s not irreplaceable, at least on Linux. userfaultfd allows you to define custom page fault handling. With it, you can even do crazy things like “mmap” a remote resource by making HTTP range requests on a read fault.

Is there anything like that on Windows?

Re: Using mmap to make LLaMA load faster

#158

Post by astrange »

lostmsu wrote: post #156
10000truths wrote: post #138

Earlier quoted context omitted.

Except it’s not irreplaceable, at least on Linux. userfaultfd allows you to define custom page fault handling. With it, you can even do crazy things like “mmap” a remote resource by making HTTP range requests on a read fault.

Is there anything like that on Windows?

Yes, it's how JITs like Java work. Though I don't specifically know what the Windows API is.

Re: Using mmap to make LLaMA load faster

#159

Post by EGreg »

bestcoder69 wrote: post #139
ncann wrote: post #126

Earlier quoted context omitted.

I found this summary from that thread: https://rentry.org/Jarted

Click the 4chan links and you’ll see in their own words what this was really about (trans maintainer, channers worried about their waifu bot getting cucked.) Most seem to not know what they’re talking about- some admit to being retarded. Highly suspicious of this being tech related. Also note the stats on GH subscribers and stuff. This is a lolcow dossier… Events like this make me glad I don’t contribute OSS. I’ll ke…

I didn’t know Justine was trans. Her wikipedia article doesn’t mention it, and has another female middle name which suggests that her parents gave the names to her.

I am not much familiar with her work except the impressive Cosmopolitan / Redbean mentioned on HN in the past. But she seems to be quite a controversial figure that is for some weird technocracy and against democracy and leftists, despite being a leader in the zucotti park protests… in short, someone who is no stranger to drama and controversy, and actively courts it:

https://www.thedailybeast.com/articles/2014/08/01/occupying-...

Re: Using mmap to make LLaMA load faster

#160

Post by dang »

menzoic wrote: post #8
valine wrote: post #3

It’s hard to keep up with all developments around LLaMA. What’s the best RLHF alpaca like model you can download right now?

Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality by the Team with members from UC Berkeley, CMU, Stanford, and UC San Diego https://vicuna.lmsys.org/

Related:

Vicuna: An open-source chatbot impressing GPT-4 with 90% ChatGPT quality - https://news.ycombinator.com/item?id=35378683 - March 2023 (167 comments)

Post reply on HN