Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Using mmap to make LLaMA load faster

justine.lol

Post reply on HN
Print view View original

121–130 of 186 posts

Re: Using mmap to make LLaMA load faster

#121

Post by anshumankmr »

astrange wrote: post #38
candrewlee14 wrote: post #14

jart is a genius. What they’ve done with Blink, Cosmopolitan C, Redbean, and now llama.cpp is incredible. It gives me hope for the future of systems/low-level programming.

Using mmap is basic systems programming knowledge and not "incredible" even though it's a useful contribution.

Systems Programmer is itself a niche field, when compares to other fields of interest in Software Developement. So let's not trivialize anything, cause it can be amazing to someone, but not to another. It depends from person to person.

Re: Using mmap to make LLaMA load faster

#122

Post by jesse__ »

astrange wrote: post #38
candrewlee14 wrote: post #14

jart is a genius. What they’ve done with Blink, Cosmopolitan C, Redbean, and now llama.cpp is incredible. It gives me hope for the future of systems/low-level programming.

Using mmap is basic systems programming knowledge and not "incredible" even though it's a useful contribution.

I think reducing this to "using mmap is basic" is pretty unfair.

The trick she did overriding malloc & friends to validate that the optimization would be worth doing is, in my mind, one of the high-points of the paper. It's a very clever way of making a meaningful measurement, which was the keystone of the entire change.

I've never heard of, thought of, or used that trick, and the fact she had it in her arsenal to apply to this very specific situation is pretty impressive, to me at least.

Re: Using mmap to make LLaMA load faster

#124

Post by HopenHeyHi »

MuffinFlavored wrote: post #109
HopenHeyHi wrote: post #96

Earlier quoted context omitted.

No? I mean yes, probably will get downvotes from the usual crowd. There is as a matter of fact some of that explicitly going on in the GitHub issue, which derailed the conversation. Such are the woes of being an internet micro celebrity I guess. Truly we live in the stupidest timeline.

> There is as a matter of fact some of that explicitly going on in the GitHub issue Could you link an example so I can learn/retract my comment? I didn't think there was actually going on.

You can just read it (at least ctrl-f jart) before telling people what is/isn't appropriate. As I recall in short order people started arguing about jealousy/credit, the file format initials being changed in jarts honor, "I'm not technical and don't know what's going on but..", 'etc. Goofy stuff.

Re: Using mmap to make LLaMA load faster

#125

Post by Technetium »

ingenieroariel wrote: post #5
ftxbro wrote: post #2

related hacker news discussion https://news.ycombinator.com/item?id=35411909

Should we be linking flagged items? That discussion was very ugly and unproductive.

Usually you'd be very right, but it's contextually relevant here due to slaren and jart replying there.

Re: Using mmap to make LLaMA load faster

#126

Post by ncann »

version_five wrote: post #83
antimatter15 wrote: post #76

Note that as a result of some llama drama associated with this change, @jart (author of this post) and @anzz1 are apparently no longer welcome as collaborators on llama.cpp: https://github.com/ggerganov/llama.cpp/pull/711#issuecomment...

I can't really understand what's going on after a few minutes poking around (best I can tell an argument over attribution?), but I certainly sympathize with the project author having to deal with some stupid internet drama that takes away from a cool project. This seems to be one of the less-discussed challenges with popular open-source.

I found this summary from that thread:

https://rentry.org/Jarted

Re: Using mmap to make LLaMA load faster

#127

Post by microtonal »

yieldcrv wrote: post #30
stephc_int13 wrote: post #25

Earlier quoted context omitted.

I feel the same. I maybe should not be surprised, given that we live in the era of Unity and Electron, but using mmap() to load large files should be not be seen as rocket science. And this is basically available on almost any platform with a MMU and a kernel.

I think there just hasn't been a consumer application that is really resource constrained, for a long time now. Only things for enthusiasts have been. LLMs have product market fit, but running a useful one client side is resource constrained, but instead of it truly being a consumer hardware limitation, it just turns out they were never optimized to begin with - coming from the perceived "top AI/ML minds" at FAANGs,…

Word embeddings were big for their time (especially with subword embeddings like fastText). We mmaped word embeddings for similar reasons. But yeah, I was kinda surprised that one post about LLaMa.cpp mmap support talked about a 'fairly new technique'. mmap has been in a UNIX programmer's tool belt for literally decades.

Re: Using mmap to make LLaMA load faster

#128

Post by kccqzy »

dasl wrote: post #101

> One of the downsides of the Linux cp command, is copying a file larger than RAM will destroy every existing entry in the file cache. Under normal circumstances this is a good thing, since a least recently used strategy usually works. However it can be problematic if you're just organizing your files on a production system where you don't want to disrupt performance. As far as I know, no standard command line utilit…

I choose to use a modern file system and use cp --reflink=always.

Re: Using mmap to make LLaMA load faster

#129

Post by bob1029 »

manv1 wrote: post #116

There was a post a few months ago about a developer who's job it was to rework ML stuff into actual efficient code. This reminds me of that post because it seems that lots of ML stuff is just plain inefficient...in that they use way too many resources given the problem. And they're arguing about mmap, something that's been around forever. It reminds me of that speedup in a package manager because they were reading un…

> it seems that lots of ML stuff is just plain inefficient

This is just like any other technology. Use it wrong, you will get burned. Doesn't matter how shiny the container is.

"ML stuff" also (mostly?) includes purely statistical methods from the 90s that are deterministic and arguably the best way to solve a large variety of non-generative problems.

In fact, unless generation of arbitrary output is a major objective, it's likely you can solve whatever ML task on a workstation from 2010 that uses intel integrated graphics.

Re: Using mmap to make LLaMA load faster

#130

Post by empathy_m »

IAmNotACellist wrote: post #84
514723553 wrote: post #44

Earlier quoted context omitted.

I don't get the "plagiarism/miscrediting" accusations. This was in the original PR ( https://github.com/ggerganov/llama.cpp/pull/613 ): > This PR was written in collaboration with @slaren. This PR is also rebased on PR #586 so please do not squash merge! Use either merge or rebase. jart made sure to that the other user got credit, in addition to making sure that their name was properly attributed in the commit log. G…

That's not the original PR. jart was working on a malloc() approach that didn't work and slaren wrote all the code actually doing mmap, which jart then rebased in a random new PR, changed to support an unnecessary version change, magic numbers, a conversion tool, and WIN32 support when that was already working in the draft PR. https://archive.ph/Uva8c This is the original PR: https://github.com/ggerganov/llama.cpp/pu…

Hmm, based on what you've quoted here and knowing nothing else but a few messages on AI Twitter I would invest in jart.

This is BillG-style product skill -- there is a ton of work that goes into representing a piece of software as something important and valuable that people should buy into.

Post reply on HN