Earlier quoted context omitted.
jart is a woman, for what it's worth.
[flagged]
Did you make a new account just to do this?
Using mmap to make LLaMA load faster
111–120 of 186 posts
Re: Using mmap to make LLaMA load faster
#112I still don't understand: why was the magic number changed in addition to the file version number? Edit: can someone running llama.cpp ask it whether it thinks it's a good idea to concatenate a running list of vanity initials of important developers into a magic filetype constant?
It was a significant change, and Greg - the original author of llama.cpp was fine with it:
> Regarding the version comment - yes, the plan was to bump versions and no the magic. But I'm ok to change the magic to commemorate the significance of this update. In fact, maybe we can make this a thing and everybody who makes a significant contribution to the project will get their initials appended to the version. What do you think? smile
Re: Using mmap to make LLaMA load faster
#113Earlier quoted context omitted.
only thing this discussion has showed me is that more people need Computer Science degrees again like, wow, mmap and paging. really guys?
I feel the same. I maybe should not be surprised, given that we live in the era of Unity and Electron, but using mmap() to load large files should be not be seen as rocket science. And this is basically available on almost any platform with a MMU and a kernel.
Using memory mapped files is not always the right answer.
Memory mapped files have their disadvantages. The biggest disadvantage is that any disk read error (or yanking the USB drive) becomes an access violation exception (also known as a crash), just like you read from a bad pointer. You need to have robust exception handling, which is a taller order than just checking a return value.
Another disadvantage is that even when you have your pages mapped into memory, calling the page fault handler and getting your page has a cost of ~1200 CPU cycles on Windows just to do the UserKernel mode transition, plus the cost of actually performing the IO. "Just reading the file" skips many UserKernel mode transitions, so it's one per read call rather than one per page fault.
Re: Using mmap to make LLaMA load faster
#114Earlier quoted context omitted.
I may have parsed your statement incorrectly, but I'm assuming you are talking about the copy of data when using either mmap or File IO (memcpy versus write) Whether you do File IO versus mmap, there's going to be copy. With files, the copy occurs within kernel space with data being copied into the pages in the buffer cache, with mmap the copy occurs in userspace with data being copied into the address space. Swappin…
> With files, the copy occurs within kernel space with data being copied into the pages in the buffer cache, with mmap the copy occurs in userspace with data being copied into the address space. There is no copy with mmap, the page is either unwritable or CoW. There's always a copy with read(). (But read() can still be faster and more memory efficient nevertheless.) > An advantage of copying in userspace is the abili…
> Darwin kernel does though.
It does not. Compare the implementation of _bcopyout against _platform_memmove, you'll see the difference :)
Re: Using mmap to make LLaMA load faster
#115Earlier quoted context omitted.
Unfortunately Justine has attracted a peculiar fanbase+haterbase. As their numbers swell the collective intelligence and technical understanding diminishes. So the discussions end up gravitating towards weird drama. I wish you wouldn't have linked this thread. Theres going to be a bunch of stupid comments here as well about how great/awful jart is.
Is this related to her advocacy for neoreactionary politics or is it just a transphobia thing?
I mean, there's also the part where she's wrong a lot.
Re: Using mmap to make LLaMA load faster
#116There was a post a few months ago about a developer who's job it was to rework ML stuff into actual efficient code. This reminds me of that post because it seems that lots of ML stuff is just plain inefficient...in that they use way too many resources given the problem.
And they're arguing about mmap, something that's been around forever.
It reminds me of that speedup in a package manager because they were reading uncached byte-at-a-time off of disk. You need to explicitly turn buffered reads off...but why would you do that in the first place? Unbuffered reads are almost never a good idea, ever.
It makes me wonder what other weird sub-optimal stuff is lying underneath the resource behemoth that is ML.
Re: Using mmap to make LLaMA load faster
#117Worth pointing out, there has been quite a bit of contention around this change, both technical, and some accusations of plagiarism/miscrediting here. https://github.com/ggerganov/llama.cpp/pull/711
I don't get the "plagiarism/miscrediting" accusations. This was in the original PR ( https://github.com/ggerganov/llama.cpp/pull/613 ): > This PR was written in collaboration with @slaren. This PR is also rebased on PR #586 so please do not squash merge! Use either merge or rebase. jart made sure to that the other user got credit, in addition to making sure that their name was properly attributed in the commit log. G…
> > This PR was written in collaboration with @slaren. This PR is also rebased on PR #586 so please do not squash merge! Use either merge or rebase.
I read that PR (didn't click any links) and here on HN posted a "Great work" to jart. The reason I did that is precisely because those final lines in the PR came across as an upright acknowledgement that some people helped out. I also got the impression that jart was a co-owner of the project with all the "we"s that were thrown around.
If I was writing that PR, it would be something like "this PR consolidates slaren's mmap approach with additional work done for ... by myself". After hearing about the drama, actually reading slaren's PR, and reviewing jart's comments in issues and the PR and the hn show and tell, I am now convinced this is someone who wants to steal other people's thunder. Heck, even this front page article is yet another PR stunt. I suspect "faster fork of llama.cpp" posts will follow.
Giorgi Gerganov remains for me the hacker hero here as far as LLMs are concerned -- mmap is kiddie stuff to be frank, but anyone who gets whisper and llama to work on my laptop with a handful of files (many thanks to you sir) has my technical respect. And I think he has made the right call regarding the project.
Re: Using mmap to make LLaMA load faster
#118Earlier quoted context omitted.
Using mmap is basic systems programming knowledge and not "incredible" even though it's a useful contribution.
Glad to see we're slowly coming to the realization that wrangling NumPy and Pandas doesn't actually count as Software Engineering.
Let's not gatekeep what Software Engineering is or isn't.
Re: Using mmap to make LLaMA load faster
#119Earlier quoted context omitted.
Glad to see we're slowly coming to the realization that wrangling NumPy and Pandas doesn't actually count as Software Engineering.
it's as much engineering as is gluing mmap, recv and send... there are people who can do that and can't piece together a dataframe pipeline, no need to be passive agressive here
> "it's as much engineering as is gluing mmap, recv and send"
As you can see from comments up this thread (ref. various GitHub issues) - the people "gluing mmap" don't actually have a single clue. They can't properly measure memory consumption (they don't understand what the numbers they're seeing actually mean). They don't understand how paging, swapping or virtual memory work. They don't actually understand the concept of memory-mapped files, why they're there and how they work. They can't explain why their code behaves differently when using memory-mapped files.
Moments like this are here to remind you that there's actual knowledge and skill to building scalable and efficient software, and that hustling and copy-pasting StackOverflow examples will only get you so-far, as will "piecing together dataframe pipelines" in Python.
Re: Using mmap to make LLaMA load faster
#120Note that as a result of some llama drama associated with this change, @jart (author of this post) and @anzz1 are apparently no longer welcome as collaborators on llama.cpp: https://github.com/ggerganov/llama.cpp/pull/711#issuecomment...
justine worked hard on sorting out these changes and was attacked and experienced character assassination via orchestrated trolling, it’s so sad
There's something extremely odd about the drama that occurs around justine in particular. It makes me feel there's some iceberg of things I don't know going on and I genuinely don't have the faintest clue what it is. I just know they make some cool software, like APE, cosmopolitan libc, and I believe landlock-make, which are inspirational projects to people who love clever yet practical hacks.
As for these LLaMA changes, I ran it on my machine for fun, and it worked perfectly. I wound up re-converting my models, but it doesn't take terribly long to do so even for 65B. After that, generation starts nearly instantaneously, which is very impressive. I wouldn't be surprised if there are legitimate problems with the change. Obviously people who deleted their local copy of the original model to save disk space are probably displeased, and maybe it is a massive performance reduction in some cases.
I wish I understood, and yet I fear I don't really want to know at the same time.
edit: At least in this case, it seems like it's mostly drama around attribution and unnecessary changes. Kind of sad that an otherwise really useful code change wound up being marred by probably-avoidable drama, but such is life ¯\_(ツ)_/¯ Honestly, I don't have any input, I just hope everyone can resolve their gripes amicably in due time.