51–60 of 186 posts
I'm not sure what this means but I'm pretty sure I can name several "high level libraries" that mmap things. None of those are the STL, but it's not exactly perfect design.
Is this AI people learning that mmap exists? People simping cos' there a Justine involved?
Earlier quoted context omitted.
only thing this discussion has showed me is that more people need Computer Science degrees again like, wow, mmap and paging. really guys?
> only thing this discussion has showed me is that more people need Computer Science degrees again You have too much faith in unis. Mine did not teach me about mmap at all.
I say this to highlight the parent comment. I'm essentially in a computer science program and we have learned absolutely 0 about paging or memory in any of my required courses. We practically don't touch OS anything in any of the classes. That's not to say the courses for that aren't offered but they aren't part of the core curriculum and over my time in my program, they've mostly not been offered due to lack of student interest.
I did learn how to use linked lists like a champion though!
Worth pointing out, there has been quite a bit of contention around this change, both technical, and some accusations of plagiarism/miscrediting here. https://github.com/ggerganov/llama.cpp/pull/711
[flagged]
> I don't think I've ever seen a high-level library that's able to do what mmap() does, because it defies attempts at abstraction. I'm not sure what this means but I'm pretty sure I can name several "high level libraries" that mmap things. None of those are the STL, but it's not exactly perfect design.
It's already extremely high level from a certain perspective :)
Earlier quoted context omitted.
Copying the file backed pages to heap memory and possibly having to swap them out.
I may have parsed your statement incorrectly, but I'm assuming you are talking about the copy of data when using either mmap or File IO (memcpy versus write) Whether you do File IO versus mmap, there's going to be copy. With files, the copy occurs within kernel space with data being copied into the pages in the buffer cache, with mmap the copy occurs in userspace with data being copied into the address space. Swappin…
There is no copy with mmap, the page is either unwritable or CoW. There's always a copy with read(). (But read() can still be faster and more memory efficient nevertheless.)
> An advantage of copying in userspace is the ability to use more performant instructions to perform the memcopy, which the kernel does not typically have access to (https://www.mongodb.com/blog/post/getting-storage-engines-re...)
Darwin kernel does though.
I believe Linux uses the builtin old memcpy instructions on Intel, just to force CPU vendors to keep them usable.
> I don't think I've ever seen a high-level library that's able to do what mmap() does, because it defies attempts at abstraction. I'm not sure what this means but I'm pretty sure I can name several "high level libraries" that mmap things. None of those are the STL, but it's not exactly perfect design.
I'm not sure what really needs abstracting about it other than tuning stuff for a given kernel/system. It's already extremely high level from a certain perspective :)
If you send it over IPC it's nice to keep it mmapped instead of accidentally copying it too.
Since the post is from day, so the improvements were all ‘real’? I didn’t follow closely but I remember multiple points people brought up earlier like: is the memory counting correct, why aren’t all the weights accessed for a query, whether quantisation is a problem etc. Were all these fixed?
There are no memory improvements, people were not measuring correct. The giant improvement is the load times after the first run(if you do not invalidate your caches). Quantization to 4 bit is a big gain, the loss appears to be minimal from benchmarks. So with quantization you gain the ability to try a bigger model, if you have the hardware to fit the biggest model then you can skip it but for most people we need to…
[1] The kernel doesn't necessarily load the whole thing into page cache at once and keep it around indefinitely. It might have been recognizing a sequential loading pattern before and basically discarding pages almost immediately, where as now it might be keeping them for much longer. Or it might now be essentially skipping loading the whole thing in at once and doing it page-by-page on demand, which could be more RAM-efficient but slower. To some extent, you can control these behaviors with madvise, mlock, MAP_LOCKED, MAP_POPULATE, as well as various sysctls. Also, if it had to page out before, the anonymous memory was "dirty" and thus had to be swapped (written out to disk) where as the mmap()ed bytes are "clean" and can simply be discarded and (if needed to be paged back in later) reread from the existing file unchanged.