171–180 of 186 posts
Since the post is from day, so the improvements were all ‘real’? I didn’t follow closely but I remember multiple points people brought up earlier like: is the memory counting correct, why aren’t all the weights accessed for a query, whether quantisation is a problem etc. Were all these fixed?
Earlier quoted context omitted.
Is there anything like that on Windows?
Yes, it's how JITs like Java work. Though I don't specifically know what the Windows API is.
Earlier quoted context omitted.
I found this summary from that thread: https://rentry.org/Jarted
> @slaren made 7 commits in his fork, which @jart then squashed down into one Good lord, it's terrible when the peanut gallery feels like they have to comment on development practice. Why would numbers of commits be a relevant metric in an Open Source project? Of course squashed commits are easier to handle during reabses and such, and when that work can be squashed to a single "initial mmap support" commit, then tha…
What I'd care more about is whether this issue had harmed the project in a technical way, which, unfortunately, it apparently has.
This is from today apr 5 saying the mmap change loads twice as big models with x100 speed up - is this not a blatant lie? Wasn’t it discovered last week that loading larger models was an error in measurement and the speed up was from keeping things in memory after the first loading? Please do correct me if I’m wrong.
Justine knows this and it is stated right there on the page: > The first time you load a model after rebooting your computer, it's still going to go slow, because it has to load the weights from disk. However each time it's loaded afterwards, it should be fast (at least until memory pressure causes your file cache to be evicted).
Earlier quoted context omitted.
Not sure if that's true any longer on recent kernels. There have been some changes with folios that I think enable huge pages in the page cache. $ cat /proc/ /smaps [...] 7efca62e3000-7efcaa13d000 r-xp 00ae3000 00:18 75354786 /usr/lib/libLLVM-15.so Size: 63848 kB KernelPageSize: 4 kB MMUPageSize: 4 kB Rss: 58420 kB Pss: 19213 kB Pss_Dirty: 0 kB Shared_Clean: 56372 kB Shared_Dirty: 0 kB Private_Clean: 2048 kB Private_…
Exciting if true! I see https://docs.kernel.org/filesystems/proc.html describes FilePmdMapped as "Page cache mapped into userspace with huge pages", consistent with what you are saying. I don't fully understand the distinction between that and FileHugePages: "Memory used for filesystem data (page cache) allocated with huge pages". I wouldn't think it'd be possible to map it into userspace as huge pages if the kernel…
I also see something about MADV_COLLAPSE that supposedly supports file-backed pages. [1]
Worth pointing out, there has been quite a bit of contention around this change, both technical, and some accusations of plagiarism/miscrediting here. https://github.com/ggerganov/llama.cpp/pull/711
only thing this discussion has showed me is that more people need Computer Science degrees again like, wow, mmap and paging. really guys?
Self respecting computer engineering curriculums will cover MMUs, page tables, TLBs, hardware interrupts, and page caches which once you know about mmap is fairly simple to understand.
The fundamentals really haven’t changed much in the past 40 years.
Earlier quoted context omitted.
> Darwin kernel does though. It does not. Compare the implementation of _bcopyout against _platform_memmove, you'll see the difference :)
Huh, maybe I was thinking of "you can use floating point in the kernel". That doesn't work in every kernel because they don't want to bother saving/restoring the extra registers.
Earlier quoted context omitted.
You don't actually need transparent huge pages, but maybe you can work with explicit huge pages.
Is there a working way to request explicit huge pages for a file-backed mmap on ext4/xfs/btrfs? I'm not aware of it.
I didn't find any documentation that would indicate that explicit huge pages didn't work with on-disk filesystems, but sure enough, it doesn't seem to work on ext4.
Earlier quoted context omitted.
Exciting if true! I see https://docs.kernel.org/filesystems/proc.html describes FilePmdMapped as "Page cache mapped into userspace with huge pages", consistent with what you are saying. I don't fully understand the distinction between that and FileHugePages: "Memory used for filesystem data (page cache) allocated with huge pages". I wouldn't think it'd be possible to map it into userspace as huge pages if the kernel…
Huh. I did a little digging through kernel source. There's been a CONFIG_READ_ONLY_THP_FOR_FS since 2019. It's still marked as experimental and isn't enabled on the precompiled kernel I'm using (with Ubuntu 22.10). Is that option set on your kernel, or is this something else? I also see something about MADV_COLLAPSE that supposedly supports file-backed pages. [1] [1] https://lwn.net/Articles/913363/
$ zgrep 'CONFIG_READ_ONLY_THP_FOR_FS' /proc/config.gz
CONFIG_READ_ONLY_THP_FOR_FS=y
kernel 6.2.8-arch1-1
> What filesystem type is this?
btrfs
> I don't fully understand the distinction between that and FileHugePages: "Memory used for filesystem data (page cache) allocated with huge pages".
Probably pages in the page cache that aren't mapped into a user process. Which is what happens when you read()
Earlier quoted context omitted.
Hmm, based on what you've quoted here and knowing nothing else but a few messages on AI Twitter I would invest in jart. This is BillG-style product skill -- there is a ton of work that goes into representing a piece of software as something important and valuable that people should buy into.
Jart is a pretty exceptional engineer, even if she wrote this patch single-handedly it would hardly be a footnote in her list of professional accomplishments. This is the author of Cosmopolitan libc, redbean and APE we're talking about, after all. That being said, it's important to attribute work properly. It can be easy to mix things up (eg. "my patch" is excusable) but repeatedly insisting authorship when you're no…
Is this? If she so easily misrepresented slarens work as hers in this case, what other work isn't actually attributable to jart?