Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Using mmap to make LLaMA load faster

justine.lol

Post reply on HN
Print view View original

161–170 of 186 posts

Re: Using mmap to make LLaMA load faster

#161

Post by bestcoder69 »

EGreg wrote: post #159
bestcoder69 wrote: post #139

Earlier quoted context omitted.

Click the 4chan links and you’ll see in their own words what this was really about (trans maintainer, channers worried about their waifu bot getting cucked.) Most seem to not know what they’re talking about- some admit to being retarded. Highly suspicious of this being tech related. Also note the stats on GH subscribers and stuff. This is a lolcow dossier… Events like this make me glad I don’t contribute OSS. I’ll ke…

I didn’t know Justine was trans. Her wikipedia article doesn’t mention it, and has another female middle name which suggests that her parents gave the names to her. I am not much familiar with her work except the impressive Cosmopolitan / Redbean mentioned on HN in the past. But she seems to be quite a controversial figure that is for some weird technocracy and against democracy and leftists, despite being a leader i…

All true, stuff I didn’t like about her, but also ancient history (article is from 2014!)

And like I said, you can follow the links and see the brigade discussing something else entirely.

Personally I don’t want an OSS ecosystem that banishes trans people or people who had weird proto-alt-right politics pre-Trump. If you’re gonna banish anyone, banish the ones posing existential risks to projects by their brigading against contributors they don’t like.

Re: Using mmap to make LLaMA load faster

#162

Post by dundarious »

10000truths wrote: post #138
dundarious wrote: post #64

Earlier quoted context omitted.

I read it to mean mmap is irreplaceable. There is no other sophisticated dance of system calls or userspace trickery that can achieve what mmap can achieve. She's saying that everything up and down the stack, including high level libraries, do just call mmap, because there would be no DIY alternative with similar cost-benefit.

Except it’s not irreplaceable, at least on Linux. userfaultfd allows you to define custom page fault handling. With it, you can even do crazy things like “mmap” a remote resource by making HTTP range requests on a read fault.

You’re right. Irreplaceable was a stronger way of putting it than the original, I think, so that’s more my mistake than hers, and I think the contrast with userspace stands.

mmap sits at this lovely intersection between virtual memory and the disk, and it’s been around for a long time. By now there are other means of playing within that nice intersection, but mmap is the pop classic.

Re: Using mmap to make LLaMA load faster

#163

Post by scottlamb »

the8472 wrote: post #131
scottlamb wrote: post #42

Earlier quoted context omitted.

If you're hoping file-backed mmaps from any filesystem other than tmpfs/shmem will use huge pages, you will be disappointed. :-( See the following note at https://www.kernel.org/doc/html/latest/admin-guide/mm/transh... > Currently THP only works for anonymous memory mappings and tmpfs/shmem.

Not sure if that's true any longer on recent kernels. There have been some changes with folios that I think enable huge pages in the page cache. $ cat /proc/ /smaps [...] 7efca62e3000-7efcaa13d000 r-xp 00ae3000 00:18 75354786 /usr/lib/libLLVM-15.so Size: 63848 kB KernelPageSize: 4 kB MMUPageSize: 4 kB Rss: 58420 kB Pss: 19213 kB Pss_Dirty: 0 kB Shared_Clean: 56372 kB Shared_Dirty: 0 kB Private_Clean: 2048 kB Private_…

Exciting if true!

I see https://docs.kernel.org/filesystems/proc.html describes FilePmdMapped as "Page cache mapped into userspace with huge pages", consistent with what you are saying. I don't fully understand the distinction between that and FileHugePages: "Memory used for filesystem data (page cache) allocated with huge pages". I wouldn't think it'd be possible to map it into userspace as huge pages if the kernel hasn't allocated it as contiguous physical memory (and consistently aligned with the userspace virtual addresses), so there's something I'm missing.

What kernel version did that output come from? Do you happen to know if Firefox did anything special to set that up? What filesystem type is this?

Re: Using mmap to make LLaMA load faster

#164

Post by scottlamb »

leni536 wrote: post #146
scottlamb wrote: post #42

Earlier quoted context omitted.

If you're hoping file-backed mmaps from any filesystem other than tmpfs/shmem will use huge pages, you will be disappointed. :-( See the following note at https://www.kernel.org/doc/html/latest/admin-guide/mm/transh... > Currently THP only works for anonymous memory mappings and tmpfs/shmem.

You don't actually need transparent huge pages, but maybe you can work with explicit huge pages.

Is there a working way to request explicit huge pages for a file-backed mmap on ext4/xfs/btrfs? I'm not aware of it.

Re: Using mmap to make LLaMA load faster

#165

Post by dekhn »

Dwedit wrote: post #113
stephc_int13 wrote: post #25

Earlier quoted context omitted.

I feel the same. I maybe should not be surprised, given that we live in the era of Unity and Electron, but using mmap() to load large files should be not be seen as rocket science. And this is basically available on almost any platform with a MMU and a kernel.

Using memory mapped files is not always the right answer. Memory mapped files have their disadvantages. The biggest disadvantage is that any disk read error (or yanking the USB drive) becomes an access violation exception (also known as a crash), just like you read from a bad pointer. You need to have robust exception handling, which is a taller order than just checking a return value. Another disadvantage is that ev…

although it's true that many hardware problems exhibit as SIGBUS on memmapped memory, remember that this is an API and implementation written for high performance disk drives on important servers; for example, the ingres server on berkeley's research vax (IIRC mmap became used widely after one of the BSD 4.3 subreleases was released). IE, at the time, the idea of a drive that could be easily detached being used for production computing would have been crazy so I think crashing the app when a drive is removed is not completely insensible.

Re: Using mmap to make LLaMA load faster

#166

Post by dekhn »

leni536 wrote: post #143
astrange wrote: post #37

Earlier quoted context omitted.

> while some of the most basic optimizations are seemingly a lost art mmap isn't relevant to anyone except CPU-using programmers because other hardware doesn't have virtual memory paging. Firmware programmers don't care, GPU programmers don't care.

AFAIK CUDA offers unified memory which basically works with virtual address space and page faulting in data from main memory. There is also IOMMU in general.

Many of us would like to get rid of the host CPU and have ML trainers that are just GPUs and drives and NICs all attached to a northbridge. The GPU has everything required to make disk requests over the bus, and ideally the drive can receive network messages that get plumbed straight to the drive (I'm only partially joking).

Re: Using mmap to make LLaMA load faster

#168

Post by EGreg »

bestcoder69 wrote: post #161
EGreg wrote: post #159

Earlier quoted context omitted.

I didn’t know Justine was trans. Her wikipedia article doesn’t mention it, and has another female middle name which suggests that her parents gave the names to her. I am not much familiar with her work except the impressive Cosmopolitan / Redbean mentioned on HN in the past. But she seems to be quite a controversial figure that is for some weird technocracy and against democracy and leftists, despite being a leader i…

All true, stuff I didn’t like about her, but also ancient history (article is from 2014!) And like I said, you can follow the links and see the brigade discussing something else entirely. Personally I don’t want an OSS ecosystem that banishes trans people or people who had weird proto-alt-right politics pre-Trump. If you’re gonna banish anyone, banish the ones posing existential risks to projects by their brigading a…

Yeah, I definitely prefer to be part of very inclusive and open OSS ecosystems, that do things in good faith.

I am not part of the YCombinator or West Coast ecosystem, but didn't it banish gay people with weird pro-alt-right politics pre-Trump, and then supported Trump? Like, for some reason there was a movement to banish Peter Thiel: https://mashable.com/article/peter-thiel-y-combinator

How do you feel about banishing people who simply admitted to voting for Proposition 8: https://www.latimes.com/business/technology/la-fi-tn-mozilla...

If you take a look at a larger problem, you'll see that there is a lot of inconsistency with human welfare on a far larger scale.

Because this same system takes Saudi money a lot. The only moment of self-reflection came after one guy, Kashoggi, was killed: https://www.barrons.com/articles/saudi-arabia-tech-fundraisi...

But not the situation of millions of people in Yemen: https://news.un.org/en/story/2022/03/1113852 ... https://techcrunch.com/2023/04/01/andreessen-horowitz-is-now...

The US military industrial complex was largely involved in airstrikes on Yemen, as the Washington Post revealed last year: https://www.washingtonpost.com/investigations/interactive/20...

As a country, we ignore the Yemen war, and are told to only clutch pearls about taking money from Russia due to the Ukraine war. I imagine that YC stopped taking Yuri Milner's money a decade ago, partly because of his ties to the Kremlin, but probably it was just a natural parting of ways eventually: https://news.ycombinator.com/item?id=15631084

Anyway, just saying ... ecosystems aren't always perfect.

Re: Using mmap to make LLaMA load faster

#169

Post by yazzku »

Operyl wrote: post #137
ncann wrote: post #126

Earlier quoted context omitted.

I found this summary from that thread: https://rentry.org/Jarted

Which Jart contests to some degree: https://news.ycombinator.com/item?id=35413289

The blog post is a narcissistic cringe ride, and the PR did not give credit to the level this Slarin appears to have been involved. Also, the change seems to cause a performance regression in some cases, which someone had also pointed out. It seemed to optimize for the case where only a subset of the weights are needed at the expense of the case where all of them are eventually needed. Seems the kind of thing that should have been tested prior to checking in.

Overall, these people would be better off taking their drama on Twitter or LinkedIn. ggerganov did the right thing kicking them out.

Re: Using mmap to make LLaMA load faster

#170

Post by 10000truths »

lostmsu wrote: post #156
10000truths wrote: post #138

Earlier quoted context omitted.

Except it’s not irreplaceable, at least on Linux. userfaultfd allows you to define custom page fault handling. With it, you can even do crazy things like “mmap” a remote resource by making HTTP range requests on a read fault.

Is there anything like that on Windows?

Windows has vectored exception handlers, which are a bit like UNIX signal handlers but much more sanely designed. You can use that to redirect control flow when a page fault occurs, and you can check the faulting address in your handler to scope it to a particular region of memory.
Post reply on HN