Using mmap to make LLaMA load faster • Hacker News Board

Re: Using mmap to make LLaMA load faster

#61

Post by chad1n » Wed, Apr 05, 2023, 5:38 PM UTC

514723553 wrote: post #55

chad1n wrote: post #41

Earlier quoted context omitted.

[flagged]

If people were mad at Greg for changing or not changing a magic number--then they can just make a PR to fix it! That'd be so easy!

Greg didn't change it, it was changed in Jart's pull request. Also they can't just make a PR to fix it, because the models were already converted to that magic string that was changed for no reason.

Re: Using mmap to make LLaMA load faster

#62

Post by csmpltn » Wed, Apr 05, 2023, 5:38 PM UTC

astrange wrote: post #38

candrewlee14 wrote: post #14

jart is a genius. What they’ve done with Blink, Cosmopolitan C, Redbean, and now llama.cpp is incredible. It gives me hope for the future of systems/low-level programming.

Using mmap is basic systems programming knowledge and not "incredible" even though it's a useful contribution.

Glad to see we're slowly coming to the realization that wrangling NumPy and Pandas doesn't actually count as Software Engineering.

Re: Using mmap to make LLaMA load faster

#63

Post by djur » Wed, Apr 05, 2023, 5:41 PM UTC

HopenHeyHi wrote: post #16

NickGerleman wrote: post #6

Worth pointing out, there has been quite a bit of contention around this change, both technical, and some accusations of plagiarism/miscrediting here. https://github.com/ggerganov/llama.cpp/pull/711

Unfortunately Justine has attracted a peculiar fanbase+haterbase. As their numbers swell the collective intelligence and technical understanding diminishes. So the discussions end up gravitating towards weird drama. I wish you wouldn't have linked this thread. Theres going to be a bunch of stupid comments here as well about how great/awful jart is.

Is this related to her advocacy for neoreactionary politics or is it just a transphobia thing?

Re: Using mmap to make LLaMA load faster

#64

Post by dundarious » Wed, Apr 05, 2023, 5:41 PM UTC

astrange wrote: post #51

> I don't think I've ever seen a high-level library that's able to do what mmap() does, because it defies attempts at abstraction. I'm not sure what this means but I'm pretty sure I can name several "high level libraries" that mmap things. None of those are the STL, but it's not exactly perfect design.

I read it to mean mmap is irreplaceable. There is no other sophisticated dance of system calls or userspace trickery that can achieve what mmap can achieve. She's saying that everything up and down the stack, including high level libraries, do just call mmap, because there would be no DIY alternative with similar cost-benefit.

Re: Using mmap to make LLaMA load faster

#65

Post by plorkyeran » Wed, Apr 05, 2023, 5:42 PM UTC

mhh__ wrote: post #56

astrange wrote: post #51

> I don't think I've ever seen a high-level library that's able to do what mmap() does, because it defies attempts at abstraction. I'm not sure what this means but I'm pretty sure I can name several "high level libraries" that mmap things. None of those are the STL, but it's not exactly perfect design.

I'm not sure what really needs abstracting about it other than tuning stuff for a given kernel/system. It's already extremely high level from a certain perspective :)

Boost.interprocess is an example of an abstraction over mmap which solves some of the things the blog post mentions. It abstracts away the difference between mmap() and CreateViewOfFile(), and gives you smart pointers and container types which are close to being drop-in replacements for std::vector and std::map that can be stored in a memory-mapped file.

Re: Using mmap to make LLaMA load faster

#66

Post by kuroguro » Wed, Apr 05, 2023, 5:42 PM UTC

Lol, there's a "broken" image in the footer that claims your IP for ipv4.games http://ipv4.games/user.html?name=jart

Re: Using mmap to make LLaMA load faster

#67

Post by eternalban » Wed, Apr 05, 2023, 5:42 PM UTC

yieldcrv wrote: post #11

NickGerleman wrote: post #6

Worth pointing out, there has been quite a bit of contention around this change, both technical, and some accusations of plagiarism/miscrediting here. https://github.com/ggerganov/llama.cpp/pull/711

only thing this discussion has showed me is that more people need Computer Science degrees again like, wow, mmap and paging. really guys?

I actually don't understand this. I think a lot of regular hn commentators are just avoiding these threads [given the questionable circumstances surrounding the related PRs and the "drama"].

We've had regular discussions on HN about various storage engines, how the latencies are cut down, etc. I share your surprise at hearing 'wow, mmap!' and all the debates in the issues as what it actually does.

https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...

Re: Using mmap to make LLaMA load faster

#68

Post by oceanplexian » Wed, Apr 05, 2023, 5:44 PM UTC

HopenHeyHi wrote: post #16

NickGerleman wrote: post #6

Worth pointing out, there has been quite a bit of contention around this change, both technical, and some accusations of plagiarism/miscrediting here. https://github.com/ggerganov/llama.cpp/pull/711

Unfortunately Justine has attracted a peculiar fanbase+haterbase. As their numbers swell the collective intelligence and technical understanding diminishes. So the discussions end up gravitating towards weird drama. I wish you wouldn't have linked this thread. Theres going to be a bunch of stupid comments here as well about how great/awful jart is.

I'm not a fan or a hater, I didn't even know who this person was until this thread.

Does the change deserve a blog post or wild claims like "llama.cpp is 100x faster and uses half the memory!"? No. The original PR looks like a decent addition but the blog posts reads as incredibly narcissistic (i.e. lots of language like "We spent several weeks volunteering" and "our project") uh whatever. It also breaks a backwards compatibility when there's no technical reason it couldn't have been optional or put behind a feature flag, plus a ton of condescending language in the PR. Not really the kind of work I'd be proud of or would be advertising in a blog post.

Re: Using mmap to make LLaMA load faster

#69

Post by HopenHeyHi » Wed, Apr 05, 2023, 5:46 PM UTC

djur wrote: post #63

HopenHeyHi wrote: post #16

Earlier quoted context omitted.

Unfortunately Justine has attracted a peculiar fanbase+haterbase. As their numbers swell the collective intelligence and technical understanding diminishes. So the discussions end up gravitating towards weird drama. I wish you wouldn't have linked this thread. Theres going to be a bunch of stupid comments here as well about how great/awful jart is.

Is this related to her advocacy for neoreactionary politics or is it just a transphobia thing?

[flagged]

Re: Using mmap to make LLaMA load faster

#70

Post by MuffinFlavored » Wed, Apr 05, 2023, 5:47 PM UTC

mhh__ wrote: post #52

I've seen this on my HN feed about 5 times now, twitter twice. Why? (Author's blog is acceptable as they did the change, but the other 4?) Is this AI people learning that mmap exists? People simping cos' there a Justine involved?

[flagged]