Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Using mmap to make LLaMA load faster

justine.lol

Post reply on HN
Print view View original

81–90 of 186 posts

Re: Using mmap to make LLaMA load faster

#81

Post by VadimPR »

adeon wrote: post #50
menzoic wrote: post #8

Earlier quoted context omitted.

Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality by the Team with members from UC Berkeley, CMU, Stanford, and UC San Diego https://vicuna.lmsys.org/

I have got Vicuna-13B working on GTX 3090 Ti + OpenCL + CPU with 90% of weights on the GPU (otherwise running out of memory) at around 500ms per token. This model is really good for a (semi-)open source model. I think this may be the first locally runnable model that I will actually use for real stuff rather than just play around for fun. It's not ChatGPT level but it's not that far behind. It will draw ASCII art HUD…

Mind sharing how you got it to work in your setup?

Re: Using mmap to make LLaMA load faster

#82

Post by BeefWellington »

chad1n wrote: post #41
NickGerleman wrote: post #6

Worth pointing out, there has been quite a bit of contention around this change, both technical, and some accusations of plagiarism/miscrediting here. https://github.com/ggerganov/llama.cpp/pull/711

[flagged]

Greg's own comment on the changes:

> Regarding the version comment - yes, the plan was to bump versions and no the magic. But I'm ok to change the magic to commemorate the significance of this update. In fact, maybe we can make this a thing and everybody who makes a significant contribution to the project will get their initials appended to the version. What do you think? smile

Re: Using mmap to make LLaMA load faster

#83

Post by version_five »

antimatter15 wrote: post #76

Note that as a result of some llama drama associated with this change, @jart (author of this post) and @anzz1 are apparently no longer welcome as collaborators on llama.cpp: https://github.com/ggerganov/llama.cpp/pull/711#issuecomment...

I can't really understand what's going on after a few minutes poking around (best I can tell an argument over attribution?), but I certainly sympathize with the project author having to deal with some stupid internet drama that takes away from a cool project. This seems to be one of the less-discussed challenges with popular open-source.

Re: Using mmap to make LLaMA load faster

#84

Post by IAmNotACellist »

514723553 wrote: post #44
NickGerleman wrote: post #6

Worth pointing out, there has been quite a bit of contention around this change, both technical, and some accusations of plagiarism/miscrediting here. https://github.com/ggerganov/llama.cpp/pull/711

I don't get the "plagiarism/miscrediting" accusations. This was in the original PR ( https://github.com/ggerganov/llama.cpp/pull/613 ): > This PR was written in collaboration with @slaren. This PR is also rebased on PR #586 so please do not squash merge! Use either merge or rebase. jart made sure to that the other user got credit, in addition to making sure that their name was properly attributed in the commit log. G…

That's not the original PR. jart was working on a malloc() approach that didn't work and slaren wrote all the code actually doing mmap, which jart then rebased in a random new PR, changed to support an unnecessary version change, magic numbers, a conversion tool, and WIN32 support when that was already working in the draft PR. https://archive.ph/Uva8c

This is the original PR: https://github.com/ggerganov/llama.cpp/pull/586.

Jart's archived comments:

"my changes"

"Here's how folks in the community have been reacting to my work."

"I just wrote a change that's going to let your LLaMA models load instantly..."

https://archive.ph/PyPFZ

"I'm the author"

https://archive.ph/qFrcY

"Author here..."

"Tragedy of the commons...We're talking to a group of people who live inside scientific papers and jupyer notebooks."

"My change helps inference go faster."

"The point of my change..."

"I stated my change offered a 2x improvement in memory usage."

https://archive.ph/k34V2

"I can only take credit for a 2x recrease in RAM usage."

https://archive.ph/MBPN0

"I just wrote a change that's going to let your LLaMA models load instantly, thanks to custom malloc() and the power of mmap()"

https://archive.ph/yrMwh

slaren replied to jart on HN asking her why she was doing and saying those things, and she didn't bother to reply to him, despite replying to others in that subthread within minutes. https://archive.ph/zCfiJ

Re: Using mmap to make LLaMA load faster

#85

Post by marginalia_nu »

jjoonathan wrote: post #79
marginalia_nu wrote: post #49

Earlier quoted context omitted.

> only work when you have much more RAM than the files you're mapping in. Really depends on what you're doing, like memory access patterns. I've definitely seen scenarios when mapping hundreds of gigabytes of data on dozens of gigabytes of ram where mmap has been an almost absurd performance boost over traditional I/O, both immediately but also asymptotically as all the most frequently accessed data ends up in cache…

> least accessed data is paged out Aren't all the weights touched in every pass?

Speaking in general.

Re: Using mmap to make LLaMA load faster

#86

Post by whimsicalism »

chad1n wrote: post #80
vore wrote: post #53

Earlier quoted context omitted.

jart is a woman, for what it's worth.

I just presented the drama and I referred to him as `he` as a developer, my teammates are mostly male, so I usually say `he`.

Right, so maybe you shouldn't do that?

Re: Using mmap to make LLaMA load faster

#87

Post by scottlamb »

simion314 wrote: post #71
scottlamb wrote: post #59

Earlier quoted context omitted.

Unless the prior code was using O_DIRECT, the data was getting loaded into the kernel's page cache, and then the application was copying it into its own anonymous memory. Now the copy isn't happening. There are some subtleties involved [1] but it's not crazy to claim approximately half the RAM usage, even before bringing multiple processes into the picture. [1] The kernel doesn't necessarily load the whole thing into…

Thanks for the extra clarifications, but the claims were something impossible like a 23 Gb model only using 6Gb with this change. So maybe before this change it would have used a lot more of 23 Gb. I was referring to those miracle memory reductions, unfortunetly not possible, I would like to try 3 bit qunatizations when models and software will be ready(found none in my searches today)

Yes, those claims were a bit much, and in fairness jart chimed in to say so too. [1]

fwiw, I'm not a ML person, but it doesn't seem entirely crazy to me to think that SSDs are becoming fast enough that you could avoid keeping a huge model in RAM in some cases. Especially if "computational SSDs" (SSDs that can do some basic first-stage computation without transferring the input data over PCIe) ever become common. (I think some of the ML accelerators for sale today might be approximately this.)

[1] https://news.ycombinator.com/item?id=35393615

Re: Using mmap to make LLaMA load faster

#88

Post by whimsicalism »

oceanplexian wrote: post #68
HopenHeyHi wrote: post #16

Earlier quoted context omitted.

Unfortunately Justine has attracted a peculiar fanbase+haterbase. As their numbers swell the collective intelligence and technical understanding diminishes. So the discussions end up gravitating towards weird drama. I wish you wouldn't have linked this thread. Theres going to be a bunch of stupid comments here as well about how great/awful jart is.

I'm not a fan or a hater, I didn't even know who this person was until this thread. Does the change deserve a blog post or wild claims like "llama.cpp is 100x faster and uses half the memory!"? No. The original PR looks like a decent addition but the blog posts reads as incredibly narcissistic (i.e. lots of language like "We spent several weeks volunteering" and "our project") uh whatever. It also breaks a backwards…

> The original PR looks like a decent addition but the blog posts reads as incredibly narcissistic

Better than most stuff I see in the corporate world.

Re: Using mmap to make LLaMA load faster

#89

Post by saagarjha »

csmpltn wrote: post #62
astrange wrote: post #38

Earlier quoted context omitted.

Using mmap is basic systems programming knowledge and not "incredible" even though it's a useful contribution.

Glad to see we're slowly coming to the realization that wrangling NumPy and Pandas doesn't actually count as Software Engineering.

…I don’t see how this is related?

Re: Using mmap to make LLaMA load faster

#90

Post by wensheng »

adeon wrote: post #50
menzoic wrote: post #8

Earlier quoted context omitted.

Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality by the Team with members from UC Berkeley, CMU, Stanford, and UC San Diego https://vicuna.lmsys.org/

I have got Vicuna-13B working on GTX 3090 Ti + OpenCL + CPU with 90% of weights on the GPU (otherwise running out of memory) at around 500ms per token. This model is really good for a (semi-)open source model. I think this may be the first locally runnable model that I will actually use for real stuff rather than just play around for fun. It's not ChatGPT level but it's not that far behind. It will draw ASCII art HUD…

How did you distribute the weights between CPU and GPU? Thanks
Post reply on HN