Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Using mmap to make LLaMA load faster

justine.lol

Post reply on HN
Print view View original

101–110 of 186 posts

Re: Using mmap to make LLaMA load faster

#101

Post by dasl »

> One of the downsides of the Linux cp command, is copying a file larger than RAM will destroy every existing entry in the file cache. Under normal circumstances this is a good thing, since a least recently used strategy usually works. However it can be problematic if you're just organizing your files on a production system where you don't want to disrupt performance. As far as I know, no standard command line utility offers a way to exploit this functionality.

I think `dd` in conjunction with the `oflag=direct` has this functionality. See: https://stackoverflow.com/questions/33485108/why-is-dd-with-...

Re: Using mmap to make LLaMA load faster

#102

Post by jancsika »

I still don't understand: why was the magic number changed in addition to the file version number?

Edit: can someone running llama.cpp ask it whether it thinks it's a good idea to concatenate a running list of vanity initials of important developers into a magic filetype constant?

Re: Using mmap to make LLaMA load faster

#103

Post by dekhn »

scottlamb wrote: post #87
simion314 wrote: post #71

Earlier quoted context omitted.

Thanks for the extra clarifications, but the claims were something impossible like a 23 Gb model only using 6Gb with this change. So maybe before this change it would have used a lot more of 23 Gb. I was referring to those miracle memory reductions, unfortunetly not possible, I would like to try 3 bit qunatizations when models and software will be ready(found none in my searches today)

Yes, those claims were a bit much, and in fairness jart chimed in to say so too. [1] fwiw, I'm not a ML person, but it doesn't seem entirely crazy to me to think that SSDs are becoming fast enough that you could avoid keeping a huge model in RAM in some cases. Especially if "computational SSDs" (SSDs that can do some basic first-stage computation without transferring the input data over PCIe) ever become common. (I t…

much of performance in computing is about moving the memory hierarchy around in ways that are inconvenient to programmers.

I made an SSD into a spare swap device, and basically treated my system as having RAM+SSD's worth of RAM. It allowed me to finish a few big jobs (~96GB RAM) overnight that wouldn't have otherwise.

Re: Using mmap to make LLaMA load faster

#105

Post by baffo32 »

antimatter15 wrote: post #76

Note that as a result of some llama drama associated with this change, @jart (author of this post) and @anzz1 are apparently no longer welcome as collaborators on llama.cpp: https://github.com/ggerganov/llama.cpp/pull/711#issuecomment...

justine worked hard on sorting out these changes and was attacked and experienced character assassination via orchestrated trolling, it’s so sad

Re: Using mmap to make LLaMA load faster

#106

Post by adeon »

VadimPR wrote: post #81
adeon wrote: post #50

Earlier quoted context omitted.

I have got Vicuna-13B working on GTX 3090 Ti + OpenCL + CPU with 90% of weights on the GPU (otherwise running out of memory) at around 500ms per token. This model is really good for a (semi-)open source model. I think this may be the first locally runnable model that I will actually use for real stuff rather than just play around for fun. It's not ChatGPT level but it's not that far behind. It will draw ASCII art HUD…

Mind sharing how you got it to work in your setup?

I work on an independent LLM implementation here: https://github.com/Noeda/rllama/

I only got it working at all yesterday and there's no nice UX at all. Not sure I recommend trying to use this as llama.cpp will probably have this in no time with a much better user experience, although I am also trying to make it more usable.

If you follow the instructions on Vicuna page over how to apply the deltas, and you can compile the project, then you could run:

cargo run --release --features opencl -- --model-path /models/vicuna13b --param-path /models/vicuna13b/config.json --tokenizer-path /models/vicuna13b/tokenizer.model --prompt-file prompt --top-p 1.0 --top-k 20 --repetition-penalty 1 --temperature 0.9 --max-seq-len 2048 --f16 --percentage-to-gpu 0.9

Where /models/vicuna13b is the HuggingFace-compatible model. This will put 90% of weights on GPU and remaining 10% non CPU which is just barely enough to not run out of GPU memory (on a 24 gig card)

Create a text file 'prompt' with the prompt. I've been using this template:

You are a helpful and precise assistant for checking the quality of the answer.###Human: Can you explain nuclear power to me?###Assistant:

(the model seems to use ### as delimiters to distinguish Human and Assistant). The "system prompt" is whatever text is written at the beginning.

Re: Using mmap to make LLaMA load faster

#107

Post by mew12623 »

dekhn wrote: post #100
timpani wrote: post #94

Earlier quoted context omitted.

[flagged]

This sort of statement really don't contribute anythign to the discussion and in fact greatly distracts from the technical content. We don't need to hear your opinions about trans people in this thread.

[dead]

Re: Using mmap to make LLaMA load faster

#108

Post by mew12623 »

baffo32 wrote: post #105
antimatter15 wrote: post #76

Note that as a result of some llama drama associated with this change, @jart (author of this post) and @anzz1 are apparently no longer welcome as collaborators on llama.cpp: https://github.com/ggerganov/llama.cpp/pull/711#issuecomment...

justine worked hard on sorting out these changes and was attacked and experienced character assassination via orchestrated trolling, it’s so sad

[dead]

Re: Using mmap to make LLaMA load faster

#109

Post by MuffinFlavored »

HopenHeyHi wrote: post #96
MuffinFlavored wrote: post #70

Earlier quoted context omitted.

[flagged]

No? I mean yes, probably will get downvotes from the usual crowd. There is as a matter of fact some of that explicitly going on in the GitHub issue, which derailed the conversation. Such are the woes of being an internet micro celebrity I guess. Truly we live in the stupidest timeline.

> There is as a matter of fact some of that explicitly going on in the GitHub issue

Could you link an example so I can learn/retract my comment? I didn't think there was actually going on.

Re: Using mmap to make LLaMA load faster

#110

Post by adeon »

wensheng wrote: post #90
adeon wrote: post #50

Earlier quoted context omitted.

I have got Vicuna-13B working on GTX 3090 Ti + OpenCL + CPU with 90% of weights on the GPU (otherwise running out of memory) at around 500ms per token. This model is really good for a (semi-)open source model. I think this may be the first locally runnable model that I will actually use for real stuff rather than just play around for fun. It's not ChatGPT level but it's not that far behind. It will draw ASCII art HUD…

How did you distribute the weights between CPU and GPU? Thanks

See my response on the sibling comment; I implemented it in a custom Rust implementation.
Post reply on HN