Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

github.com

Post reply on HN
Print view View original

71–80 of 298 posts

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#71

Post by rastapasta42 »

londons_explore wrote: post #67
recuter wrote: post #60

> Currently, only LLaMA-7B is supported since I haven't figured out how to merge the tensors of the bigger models. However, in theory, you should be able to run 65B on a 64GB MacBook Suddenly the choices Apple made with its silicon are looking like pure genius as there will be a lot of apps using this that are essentially exclusive to their platform. Even with the egregious ram pricing. With a lot of fine tuning if y…

LLaMA doesn't perform very well with answering questions. If you ask it "What color is the sky?" It will reply with something like "Why is ice cold? Why do we exist?"

Ask dumb questions receive dumb answers.

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#72

Post by guestbest »

RoboTeddy wrote: post #70
londons_explore wrote: post #67

Earlier quoted context omitted.

LLaMA doesn't perform very well with answering questions. If you ask it "What color is the sky?" It will reply with something like "Why is ice cold? Why do we exist?"

LLaMA isn't built on RLHF, so it may be necessary to create a more extensive prompt. For example: ``` You are a super intelligent honest question-answering system. Q: What's 2+2? A: 4 Q: What color is the sky? A: ```

This is Commodore 64 tier answers

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#73

Post by alphanullmeric »

recuter wrote: post #60

> Currently, only LLaMA-7B is supported since I haven't figured out how to merge the tensors of the bigger models. However, in theory, you should be able to run 65B on a 64GB MacBook Suddenly the choices Apple made with its silicon are looking like pure genius as there will be a lot of apps using this that are essentially exclusive to their platform. Even with the egregious ram pricing. With a lot of fine tuning if y…

Wonder why AMD/Intel/Nvidia haven’t invented some sort of device that allows the processor and graphics to share memory like Apple has done.

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#74

Post by tempaccount420 »

DesiLurker wrote: post #31

but why would you do C++ when its quite clear ML load is highly parallel. the page says vectorized by NEON but no mention whether its autovectorized by gcc or hand optimized. That will have a pretty significant performance impact.

You can read the code. It's open source.

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#76

Post by bigstumpy »

AlecSchueler wrote: post #64
attractivechaos wrote: post #15

Earlier quoted context omitted.

Both are very impressive. A nice thing about Gerganov's implementation is that it is written in almost pure C. Arguably easier for deployment.

What does almost mean in this case?

There is also C++

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#77

Post by slaymaker1907 »

alphanullmeric wrote: post #73
recuter wrote: post #60

> Currently, only LLaMA-7B is supported since I haven't figured out how to merge the tensors of the bigger models. However, in theory, you should be able to run 65B on a 64GB MacBook Suddenly the choices Apple made with its silicon are looking like pure genius as there will be a lot of apps using this that are essentially exclusive to their platform. Even with the egregious ram pricing. With a lot of fine tuning if y…

Wonder why AMD/Intel/Nvidia haven’t invented some sort of device that allows the processor and graphics to share memory like Apple has done.

Nvidia actually does support this via unified memory. It’s actually an amazing performance trick since you can avoid launching a bunch of kernels just to ferry data back and forth. I did this on a a GTX 1080 for a school project to speed up random shuffling of large amounts of data.

However, even without this feature, you can implement this sort of thing manually in most cases and you’re already being careful on a GPU to respect the cache (only working with one contiguous set of data of memory at a time).

Really, we just need some good systems devs working on running these huge models.

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#78

Post by PragmaticPulp »

alphanullmeric wrote: post #73
recuter wrote: post #60

> Currently, only LLaMA-7B is supported since I haven't figured out how to merge the tensors of the bigger models. However, in theory, you should be able to run 65B on a 64GB MacBook Suddenly the choices Apple made with its silicon are looking like pure genius as there will be a lot of apps using this that are essentially exclusive to their platform. Even with the egregious ram pricing. With a lot of fine tuning if y…

Wonder why AMD/Intel/Nvidia haven’t invented some sort of device that allows the processor and graphics to share memory like Apple has done.

PCIe devices like GPUs can access system memory.

Integrated GPUs also access system memory via the same bus as the CPU.

It’s not really a new technique. Apple just shipped a highly integrated unit with large memory bandwidth.

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#79

Post by kkielhofner »

alphanullmeric wrote: post #73
recuter wrote: post #60

> Currently, only LLaMA-7B is supported since I haven't figured out how to merge the tensors of the bigger models. However, in theory, you should be able to run 65B on a 64GB MacBook Suddenly the choices Apple made with its silicon are looking like pure genius as there will be a lot of apps using this that are essentially exclusive to their platform. Even with the egregious ram pricing. With a lot of fine tuning if y…

Wonder why AMD/Intel/Nvidia haven’t invented some sort of device that allows the processor and graphics to share memory like Apple has done.

The Nvidia Jetson/Tegra line did (and does) unified memory. Released 2014[0]. Nvidia already has ARM cores. They’re just terrible compared to Apple, Samsung, etc.

This is what the attempted ARM acquisition by Nvidia was about - with the ARM talent, IP, etc they’d be able to integrate more than just memory (GPU, CPU, connectivity via Mellanox, etc).

Regulators shut it down (for good reason) but I can’t help but think we would have seen some really interesting and revolutionary platforms come from it.

[0] - https://en.m.wikipedia.org/wiki/Tegra#Tegra_K1

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#80

Post by eigenvalue »

DesiLurker wrote: post #31

but why would you do C++ when its quite clear ML load is highly parallel. the page says vectorized by NEON but no mention whether its autovectorized by gcc or hand optimized. That will have a pretty significant performance impact.

Seems like it's extremely high performance for me on an M1 running the 7B model. Totally usable.
Post reply on HN