Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

github.com

Post reply on HN
Print view View original

21–30 of 146 posts

Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

#21

Post by rgbrgb »

austinvhuang wrote: post #14

Hi, one of the authors austin here. Happy to answer any questions the best I can. To get a few common questions out of the way: - This is separate / independent of llama.cpp / ggml. I'm a big fan of that project and it was an inspiration (we say as much in the README). I've been a big advocate of gguf + llama.cpp support for gemma and am happy for people to use that. - how is it different than inference runtime X? ge…

Thanks for releasing this! What is your use case for this rather than llama.cpp? For the on-device AI stuff I mostly do, llama.cpp is better because of GPU/metal offloading.

Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

#22

Post by dartharva »

austinvhuang wrote: post #14

Hi, one of the authors austin here. Happy to answer any questions the best I can. To get a few common questions out of the way: - This is separate / independent of llama.cpp / ggml. I'm a big fan of that project and it was an inspiration (we say as much in the README). I've been a big advocate of gguf + llama.cpp support for gemma and am happy for people to use that. - how is it different than inference runtime X? ge…

So... llamafile release?

https://github.com/Mozilla-Ocho/llamafile

Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

#23

Post by superkuh »

a1o wrote: post #2

If I want to put a Gemma model in a minimalist command line interface, build it to a standalone exe file that runs offline, what is the size of my final executable? I am interested in how small can the size of something like this be and it still be functional.

*EDIT*: Nevermind, llamafile hasn't been updated in a full month and gemma support was only added to llama.cpp on the 21st of this month. Disregard this post for now and come back when mozilla updates llamafile.

---

llama.cpp has integrated gemma support. So you can use llamafile for this. It is a standalone executable that is portable across most popular OSes.

https://github.com/Mozilla-Ocho/llamafile/releases

So, download the executable from the releases page under assets. You want either just main and server and llava. Don't get the huge ones with the model inlined in the file. The executable is about 30MB in size,

https://github.com/Mozilla-Ocho/llamafile/releases/download/...

Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

#24

Post by austinvhuang »

dartharva wrote: post #22
austinvhuang wrote: post #14

Hi, one of the authors austin here. Happy to answer any questions the best I can. To get a few common questions out of the way: - This is separate / independent of llama.cpp / ggml. I'm a big fan of that project and it was an inspiration (we say as much in the README). I've been a big advocate of gguf + llama.cpp support for gemma and am happy for people to use that. - how is it different than inference runtime X? ge…

So... llamafile release? https://github.com/Mozilla-Ocho/llamafile

gguf files are out there, so anyone should be able to do this! are people looking for an "official" version?

ps i'm a fan of cosmopolitan as well.

Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

#25

Post by beoberha »

austinvhuang wrote: post #14

Hi, one of the authors austin here. Happy to answer any questions the best I can. To get a few common questions out of the way: - This is separate / independent of llama.cpp / ggml. I'm a big fan of that project and it was an inspiration (we say as much in the README). I've been a big advocate of gguf + llama.cpp support for gemma and am happy for people to use that. - how is it different than inference runtime X? ge…

> Although this is from Google, we're a very small team that wanted such a codebase to exist. We have lots of plans to use it ourselves and we hope other people like it and find it useful.

This is really cool, Austin. Kudos to your team!

Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

#27

Post by sillysaurusx »

a1o wrote: post #16
brucethemoose2 wrote: post #8

Earlier quoted context omitted.

The code is a basically irrelevant fraction of the model weights. The raw FP16 is like 17GB. In practice your priority would be fancy quantization, and just any library that compiles down to an executable (like this, MLC-LLM or llama.cpp)

17GB looks like a lot. Thanks, I will wait until people figure how to make these smaller before trying to use to make something standalone.

The average PS5 game seems to be around 45GB. Cyberpunk was 250GB.

Distributing 17GB isn’t a big deal if you shove it into Cloudflare R2.

Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

#28

Post by brucethemoose2 »

a1o wrote: post #16
brucethemoose2 wrote: post #8

Earlier quoted context omitted.

The code is a basically irrelevant fraction of the model weights. The raw FP16 is like 17GB. In practice your priority would be fancy quantization, and just any library that compiles down to an executable (like this, MLC-LLM or llama.cpp)

17GB looks like a lot. Thanks, I will wait until people figure how to make these smaller before trying to use to make something standalone.

In theory quantized weights of smaller models are under a gigabyte.

If you are looking for megabytes, yeah, those "chat" llms are pretty unusable at that size.

Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

#29

Post by swatcoder »

a1o wrote: post #16
brucethemoose2 wrote: post #8

Earlier quoted context omitted.

The code is a basically irrelevant fraction of the model weights. The raw FP16 is like 17GB. In practice your priority would be fancy quantization, and just any library that compiles down to an executable (like this, MLC-LLM or llama.cpp)

17GB looks like a lot. Thanks, I will wait until people figure how to make these smaller before trying to use to make something standalone.

It's always going to be a huge quantity of data. Even as efficiency improves, storage and bandwidth are so cheap now that the incentive will be to convert that efficiency towards performance (models with more parameters, ensembles of models, etc) rather than chasing some micro-model that doesn't do as well. It might not always be 17GB, but don't expect some lesser order of magnitude for anything competitive.

As maturity arrives, we'll likely see a handful of competing local models shipped as part of the OS or as redistributable third-party bundles (a la the .NET or Java runtimes) so that individual applications don't all need to be massive.

You'll either need to wait for that or bite the bullet and make something chonky. It's never going to get that small.

Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

#30

Post by brucethemoose2 »

wg0 wrote: post #20
a1o wrote: post #16

Earlier quoted context omitted.

17GB looks like a lot. Thanks, I will wait until people figure how to make these smaller before trying to use to make something standalone.

These won't be smaller I guess. Given we keep the number of parameters same. Pre LLM era (let's say 2020), the hardware used to look decently powerful for most use cases (disks in hundreds of GBs, dozen or two of RAM and quad or hex core processors) but with the advent of LLMs, even disk drives start to look pretty small let alone compute and memory.

And cache! The talk of AI hardware is now "how do we fit these darn things inside SRAM?"
Post reply on HN