Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

github.com

Post reply on HN
Print view View original

61–70 of 146 posts

Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

#61

Post by moffkalast »

austinvhuang wrote: post #60
moffkalast wrote: post #58

Earlier quoted context omitted.

In my experience there's really no reason to run any model above Q6_K, the performance is identical and you shave off almost 2 GB of VRAM of a 7B model compared to Q8. To those of us with single digit amounts, that's highly significant. But most people seem to go for 4 bits anyway and it's the AWQ standard too. If you think it'll make the model look bad, then don't worry, it's only the relative performance that matte…

A nice side effect of implementing cpu simd is you just need enough regular RAM, which tends to be far less scarce than VRAM. Nonetheless, I get your point that more aggressive quantization is valuable + will share with the modeling team.

True, it's the only way I can for example run Mixtral on a 8GB GPU, but main memory will always have more latency so some tradeoff tends to be worth it. And parts like the prompt batch buffer and most of the context generally have to be in VRAM if you want to use cuBLAS, with OpenBLAS it's maybe less of a problem, but it is slower.

Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

#62

Post by verticalscaler »

austinvhuang wrote: post #14

Hi, one of the authors austin here. Happy to answer any questions the best I can. To get a few common questions out of the way: - This is separate / independent of llama.cpp / ggml. I'm a big fan of that project and it was an inspiration (we say as much in the README). I've been a big advocate of gguf + llama.cpp support for gemma and am happy for people to use that. - how is it different than inference runtime X? ge…

Hi Austin, what say you about how the Gemma rollout was handled, issues raised, and atmosphere around the office? :)

Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

#63

Post by [deleted] »

austinvhuang wrote: post #14

Hi, one of the authors austin here. Happy to answer any questions the best I can. To get a few common questions out of the way: - This is separate / independent of llama.cpp / ggml. I'm a big fan of that project and it was an inspiration (we say as much in the README). I've been a big advocate of gguf + llama.cpp support for gemma and am happy for people to use that. - how is it different than inference runtime X? ge…

[deleted]

Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

#64

Post by moffkalast »

throwaway19423 wrote: post #44

Can any kind soul explain the difference between GGUF, GGML and all the other model packaging I am seeing these days? Was used to pth and the thing tf uses. Is this all to support inference or quantization? Who manages these formats or are they brewing organically?

It's all mostly just inference, though some train LoRAs directly on quantized models too.

GGML and GGUF are the same thing, GGUF is the new version that adds more data about the model so it's easy to support multiple architectures, and also includes prompt templates. These can run CPU only, be partially or fully offloaded to a GPU. With K quants, you can get anywhere from a 2 bit to an 8 bit GGUF.

GPTQ was the GPU-only optimized quantization method that was superseded by AWQ, which is roughly 2x faster and now by EXL2 which is even better. These are usually only 4 bit.

Safetensors and pytorch bin files are raw float16 model files, these are only really used for continued fine tuning.

Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

#66

Post by austinvhuang »

leminimal wrote: post #53
austinvhuang wrote: post #14

Hi, one of the authors austin here. Happy to answer any questions the best I can. To get a few common questions out of the way: - This is separate / independent of llama.cpp / ggml. I'm a big fan of that project and it was an inspiration (we say as much in the README). I've been a big advocate of gguf + llama.cpp support for gemma and am happy for people to use that. - how is it different than inference runtime X? ge…

Kudos on your release! I know this was just made available but - Somewhere the README, consider adding the need for a `-DWEIGHT_TYPE=hwy::bfloat16_t` flag for non-sfp. Maybe around step 3. - The README should explicitly say somehere that there's no GPU support (at the moment) - "Failed to read cache gating_ein_0 (error 294)" is pretty obscure. I think even "(error at line number 294)" would be a big improvement when…

Yes - thanks for pointing that out. The README is being updated, you can see an updated WIP in the dev branch: https://github.com/google/gemma.cpp/tree/dev?tab=readme-ov-f... and improving error messages is a high priority.

The weights should be the same across formats, but it's easy for differences to arise due to quantization and/or subtle implementation differences. Minor implementation differences has been a pain point in the ML ecosystem for a while (w/ IRs, onnx, python vs. runtime, etc.), but hopefully the differences aren't too significant (if they are, it's a bug in one of the implementations).

There were quantization fixes like https://twitter.com/ggerganov/status/1760418864418934922 and other patches happening, but it may take a few days for patches to work their way through the ecosystem.

Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

#67

Post by dankle »

austinvhuang wrote: post #14

Hi, one of the authors austin here. Happy to answer any questions the best I can. To get a few common questions out of the way: - This is separate / independent of llama.cpp / ggml. I'm a big fan of that project and it was an inspiration (we say as much in the README). I've been a big advocate of gguf + llama.cpp support for gemma and am happy for people to use that. - how is it different than inference runtime X? ge…

What's the reason to not integrate with llama.cpp instead of a separate app? In what ways this better than llama.cpp?

Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

#68

Post by liuliu »

throwaway19423 wrote: post #44

Can any kind soul explain the difference between GGUF, GGML and all the other model packaging I am seeing these days? Was used to pth and the thing tf uses. Is this all to support inference or quantization? Who manages these formats or are they brewing organically?

pth can include Python code (PyTorch code) for inference. TF includes the complete static graph.

GGUF is just weights, safetensors the same thing. GGUF doesn't need a JSON decoder for the format while safetensors needs that.

I personally think having a JSON decoder is not a big deal and make the format more amendable, given GGUF evolves too.

Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

#69

Post by ComputerGuru »

next_xibalba wrote: post #40

Is this neutered in the way Gemini is (i.e. is the "censorship" built in) or is that a "feature" of the Gemini application?

It depends on the model you load/use, the team released both censored and "PT" versions.

Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

#70

Post by cyanydeez »

ertgbnm wrote: post #17
colesantiago wrote: post #9

Isn't there a huge risk that Google could most likely deprecate Gemini, Gemma and Gemma.cpp? Not really smart to build on anything with Google e.g. Google Cloud for AI. Has this perception changed or pretty much the same?

The weights are downloadable so there isn't much of a risk if Google stops hosting Gemma apart from the fact that it won't get new versions that you swap out in the future.

even if there's a new model, I'm not seeing how these models provide any reliability metric.

if you figure out a money making software/service, you're gonna be tied to that model to some significant degree.

Post reply on HN