Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

github.com

Post reply on HN
Print view View original

141–146 of 146 posts

Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

#141

Post by null_point »

null_point wrote: post #99
brucethemoose2 wrote: post #57

Earlier quoted context omitted.

It's complicated, but basically because most are llama architecture. Meta all but set the standard for open source llms when they released llama1, and anyone trying to deviate from it has run into trouble because the models don't work with the hyper optimized llama runtumes. Also, there's a lot of magic going on behind the scenes with configs stored in gguf/huggingface format models, and the libraries that use them.…

I found the magic! https://github.com/search?q=repo%3Aggerganov%2Fggml%20magic&...

Hey, c'mon now. Just being playful about the "magic" string used in GGUF files to detect that it is in-fact a GGUF file.

Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

#143

Post by leminimal »

austinvhuang wrote: post #66
leminimal wrote: post #53

Earlier quoted context omitted.

Kudos on your release! I know this was just made available but - Somewhere the README, consider adding the need for a `-DWEIGHT_TYPE=hwy::bfloat16_t` flag for non-sfp. Maybe around step 3. - The README should explicitly say somehere that there's no GPU support (at the moment) - "Failed to read cache gating_ein_0 (error 294)" is pretty obscure. I think even "(error at line number 294)" would be a big improvement when…

Yes - thanks for pointing that out. The README is being updated, you can see an updated WIP in the dev branch: https://github.com/google/gemma.cpp/tree/dev?tab=readme-ov-f... and improving error messages is a high priority. The weights should be the same across formats, but it's easy for differences to arise due to quantization and/or subtle implementation differences. Minor implementation differences has been a pain…

Thanks, I'm glad to see your time machine caught my comment.

I'm using the 32-bit GGUF model from the Google repo, not a different quantized model, so I could have one less source of error. It's hard to tell with LLMs if its a bug. It just gives slightly stranger answers sometimes, but it's not completely gibberish. or incoherent sentences or have extra punctuations like with some other LLM bugs I've seen.

Still, I'll wait a few days to build llama.cpp again to see if there are any changes.

Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

#144

Post by rldjbpin »

zoogeny wrote: post #71

I know a lot of people chide Google for being behind OpenAI in their commercial offerings. We also dunk on them for the over-protective nature of their fine-tuning. But Google is scarily capable on the LLM front and we shouldn't count them out. OpenAI might have the advantage of being quick to move, but when the juggernaut gets passed its resting inertia and starts to gain momentum it is going to leave an impression.…

more than 5 years ago, Google already had an advanced platform for serving llm with dialogflow (ex-api.ai before they got acquired by Google). their feature list vastly outdo what openai is offering today.

the core difference is the core model's performance, but the maximum potential is night and day imho. they had a good chance to do well in this front but did not end up making the most yet.

Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

#146

Post by austinvhuang »

ManasRao wrote: post #145

Hello Austin, I would like to inquire about the accessibility of the code and weights for the Gemma Model. Is this information publicly available?

Hi, haven't followed this thread for a while so just happened to see this now.

I'm assuming you mean in other languages/implementations? (since the gemma.cpp repo linked above has code + links for gemma.cpp specific weights)

If so, you can find the weights here https://www.kaggle.com/models/google/gemma - each of the "model variations" (flax, jax, pytorch, keras, etc.) has a download for the weights and links to its code.

If you're comfortable with flax, that's DM's own reference implementation: https://github.com/google-deepmind/gemma

Post reply on HN