Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

github.com

Post reply on HN
Print view View original

91–100 of 146 posts

Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

#91

Post by brigadier132 »

zoogeny wrote: post #71

I know a lot of people chide Google for being behind OpenAI in their commercial offerings. We also dunk on them for the over-protective nature of their fine-tuning. But Google is scarily capable on the LLM front and we shouldn't count them out. OpenAI might have the advantage of being quick to move, but when the juggernaut gets passed its resting inertia and starts to gain momentum it is going to leave an impression.…

There was a podcast yesterday that explained well why Google is in a tough position.

https://youtu.be/-i9AGk3DJ90?t=616

In essence, Google already rules information retrieval. Their margins are insane. Switching to LLM based search cuts into their margins and increases their costs dramatically. Also, the advantage they've built over decades has been cut down.

All of this means there is potential for less profit and a shrinking valuation. A shrinking valuation means issues with employee retention and it could lead to long term stagnation.

Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

#92

Post by trisfromgoogle »

brucethemoose2 wrote: post #5

...Also, we have eval'd Gemma 7B internally in a deterministic, zero temperature test, and its error rate is like double Mistral Instruct 0.2. Well below most other 7Bs. Was not very impressed with the chat either. So maybe this is neat for embedded projects, but if it's Gemma only, that would be quite a sticking point for me.

Any chance you can share more details on your measurement setup and eval protocols? You're likely seeing some config snafus, which we're trying to track down.

Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

#93

Post by verticalscaler »

trisfromgoogle wrote: post #87
verticalscaler wrote: post #62

Earlier quoted context omitted.

Hi Austin, what say you about how the Gemma rollout was handled, issues raised, and atmosphere around the office? :)

I'm not Austin, but I am Tris, the friendly neighborhood product person on Gemma. Overall, I think that the main feeling is: incredibly relieved to have had the launch go as smoothly as it has! The complexity of the launch is truly astounding: 1) Reference implementations in JAX, PyTorch, TF with Keras 3, MaxText/JAX, more... 2) Full integration at launch with HF including Transformers + optimization therein 3) Tenso…

[flagged]

Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

#94

Post by Gracana »

moffkalast wrote: post #64
throwaway19423 wrote: post #44

Can any kind soul explain the difference between GGUF, GGML and all the other model packaging I am seeing these days? Was used to pth and the thing tf uses. Is this all to support inference or quantization? Who manages these formats or are they brewing organically?

It's all mostly just inference, though some train LoRAs directly on quantized models too. GGML and GGUF are the same thing, GGUF is the new version that adds more data about the model so it's easy to support multiple architectures, and also includes prompt templates. These can run CPU only, be partially or fully offloaded to a GPU. With K quants, you can get anywhere from a 2 bit to an 8 bit GGUF. GPTQ was the GPU-on…

> and also includes prompt templates

That sounds very convenient. What software makes use of the built-in prompt template?

Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

#95

Post by trisfromgoogle »

verticalscaler wrote: post #93
trisfromgoogle wrote: post #87

Earlier quoted context omitted.

I'm not Austin, but I am Tris, the friendly neighborhood product person on Gemma. Overall, I think that the main feeling is: incredibly relieved to have had the launch go as smoothly as it has! The complexity of the launch is truly astounding: 1) Reference implementations in JAX, PyTorch, TF with Keras 3, MaxText/JAX, more... 2) Full integration at launch with HF including Transformers + optimization therein 3) Tenso…

[flagged]

Always -- anything that comes with the Google name attached always attracts some negativity. There's plenty of valid criticism, most of which we hope to address in the coming weeks and months =).

Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

#96

Post by verticalscaler »

trisfromgoogle wrote: post #95
verticalscaler wrote: post #93

Earlier quoted context omitted.

[flagged]

Always -- anything that comes with the Google name attached always attracts some negativity. There's plenty of valid criticism, most of which we hope to address in the coming weeks and months =).

[flagged]

Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

#97

Post by trisfromgoogle »

verticalscaler wrote: post #96
trisfromgoogle wrote: post #95

Earlier quoted context omitted.

Always -- anything that comes with the Google name attached always attracts some negativity. There's plenty of valid criticism, most of which we hope to address in the coming weeks and months =).

[flagged]

I mean, many articles will have a negative cast because of the need for clicks -- e.g., the Verge's launch article is entitled "Google Gemma: because Google doesn’t want to give away Gemini yet" -- which I think is both an unfair characterization (given the free tier of Gemini Pro) and unnecessarily inflammatory.

Legitimate criticisms include not working correctly out of the box for llama.cpp due to repetition penalty and vocab size, some snafus on chat templates with huggingface, the fact that they're not larger-sized models, etc. Lots of the issues are already fixed, and we're committed to making sure these models are great.

Honestly, not sure what you're trying to get at here -- are you trying to "gotcha" the fact that not everything is perfect? That's true for any launch.

Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

#98

Post by verticalscaler »

trisfromgoogle wrote: post #97
verticalscaler wrote: post #96

Earlier quoted context omitted.

[flagged]

I mean, many articles will have a negative cast because of the need for clicks -- e.g., the Verge's launch article is entitled "Google Gemma: because Google doesn’t want to give away Gemini yet" -- which I think is both an unfair characterization (given the free tier of Gemini Pro) and unnecessarily inflammatory. Legitimate criticisms include not working correctly out of the box for llama.cpp due to repetition penalt…

[flagged]

Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

#99

Post by null_point »

brucethemoose2 wrote: post #57
throwaway19423 wrote: post #45

Earlier quoted context omitted.

I am confused how all these things are able to interoperate. Are the creators of these models following the same IO for their models? Won't the tokenizer or token embedder be different? I am genuinely confused by how the same code works for so many different models.

It's complicated, but basically because most are llama architecture. Meta all but set the standard for open source llms when they released llama1, and anyone trying to deviate from it has run into trouble because the models don't work with the hyper optimized llama runtumes. Also, there's a lot of magic going on behind the scenes with configs stored in gguf/huggingface format models, and the libraries that use them.…

I found the magic! https://github.com/search?q=repo%3Aggerganov%2Fggml%20magic&...

Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

#100

Post by trisfromgoogle »

verticalscaler wrote: post #98
trisfromgoogle wrote: post #97

Earlier quoted context omitted.

I mean, many articles will have a negative cast because of the need for clicks -- e.g., the Verge's launch article is entitled "Google Gemma: because Google doesn’t want to give away Gemini yet" -- which I think is both an unfair characterization (given the free tier of Gemini Pro) and unnecessarily inflammatory. Legitimate criticisms include not working correctly out of the box for llama.cpp due to repetition penalt…

[flagged]

Neither of those applies at all to Gemma, though? I'm still confused -- what are you trying to accomplish with this line of questioning?
Post reply on HN