Thank the team for the awesome repo. I have navigated gemma.cpp and run it from the first day, it is smooth in my view. So I hope gemma.cpp will continue to add cool features (something like k-quants, server,...) so it can serve more widely.
Actually, I have developed a Python wrapper for it:
https://github.com/namtranase/gemma-cpp-python
The purpose is to use easily and update every new technique from gemma.cpp team.
Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models
81–90 of 146 posts
Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models
#82I know a lot of people chide Google for being behind OpenAI in their commercial offerings. We also dunk on them for the over-protective nature of their fine-tuning. But Google is scarily capable on the LLM front and we shouldn't count them out. OpenAI might have the advantage of being quick to move, but when the juggernaut gets passed its resting inertia and starts to gain momentum it is going to leave an impression.…
Maybe someone who knows google better can answer my question here: are they behind simply because LLMs are not really their core business? In other words, it wasn't (and still isn't) obvious that LLMs will help them sell add space. And of course writing that gives me a terrible realization: product placement in LLMs is going to be a very big thing in the near future.
LLM bad because cannibalizes search ads. Wait as long as possible. OpenAI opens pandora's box. Now full speed ahead; catch up and overtake.
Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models
#83At the risk of being snarky, it's interesting that Llama.cpp was a 'grassroots' effort originating from a Bulgarian hacker google now launches a corporatized effort inspired by it.
I wonder if there's some analogies to the 80s or 90s in here.
Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models
#84Earlier quoted context omitted.
Realistically, if Google has all this talent, they should have gotten the juggernaut moving in 2020. Google has had years to get to this stage, and they've lost a lot of the talent that made their initial big splashes to OAI and competitors. Try finding someone on a sparse MoE paper from Google prior to 2022 who is still working there and not at OAI. With respect, they can hardly even beat Mistral, resorting to round…
Organizational dysfunction can squash/squander even the most talented engineers. Especially in a big org in big tech. My bet is that their inability to deliver before is probably a result of non-comittal funders/decision makers, product whiplash, corporate politics, and other non-technical challenges. Google has been the home of the talent for many years. They came on my radar in the late 00s when I used Peter Norvig…
Right, which was fine for them before there was major competition. But starting in 2020, they have basically attrited most of their talented labor force to OAI and competitors who were not similarly dysfunctional.
Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models
#85Earlier quoted context omitted.
> Although this is from Google, we're a very small team that wanted such a codebase to exist. We have lots of plans to use it ourselves and we hope other people like it and find it useful. This is really cool, Austin. Kudos to your team!
Thanks so much! Everyone working on this self-selected into contributing, so I think of it less as my team than ... a team? Specifically want to call out: Jan Wassenberg (author of https://github.com/google/highway ) and I started gemma.cpp as a small project just a few months ago + Phil Culliton, Dan Zheng, and Paul Chang + of course the GDM Gemma team.
Huge +1, this has definitely been a self-forming collective of people who love great AI, great research, and the open community.
Austin and Jan are truly amazing. The optimization work is genuinely outstanding; I get incredible CPU performance on Gemma.cpp for inference. Thanks for all of the awesomeness, Austin =)
Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models
#86Apart from the fact that they are different things, since they came out of the same organization I think it’s fair to ask:
Do these models have the same kind of odd behavior as Gemini?
Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models
#87Hi, one of the authors austin here. Happy to answer any questions the best I can. To get a few common questions out of the way: - This is separate / independent of llama.cpp / ggml. I'm a big fan of that project and it was an inspiration (we say as much in the README). I've been a big advocate of gguf + llama.cpp support for gemma and am happy for people to use that. - how is it different than inference runtime X? ge…
Hi Austin, what say you about how the Gemma rollout was handled, issues raised, and atmosphere around the office? :)
I'm not Austin, but I am Tris, the friendly neighborhood product person on Gemma. Overall, I think that the main feeling is: incredibly relieved to have had the launch go as smoothly as it has! The complexity of the launch is truly astounding:
1) Reference implementations in JAX, PyTorch, TF with Keras 3, MaxText/JAX, more...
2) Full integration at launch with HF including Transformers + optimization therein
3) TensorRT-LLM and full NVIDIA opt across the stack in partnership with that team (mentioned on the NVIDIA earnings call by Jensen, even)
4) More developer surfaces than you can shake a stick at: Kaggle, Colab, Gemma.cpp, GGUF
5) Comms landing with full coordination from Sundar + Demis + Jeff Dean, not to mention positive articles in NYT, Verge, Fortune, etc.
6) Full Google Cloud launches across several major products, including Vertex and GKE
7) Launched globally and with a permissive set of terms that enable developers to do awesome stuff
Pulling that off without any major SNAFUs is a huge relief for the team. We're excited by the potential of using all of those surfaces and the launch momentum to build a lot more great things for you all =)
Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models
#88At the risk of being snarky, it's interesting that Llama.cpp was a 'grassroots' effort originating from a Bulgarian hacker google now launches a corporatized effort inspired by it. I wonder if there's some analogies to the 80s or 90s in here.
To be clear, this is not comparable directly to llama.cpp -- Gemma models work on llama.cpp and we encourage people who love llama.cpp to use them there. We're also launched with Ollama.
Gemma.cpp is a highly optimized and lightweight system. The performance is pretty incredible on CPU, give it a try =)
Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models
#89Earlier quoted context omitted.
Hi Austin, what say you about how the Gemma rollout was handled, issues raised, and atmosphere around the office? :)
I'm not Austin, but I am Tris, the friendly neighborhood product person on Gemma. Overall, I think that the main feeling is: incredibly relieved to have had the launch go as smoothly as it has! The complexity of the launch is truly astounding: 1) Reference implementations in JAX, PyTorch, TF with Keras 3, MaxText/JAX, more... 2) Full integration at launch with HF including Transformers + optimization therein 3) Tenso…
I am not a fan of a lot of what Google does, but congratulations! That’s a massive undertaking and it is bringing the field forward. I am glad you could do this, and hope you’ll have many other successful releases.
Now, I’m off playing with a new toy :)
Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models
#90Earlier quoted context omitted.
https://ollama.com/library/gemma/tags You can see the various quantizations here, both for the 2B model and the 7B model. The smallest you can go is the q2_K quantization of the 2B model, which is 1.3GB, but I wouldn't really call that "functional". The q4_0 quantization is 1.7GB, and that would probably be functional. The size of anything but the model is going to be rounding error compared to how large the models a…
What's the use case of models this small? Can you use the "knowledge" encoded in them and ask them questions and get relevant answers, or are they used as text processors to summarize documents etc?
Gemma 2B generation quality is excellent in my own very-biased opinion. I asked it to write a response to your comment:
--
Large language models (LLMs) have achieved significant progress in recent years, with models like GPT-3 and LaMDA demonstrating remarkable abilities in various tasks such as language generation, translation, and question answering.
However, 2b parameter models are a much smaller and simpler type of LLM compared to GPT-3. While they are still capable of impressive performance, they have a limited capacity for knowledge representation and reasoning.
Despite their size, 2b parameter models can be useful in certain scenarios where the specific knowledge encoded in the model is relevant to the task at hand. For example:
- Question answering: 2b parameter models can be used to answer questions by leveraging their ability to generate text that is similar to the question.
- Text summarization: 2b parameter models can be used to generate concise summaries of documents by extracting the most important information.
- Code generation: While not as common, 2b parameter models can be used to generate code snippets based on the knowledge they have learned.
Overall, 2b parameter models are a valuable tool for tasks that require specific knowledge or reasoning capabilities. However, for tasks that involve general language understanding and information retrieval, larger LLMs like GPT-3 may be more suitable.
--
Generated in under 1s from query to full response on together.ai