Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

github.com

Post reply on HN
Print view View original

131–140 of 146 posts

Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

#131

Post by brucethemoose2 »

trisfromgoogle wrote: post #92
brucethemoose2 wrote: post #5

...Also, we have eval'd Gemma 7B internally in a deterministic, zero temperature test, and its error rate is like double Mistral Instruct 0.2. Well below most other 7Bs. Was not very impressed with the chat either. So maybe this is neat for embedded projects, but if it's Gemma only, that would be quite a sticking point for me.

Any chance you can share more details on your measurement setup and eval protocols? You're likely seeing some config snafus, which we're trying to track down.

I just loaded it in vllm with default settings.

I can't share the eval, but it's pretty simple: it asks a question about some data, and is restricted to only answer yes/no (based on the output logits and suggested in the prompt). It's called with 0 temperature and only 1 output token, so sampling shouldn't be an issue.

Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

#132

Post by okdood64 »

loudmax wrote: post #115
llm_nerd wrote: post #104

Earlier quoted context omitted.

While I generally agree with you, who has ever counted Google out? We've made fun of Google for lagging while they instead spend their engineering time renaming projects and performing algorithmic white-erasure, but we all knew they're a potent force. Google has as much or more computing power than anyone. They're massively capitalized and have a market cap of almost $2T and colossal cashflow, and have the ability to…

I certainly would not count out Google's engineering talent. But all the technical expertise in the world won't matter when the leadership is incompetent and dysfunctional. Rolling out a new product takes vision, and it means taking some risks. This is diametrically opposed to how Google operates today. Gemini could be years ahead of ChatGPT (and maybe it is now, if it weren't neutered), but Google's current leadersh…

> Google has the technical resources to become a major player here

Wait, it's not a major player in ML/AI?

Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

#134

Post by verticalscaler »

wokwokwok wrote: post #125
verticalscaler wrote: post #117

Earlier quoted context omitted.

[flagged]

Let it go. You’re being flagged and downvoted on your threads because you’re being unreasonable, not because of some moderator level conspiracy. The argument you’re making is not falsifiable. If you want to pursue this line of argument, you need to pause, think about how you can confidently and unambiguously make substantive claims; for example, substantive examples from actually using these models. The way you’re cu…

[dead]

Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

#135

Post by austinvhuang »

namtranase wrote: post #81

Thank the team for the awesome repo. I have navigated gemma.cpp and run it from the first day, it is smooth in my view. So I hope gemma.cpp will continue to add cool features (something like k-quants, server,...) so it can serve more widely. Actually, I have developed a Python wrapper for it: https://github.com/namtranase/gemma-cpp-python The purpose is to use easily and update every new technique from gemma.cpp team…

Nice, this is really cool to see! There were other threads that expressed interest in something like this.

Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

#136

Post by iaseiadit »

refulgentis wrote: post #108
zoogeny wrote: post #71

I know a lot of people chide Google for being behind OpenAI in their commercial offerings. We also dunk on them for the over-protective nature of their fine-tuning. But Google is scarily capable on the LLM front and we shouldn't count them out. OpenAI might have the advantage of being quick to move, but when the juggernaut gets passed its resting inertia and starts to gain momentum it is going to leave an impression.…

There's nothing provided here other than Jeff Dean gave a stock entry-level presentation to students at Rice, therefore "The depth of institutional knowledge that is going to be unlocked inside Google is actually frightening for me to consider." You should see Google's turnover numbers from 4 years ago, much less now. It's been years, it's broken internally, we see the results. Here, we're in awe of 1KLOC of C++ code…

> You should see Google's turnover numbers from 4 years ago, much less now.

High turnover was industry-wide a few years back because pay went through the roof and job hopping was the best way to capture that.

I suspect it’s lower now, following mass layoffs and substantially fewer openings.

Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

#137

Post by Eisenstein »

brikym wrote: post #116
brigadier132 wrote: post #91

Earlier quoted context omitted.

There was a podcast yesterday that explained well why Google is in a tough position. https://youtu.be/-i9AGk3DJ90?t=616 In essence, Google already rules information retrieval. Their margins are insane. Switching to LLM based search cuts into their margins and increases their costs dramatically. Also, the advantage they've built over decades has been cut down. All of this means there is potential for less profit and a…

I’m sure Kodak had the same problem with the digital camera.

They did. They invented and patented the digital camera back in the 70s, refused to improve on it for fear of eating their own market base, and then went bankrupt.

* https://spectrum.ieee.org/first-digital-camera-history

Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

#138

Post by olegbask »

It would be amazing to add support for M1 aka Metal: I was able to run Q8 version with llama.cpp and it's blazingly fast. The problem: I don't know how much accuracy it loses and https://huggingface.co/google/gemma-2b-it/tree/main takes too much memory which results in OOMs.

Do you have any estimates on getting Metal support similar to how llama.cpp works?

Why `.gguf` files are so giant compared to `.sbs`? Is it just because they use fp32?

Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

#139

Post by alekandreev »

dontupvoteme wrote: post #83

At the risk of being snarky, it's interesting that Llama.cpp was a 'grassroots' effort originating from a Bulgarian hacker google now launches a corporatized effort inspired by it. I wonder if there's some analogies to the 80s or 90s in here.

As a fellow Bulgarian from the 80s and 90s myself, and now a part of the Gemma team, I’d say Austin, Jan, and team very much live up to the ethos of hackers I'd meet on BBSes back then. :)

They are driven entirely by their own curiosity and a desire to push computers to the limit. Combined with their admirable low-level programming skills, you get a very solid, fun codebase, that they are sharing with the world.

Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

#140

Post by castles »

austinvhuang wrote: post #32
rgbrgb wrote: post #21

Earlier quoted context omitted.

Thanks for releasing this! What is your use case for this rather than llama.cpp? For the on-device AI stuff I mostly do, llama.cpp is better because of GPU/metal offloading.

llama.cpp is great, if it fit your needs you can use it. I think at this point llama.cpp is effectively a platform that's hardened for production. In its current form, I think of gemma.cpp is more of a direct model implementation (somewhere between the minimalism of llama2.c and the generality of ggml). I tend to think of 3 modes of usage: - hacking on inference internals - there's very little indirection, no IRs, th…

> accepting/rejecting tokens etc.

That sounds interesting

Post reply on HN