Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

github.com

Post reply on HN
Print view View original

121–130 of 146 posts

Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

#121

Post by cornel_io »

trisfromgoogle wrote: post #97
verticalscaler wrote: post #96

Earlier quoted context omitted.

[flagged]

I mean, many articles will have a negative cast because of the need for clicks -- e.g., the Verge's launch article is entitled "Google Gemma: because Google doesn’t want to give away Gemini yet" -- which I think is both an unfair characterization (given the free tier of Gemini Pro) and unnecessarily inflammatory. Legitimate criticisms include not working correctly out of the box for llama.cpp due to repetition penalt…

I think he's trying to bring up the racial image gen bias stuff that's going on with Gemini, but for some reason won't say it. He also doesn't appear to realize there is a difference between the two products/teams/launches...

Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

#122

Post by jart »

austinvhuang wrote: post #24
dartharva wrote: post #22

Earlier quoted context omitted.

So... llamafile release? https://github.com/Mozilla-Ocho/llamafile

gguf files are out there, so anyone should be able to do this! are people looking for an "official" version? ps i'm a fan of cosmopolitan as well.

Cosmopolitan is a fan of you :-) great work on gemma.cpp. I'm really impressed with it so far.

Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

#123

Post by jart »

trisfromgoogle wrote: post #97
verticalscaler wrote: post #96

Earlier quoted context omitted.

[flagged]

I mean, many articles will have a negative cast because of the need for clicks -- e.g., the Verge's launch article is entitled "Google Gemma: because Google doesn’t want to give away Gemini yet" -- which I think is both an unfair characterization (given the free tier of Gemini Pro) and unnecessarily inflammatory. Legitimate criticisms include not working correctly out of the box for llama.cpp due to repetition penalt…

> Legitimate criticisms include not working correctly out of the box for llama.cpp

I don't think that's a legitimate criticism. Especially not for something that just launched. You should be helping your own project first, before you help others. For example, it'd be nice to see AVX2 work as well as AVX512 with gemma.cpp.

Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

#124

Post by whitten »

refulgentis wrote: post #108
zoogeny wrote: post #71

I know a lot of people chide Google for being behind OpenAI in their commercial offerings. We also dunk on them for the over-protective nature of their fine-tuning. But Google is scarily capable on the LLM front and we shouldn't count them out. OpenAI might have the advantage of being quick to move, but when the juggernaut gets passed its resting inertia and starts to gain momentum it is going to leave an impression.…

There's nothing provided here other than Jeff Dean gave a stock entry-level presentation to students at Rice, therefore "The depth of institutional knowledge that is going to be unlocked inside Google is actually frightening for me to consider." You should see Google's turnover numbers from 4 years ago, much less now. It's been years, it's broken internally, we see the results. Here, we're in awe of 1KLOC of C++ code…

Are there any online transcripts or recordings of the Rice presentation from Jeff Dean ?

Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

#125

Post by wokwokwok »

verticalscaler wrote: post #117
trisfromgoogle wrote: post #112

Earlier quoted context omitted.

I've been completely honest, human-like, and non-evasive with you. I answered your questions directly and frankly. Every time, you ignored the honest and human-like answers to try and score some imaginary points. We're honestly trying our best to build open models *with* the community that you can tune and use to build neat AI research + products. Ignoring that in favor of some political narrative is really petty.

[flagged]

Let it go.

You’re being flagged and downvoted on your threads because you’re being unreasonable, not because of some moderator level conspiracy.

The argument you’re making is not falsifiable.

If you want to pursue this line of argument, you need to pause, think about how you can confidently and unambiguously make substantive claims; for example, substantive examples from actually using these models.

The way you’re currently arguing is not meaningful or compelling.

Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

#126

Post by [deleted] »

jph00 wrote: post #110
verticalscaler wrote: post #98

Earlier quoted context omitted.

[flagged]

These comments appear to be about Gemini's image generation, IIUC. Gemma, however, is a language model -- whilst I believe that a larger unreleased version of it is used as part of the Gemini product, it doesn't seem relevant to these criticisms. Also, the Gemma base model is released, which doesn't AFAIK contain any RLHF. The impression I have is that you're using the release of Gemma to complain about tangentially…

[deleted]

Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

#127

Post by refulgentis »

whitten wrote: post #124
refulgentis wrote: post #108

Earlier quoted context omitted.

There's nothing provided here other than Jeff Dean gave a stock entry-level presentation to students at Rice, therefore "The depth of institutional knowledge that is going to be unlocked inside Google is actually frightening for me to consider." You should see Google's turnover numbers from 4 years ago, much less now. It's been years, it's broken internally, we see the results. Here, we're in awe of 1KLOC of C++ code…

Are there any online transcripts or recordings of the Rice presentation from Jeff Dean ?

Yes, footer of parent: https://youtu.be/oSCRZkSQ1CE?si=Na1d1cK3TApDhkSO

Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

#129

Post by janwas »

refulgentis wrote: post #108
zoogeny wrote: post #71

I know a lot of people chide Google for being behind OpenAI in their commercial offerings. We also dunk on them for the over-protective nature of their fine-tuning. But Google is scarily capable on the LLM front and we shouldn't count them out. OpenAI might have the advantage of being quick to move, but when the juggernaut gets passed its resting inertia and starts to gain momentum it is going to leave an impression.…

There's nothing provided here other than Jeff Dean gave a stock entry-level presentation to students at Rice, therefore "The depth of institutional knowledge that is going to be unlocked inside Google is actually frightening for me to consider." You should see Google's turnover numbers from 4 years ago, much less now. It's been years, it's broken internally, we see the results. Here, we're in awe of 1KLOC of C++ code…

We understand that some teams prefer to use CPU even when mobile GPU would be available.

This code is also intended to facilitate research & experimentation, which may not fall under your definition of 'serious' :)

Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

#130

Post by janwas »

brokensegue wrote: post #54
austinvhuang wrote: post #51

Earlier quoted context omitted.

any particular hardware folks are most interested in?

I'm just looking for ballpark figures. Maybe a common aws instance type

For the 7B IT and a short factual query I see 5.3 tps on a 5 year old Skylake Gold 6154 CPU @ 3.00GHz, 16 threads. Expect a slight increase as we improve scalability.

FYI using the NUQ (4.5-bit) quantization improves throughput by about 1.4x.

Post reply on HN