Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

github.com

Post reply on HN
Print view View original

101–110 of 146 posts

Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

#101

Post by summerlight »

verticalscaler wrote: post #98
trisfromgoogle wrote: post #97

Earlier quoted context omitted.

I mean, many articles will have a negative cast because of the need for clicks -- e.g., the Verge's launch article is entitled "Google Gemma: because Google doesn’t want to give away Gemini yet" -- which I think is both an unfair characterization (given the free tier of Gemini Pro) and unnecessarily inflammatory. Legitimate criticisms include not working correctly out of the box for llama.cpp due to repetition penalt…

[flagged]

It looks like you're trying to get some sort of "confession" from relevant people based on recent memes against the company? The reality is likely that the developers sincerely believe in the value of this product and are proud of its launch. You're just adding uninteresting, irrelevant noise to the discussion and you probably won't get what you want.

Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

#102

Post by xrd »

I was discussing LLMs with a non technical person on the plane yesterday. I was explaining why LLMs aren't good at math. And, he responded, no, chatgpt is great a multivariate regression, etc.

I'm using LLMs locally almost always and eschewing API backed LLMs like chatgpt. So I'm not very familiar with plugins, and I'm assuming chatgpt plugs into a backend when it detects a math problem. So it isn't the LLM doing the math but to the user it appears to be.

Does anyone here know what LLM projects like llama.cpp or gemma.cpp support a plugin model?

I'm interested in adding to the dungeons and dragons system I built using llama.cpp. Because it doesn't do math well, the combat mode is terrible. But I was writing my own layer to break out when combat mode occurs, and I'm wondering if there is a better way with some kind of plugin approach.

Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

#103

Post by verticalscaler »

trisfromgoogle wrote: post #100
verticalscaler wrote: post #98

Earlier quoted context omitted.

[flagged]

Neither of those applies at all to Gemma, though? I'm still confused -- what are you trying to accomplish with this line of questioning?

[flagged]

Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

#104

Post by llm_nerd »

zoogeny wrote: post #71

I know a lot of people chide Google for being behind OpenAI in their commercial offerings. We also dunk on them for the over-protective nature of their fine-tuning. But Google is scarily capable on the LLM front and we shouldn't count them out. OpenAI might have the advantage of being quick to move, but when the juggernaut gets passed its resting inertia and starts to gain momentum it is going to leave an impression.…

While I generally agree with you, who has ever counted Google out? We've made fun of Google for lagging while they instead spend their engineering time renaming projects and performing algorithmic white-erasure, but we all knew they're a potent force.

Google has as much or more computing power than anyone. They're massively capitalized and have a market cap of almost $2T and colossal cashflow, and have the ability to throw enormous resources at the problem until they have a competitor. They have an enormous, benchmark-setting amount of data across their various projects to train on. That we're talking like they're some scrappy upstart is super weird.

>As OpenAI moves further from its original mission into capitalizing on its technological lead, we have to remember why the original vision they had is important.

I'm way more cynical about the open source models released by the megas, and OpenAI is probably the most honest about their intentions. Meta and Google are releasing these models arguably to kneecap any possible next OpenAI. They want to basically set the market value of anything below state of the art at $0.00, ensuring that there is no breathing room below the $2T cos. These models (Llama, Gemma, etc) are fun toys, but in the end they're completely uncompetitive and will yield zero "wins", so to speak.

Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

#105

Post by jerpint »

llm_nerd wrote: post #104
zoogeny wrote: post #71

I know a lot of people chide Google for being behind OpenAI in their commercial offerings. We also dunk on them for the over-protective nature of their fine-tuning. But Google is scarily capable on the LLM front and we shouldn't count them out. OpenAI might have the advantage of being quick to move, but when the juggernaut gets passed its resting inertia and starts to gain momentum it is going to leave an impression.…

While I generally agree with you, who has ever counted Google out? We've made fun of Google for lagging while they instead spend their engineering time renaming projects and performing algorithmic white-erasure, but we all knew they're a potent force. Google has as much or more computing power than anyone. They're massively capitalized and have a market cap of almost $2T and colossal cashflow, and have the ability to…

> Meta and Google are releasing these models arguably to kneecap any possible next OpenAI. They want to basically set the market value of anything below state of the art at $0.00, ensuring that there is no breathing room below the $2T cos

Never thought about it that way, but it makes a lot of sense. It’s also true these models are not up to par with SOTA no matter what the benchmarks say

Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

#106

Post by [deleted] »

verticalscaler wrote: post #62
austinvhuang wrote: post #14

Hi, one of the authors austin here. Happy to answer any questions the best I can. To get a few common questions out of the way: - This is separate / independent of llama.cpp / ggml. I'm a big fan of that project and it was an inspiration (we say as much in the README). I've been a big advocate of gguf + llama.cpp support for gemma and am happy for people to use that. - how is it different than inference runtime X? ge…

Hi Austin, what say you about how the Gemma rollout was handled, issues raised, and atmosphere around the office? :)

[deleted]

Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

#107

Post by corysama »

brigadier132 wrote: post #91
zoogeny wrote: post #71

I know a lot of people chide Google for being behind OpenAI in their commercial offerings. We also dunk on them for the over-protective nature of their fine-tuning. But Google is scarily capable on the LLM front and we shouldn't count them out. OpenAI might have the advantage of being quick to move, but when the juggernaut gets passed its resting inertia and starts to gain momentum it is going to leave an impression.…

There was a podcast yesterday that explained well why Google is in a tough position. https://youtu.be/-i9AGk3DJ90?t=616 In essence, Google already rules information retrieval. Their margins are insane. Switching to LLM based search cuts into their margins and increases their costs dramatically. Also, the advantage they've built over decades has been cut down. All of this means there is potential for less profit and a…

The Innovator's Dilemma over and over again.

Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

#108

Post by refulgentis »

zoogeny wrote: post #71

I know a lot of people chide Google for being behind OpenAI in their commercial offerings. We also dunk on them for the over-protective nature of their fine-tuning. But Google is scarily capable on the LLM front and we shouldn't count them out. OpenAI might have the advantage of being quick to move, but when the juggernaut gets passed its resting inertia and starts to gain momentum it is going to leave an impression.…

There's nothing provided here other than Jeff Dean gave a stock entry-level presentation to students at Rice, therefore "The depth of institutional knowledge that is going to be unlocked inside Google is actually frightening for me to consider."

You should see Google's turnover numbers from 4 years ago, much less now.

It's been years, it's broken internally, we see the results.

Here, we're in awe of 1KLOC of C++ code that runs inference on the CPU.

Nobody serious is running inference on CPU unless you're on the extreme cutting edge. (ex. I need to on Android and on the Chrome OS Linux VM, but I still use llama.cpp because it does support GPU everywhere else)

I'm not sure what else to say.

(n.b. i am a xoogler)

Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

#109

Post by notum »

brokensegue wrote: post #54
austinvhuang wrote: post #51

Earlier quoted context omitted.

any particular hardware folks are most interested in?

I'm just looking for ballpark figures. Maybe a common aws instance type

Not sure if this is of any value to you, but Ryzen 7 generates 2 tokens per second for the 7B-Instruct model.

The model itself is very unimpressive and I see no reason to play with it over the worst alternative from Hugging Face. I can only imagine this was released for some bizarre compliance reasons.

Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

#110

Post by jph00 »

verticalscaler wrote: post #98
trisfromgoogle wrote: post #97

Earlier quoted context omitted.

I mean, many articles will have a negative cast because of the need for clicks -- e.g., the Verge's launch article is entitled "Google Gemma: because Google doesn’t want to give away Gemini yet" -- which I think is both an unfair characterization (given the free tier of Gemini Pro) and unnecessarily inflammatory. Legitimate criticisms include not working correctly out of the box for llama.cpp due to repetition penalt…

[flagged]

These comments appear to be about Gemini's image generation, IIUC. Gemma, however, is a language model -- whilst I believe that a larger unreleased version of it is used as part of the Gemini product, it doesn't seem relevant to these criticisms. Also, the Gemma base model is released, which doesn't AFAIK contain any RLHF.

The impression I have is that you're using the release of Gemma to complain about tangentially related issues about Google and politics more generally. The HN guidelines warn against this: "Eschew flamebait. Avoid generic tangents... Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something. Please don't use Hacker News for political or ideological battle."

Post reply on HN