Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

github.com

Post reply on HN
Print view View original

111–120 of 146 posts

Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

#111

Post by moffkalast »

Gracana wrote: post #94
moffkalast wrote: post #64

Earlier quoted context omitted.

It's all mostly just inference, though some train LoRAs directly on quantized models too. GGML and GGUF are the same thing, GGUF is the new version that adds more data about the model so it's easy to support multiple architectures, and also includes prompt templates. These can run CPU only, be partially or fully offloaded to a GPU. With K quants, you can get anywhere from a 2 bit to an 8 bit GGUF. GPTQ was the GPU-on…

> and also includes prompt templates That sounds very convenient. What software makes use of the built-in prompt template?

Of the ones I commonly use, I've only seen it read by text-generation-webui, in the GGML days it had a long hardcoded list of known models and which templates they use so they could be auto-selected (which was often wrong), but now it just grabs it from any model directly and sets it when it's loaded.

Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

#112

Post by trisfromgoogle »

verticalscaler wrote: post #103
trisfromgoogle wrote: post #100

Earlier quoted context omitted.

Neither of those applies at all to Gemma, though? I'm still confused -- what are you trying to accomplish with this line of questioning?

[flagged]

I've been completely honest, human-like, and non-evasive with you. I answered your questions directly and frankly.

Every time, you ignored the honest and human-like answers to try and score some imaginary points.

We're honestly trying our best to build open models *with* the community that you can tune and use to build neat AI research + products. Ignoring that in favor of some political narrative is really petty.

Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

#113

Post by brokensegue »

notum wrote: post #109
brokensegue wrote: post #54

Earlier quoted context omitted.

I'm just looking for ballpark figures. Maybe a common aws instance type

Not sure if this is of any value to you, but Ryzen 7 generates 2 tokens per second for the 7B-Instruct model. The model itself is very unimpressive and I see no reason to play with it over the worst alternative from Hugging Face. I can only imagine this was released for some bizarre compliance reasons.

the metrics suggest it's much better than that

Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

#115

Post by loudmax »

llm_nerd wrote: post #104
zoogeny wrote: post #71

I know a lot of people chide Google for being behind OpenAI in their commercial offerings. We also dunk on them for the over-protective nature of their fine-tuning. But Google is scarily capable on the LLM front and we shouldn't count them out. OpenAI might have the advantage of being quick to move, but when the juggernaut gets passed its resting inertia and starts to gain momentum it is going to leave an impression.…

While I generally agree with you, who has ever counted Google out? We've made fun of Google for lagging while they instead spend their engineering time renaming projects and performing algorithmic white-erasure, but we all knew they're a potent force. Google has as much or more computing power than anyone. They're massively capitalized and have a market cap of almost $2T and colossal cashflow, and have the ability to…

I certainly would not count out Google's engineering talent. But all the technical expertise in the world won't matter when the leadership is incompetent and dysfunctional. Rolling out a new product takes vision, and it means taking some risks. This is diametrically opposed to how Google operates today. Gemini could be years ahead of ChatGPT (and maybe it is now, if it weren't neutered), but Google's current leadership would have no idea what to do with it.

Google has the technical resources to become a major player here, maybe even the dominant player. But it won't happen under current management. I won't count out Google entirely, and there's still time for the company to be saved. It starts with new leadership.

Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

#116

Post by brikym »

brigadier132 wrote: post #91
zoogeny wrote: post #71

I know a lot of people chide Google for being behind OpenAI in their commercial offerings. We also dunk on them for the over-protective nature of their fine-tuning. But Google is scarily capable on the LLM front and we shouldn't count them out. OpenAI might have the advantage of being quick to move, but when the juggernaut gets passed its resting inertia and starts to gain momentum it is going to leave an impression.…

There was a podcast yesterday that explained well why Google is in a tough position. https://youtu.be/-i9AGk3DJ90?t=616 In essence, Google already rules information retrieval. Their margins are insane. Switching to LLM based search cuts into their margins and increases their costs dramatically. Also, the advantage they've built over decades has been cut down. All of this means there is potential for less profit and a…

I’m sure Kodak had the same problem with the digital camera.

Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

#117

Post by verticalscaler »

trisfromgoogle wrote: post #112
verticalscaler wrote: post #103

Earlier quoted context omitted.

[flagged]

I've been completely honest, human-like, and non-evasive with you. I answered your questions directly and frankly. Every time, you ignored the honest and human-like answers to try and score some imaginary points. We're honestly trying our best to build open models *with* the community that you can tune and use to build neat AI research + products. Ignoring that in favor of some political narrative is really petty.

[flagged]

Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

#118

Post by SquareWheel »

trisfromgoogle wrote: post #112
verticalscaler wrote: post #103

Earlier quoted context omitted.

[flagged]

I've been completely honest, human-like, and non-evasive with you. I answered your questions directly and frankly. Every time, you ignored the honest and human-like answers to try and score some imaginary points. We're honestly trying our best to build open models *with* the community that you can tune and use to build neat AI research + products. Ignoring that in favor of some political narrative is really petty.

That user is clearly not working with the principle of charity in mind. I've flagged their comments, and would suggest disengaging with them.

Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

#119

Post by staticman2 »

xrd wrote: post #102

I was discussing LLMs with a non technical person on the plane yesterday. I was explaining why LLMs aren't good at math. And, he responded, no, chatgpt is great a multivariate regression, etc. I'm using LLMs locally almost always and eschewing API backed LLMs like chatgpt. So I'm not very familiar with plugins, and I'm assuming chatgpt plugs into a backend when it detects a math problem. So it isn't the LLM doing the…

Sillytavern is a front end for local and cloud models. They have a simple scripting language and there's been some experiments with adding game functionality with it:

https://chub.ai/characters/creamsan/team-neko-e4f1b2f8

This one says it uses javascript as well:

https://chub.ai/characters/creamsan/tessa-c4b917f9

Thise are the only two listed as SFW. There's some others if you hit the nsfw toggle and search for the scripted tag.I don't know if this is the right approach but you could also write a module for Sillytavern Extras.

Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

#120

Post by Permik »

trisfromgoogle wrote: post #97
verticalscaler wrote: post #96

Earlier quoted context omitted.

[flagged]

I mean, many articles will have a negative cast because of the need for clicks -- e.g., the Verge's launch article is entitled "Google Gemma: because Google doesn’t want to give away Gemini yet" -- which I think is both an unfair characterization (given the free tier of Gemini Pro) and unnecessarily inflammatory. Legitimate criticisms include not working correctly out of the box for llama.cpp due to repetition penalt…

I thought that reflecting what went poorly is really informative and inspiring! It really shows how you are taking this seriously, and iterating on a great project that you're building better and better, thanks for the insight and for a peek behind the curtain
Post reply on HN