Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

github.com

Post reply on HN
Print view View original

71–80 of 146 posts

Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

#71

Post by zoogeny »

I know a lot of people chide Google for being behind OpenAI in their commercial offerings. We also dunk on them for the over-protective nature of their fine-tuning.

But Google is scarily capable on the LLM front and we shouldn't count them out. OpenAI might have the advantage of being quick to move, but when the juggernaut gets passed its resting inertia and starts to gain momentum it is going to leave an impression.

That became clear to me after watching the recent Jeff Dean video [1] which was posted a few days ago. The depth of institutional knowledge that is going to be unlocked inside Google is actually frightening for me to consider.

I hope the continued competition on the open source front, which we can really thank Facebook and Llama for, keeps these behemoths sharing. As OpenAI moves further from its original mission into capitalizing on its technological lead, we have to remember why the original vision they had is important.

So thank you, Google, for this.

1. https://www.youtube.com/watch?v=oSCRZkSQ1CE&ab_channel=RiceK...

Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

#72

Post by austinvhuang »

dankle wrote: post #67
austinvhuang wrote: post #14

Hi, one of the authors austin here. Happy to answer any questions the best I can. To get a few common questions out of the way: - This is separate / independent of llama.cpp / ggml. I'm a big fan of that project and it was an inspiration (we say as much in the README). I've been a big advocate of gguf + llama.cpp support for gemma and am happy for people to use that. - how is it different than inference runtime X? ge…

What's the reason to not integrate with llama.cpp instead of a separate app? In what ways this better than llama.cpp?

On uses, see https://news.ycombinator.com/item?id=39481554#39482302 and on llama.cpp support - https://news.ycombinator.com/item?id=39481554

Gemma support has been added to llama.cpp, and we're more than happy to see people use it there.

Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

#73

Post by whimsicalism »

zoogeny wrote: post #71

I know a lot of people chide Google for being behind OpenAI in their commercial offerings. We also dunk on them for the over-protective nature of their fine-tuning. But Google is scarily capable on the LLM front and we shouldn't count them out. OpenAI might have the advantage of being quick to move, but when the juggernaut gets passed its resting inertia and starts to gain momentum it is going to leave an impression.…

Realistically, if Google has all this talent, they should have gotten the juggernaut moving in 2020.

Google has had years to get to this stage, and they've lost a lot of the talent that made their initial big splashes to OAI and competitors. Try finding someone on a sparse MoE paper from Google prior to 2022 who is still working there and not at OAI.

With respect, they can hardly even beat Mistral, resorting to rounding down a 7.8b model (w/o embeddings) to 7b.

Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

#75

Post by [deleted] »

austinvhuang wrote: post #14

Hi, one of the authors austin here. Happy to answer any questions the best I can. To get a few common questions out of the way: - This is separate / independent of llama.cpp / ggml. I'm a big fan of that project and it was an inspiration (we say as much in the README). I've been a big advocate of gguf + llama.cpp support for gemma and am happy for people to use that. - how is it different than inference runtime X? ge…

[deleted]

Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

#76

Post by dguest »

zoogeny wrote: post #71

I know a lot of people chide Google for being behind OpenAI in their commercial offerings. We also dunk on them for the over-protective nature of their fine-tuning. But Google is scarily capable on the LLM front and we shouldn't count them out. OpenAI might have the advantage of being quick to move, but when the juggernaut gets passed its resting inertia and starts to gain momentum it is going to leave an impression.…

Maybe someone who knows google better can answer my question here: are they behind simply because LLMs are not really their core business? In other words, it wasn't (and still isn't) obvious that LLMs will help them sell add space.

And of course writing that gives me a terrible realization: product placement in LLMs is going to be a very big thing in the near future.

Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

#77

Post by freedomben »

whimsicalism wrote: post #73
zoogeny wrote: post #71

I know a lot of people chide Google for being behind OpenAI in their commercial offerings. We also dunk on them for the over-protective nature of their fine-tuning. But Google is scarily capable on the LLM front and we shouldn't count them out. OpenAI might have the advantage of being quick to move, but when the juggernaut gets passed its resting inertia and starts to gain momentum it is going to leave an impression.…

Realistically, if Google has all this talent, they should have gotten the juggernaut moving in 2020. Google has had years to get to this stage, and they've lost a lot of the talent that made their initial big splashes to OAI and competitors. Try finding someone on a sparse MoE paper from Google prior to 2022 who is still working there and not at OAI. With respect, they can hardly even beat Mistral, resorting to round…

Organizational dysfunction can squash/squander even the most talented engineers. Especially in a big org in big tech. My bet is that their inability to deliver before is probably a result of non-comittal funders/decision makers, product whiplash, corporate politics, and other non-technical challenges.

Google has been the home of the talent for many years. They came on my radar in the late 00s when I used Peter Norvig's textbook in college, and they hired Ray Kurzweil in like 2012 or 2013 IIRC. They were hiring ML PhDs with talent for many years, and they pioneered most of the major innovations. They just got behind on productizing and shipping.

Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

#78

Post by freedomben »

dguest wrote: post #76
zoogeny wrote: post #71

I know a lot of people chide Google for being behind OpenAI in their commercial offerings. We also dunk on them for the over-protective nature of their fine-tuning. But Google is scarily capable on the LLM front and we shouldn't count them out. OpenAI might have the advantage of being quick to move, but when the juggernaut gets passed its resting inertia and starts to gain momentum it is going to leave an impression.…

Maybe someone who knows google better can answer my question here: are they behind simply because LLMs are not really their core business? In other words, it wasn't (and still isn't) obvious that LLMs will help them sell add space. And of course writing that gives me a terrible realization: product placement in LLMs is going to be a very big thing in the near future.

I'm an outsider and am speculating based on what I've heard, so maybe I shouldn't even comment, but to me it seems like it's been entirely corporate/organizational reasons. Non-serious funding, shifting priorities, personnel transfers/fluctuations, internal fragmentation, and more. Lack of talent has never been their problem.

Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

#79

Post by freedomben »

austinvhuang wrote: post #72
dankle wrote: post #67

Earlier quoted context omitted.

What's the reason to not integrate with llama.cpp instead of a separate app? In what ways this better than llama.cpp?

On uses, see https://news.ycombinator.com/item?id=39481554#39482302 and on llama.cpp support - https://news.ycombinator.com/item?id=39481554 Gemma support has been added to llama.cpp, and we're more than happy to see people use it there.

I think on uses you meant to link to https://news.ycombinator.com/item?id=39482581 child of https://news.ycombinator.com/item?id=39481554#39482302 ?

side note: imagine how gnarly those urls would be if HN used UUIDs instead of integers for IDs :-D

Post reply on HN