Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemma: New Open Models

blog.google

Post reply on HN
Print view View original

201–210 of 543 posts

Re: Gemma: New Open Models

#201

Post by ben_w »

politician wrote: post #83
margorczynski wrote: post #37

Is there a chance we'll get a model without the "aligment" (lobotomization)? There are many examples where answers from Gemini are garbage because of the ideological fine tuning.

More useful would be a precise characterization of the type and balance of the ideological fine tuning. They include performance benchmarks. End-users should also be aware of what thoughts are permitted in these constructs. Why omit this information?

> End-users should also be aware of what thoughts are permitted in these constructs. Why omit this information?

Can you define that in a way that's actually testable? I can't, and I've been thinking about "unthinkable thoughts" for quite some time now: https://kitsunesoftware.wordpress.com/2018/06/26/unlearnable...

Re: Gemma: New Open Models

#203

Post by nerdix »

Kostic wrote: post #48
sbarre wrote: post #39

Earlier quoted context omitted.

Can the Gemma models be downloaded to run locally, like open-source models Llama2, Mistral, etc ? Or is your definition of "open" different?

It should be possible to run it via llama.cpp[0] now. [0] https://github.com/ggerganov/llama.cpp/pull/5631

Amazing how quickly this happened.

Re: Gemma: New Open Models

#204

Post by wantsanagent »

tomp wrote: post #97
sbarre wrote: post #39

Earlier quoted context omitted.

Can the Gemma models be downloaded to run locally, like open-source models Llama2, Mistral, etc ? Or is your definition of "open" different?

Their definition of "open" is "not open", i.e. you're only allowed to use Gemma in "non-harmful" way. We all know that Google thinks that saying that 1800s English kings were white is "harmful".

Not sure why you're getting downvoted. I would have thought HN of all places would recognize the power and value of OSI licensing and the danger of the proliferation of these source available but definitely not Open Source licenses.

Re: Gemma: New Open Models

#205

Post by moffkalast »

alekandreev wrote: post #3

Hello on behalf of the Gemma team! We are really excited to answer any questions you may have about our models. Opinions are our own and not of Google DeepMind.

I'm not sure if this was mentioned in the paper somewhere, but how much does the super large 265k tokenizer vocabulary influence inference speed and how much higher is the average text compression compared to llama's usual 30k? In short, is it really worth going beyond GPT 4's 100k?

Re: Gemma: New Open Models

#206

Post by tveita »

austinvhuang wrote: post #145
nalzok wrote: post #28

Congratulations on the release! How can we download the model and run inference locally?

You can download the model checkpoints from kaggle https://www.kaggle.com/models/google/gemma and huggingface https://huggingface.co/blog/gemma Besides the python implementations, we also implemented a standalone C++ implementation that runs locally with just CPU simd https://github.com/google/gemma.cpp

Are there any cool highlights you can give us about gemma.cpp? Does it have any technical advantages over llama.cpp? It looks like it introduces its own quantization format, is there a speed or accuracy gain over llama.cpp's 8-bit quantization?

Re: Gemma: New Open Models

#209

Post by rfw300 »

jcuenod wrote: post #149
tosh wrote: post #45

Benchmarks for Gemma 7B seem to be in the ballpark of Mistral 7B +-------------+----------+-------------+-------------+ | Benchmark | Gemma 7B | Mistral 7B | Llama-2 7B | +-------------+----------+-------------+-------------+ | MMLU | 64.3 | 60.1 | 45.3 | | HellaSwag | 81.2 | 81.3 | 77.2 | | HumanEval | 32.3 | 30.5 | 12.8 | +-------------+----------+-------------+-------------+ via https://mistral.ai/news/announcing-…

Came here to post the same thing for Phi-2: +-------------+----------+-------------+ | Benchmark | Gemma 2B | Phi-2 2.7B | +-------------+----------+-------------+ | MMLU | 42.3 | 56.7 | | MBPP | 29.2 | 59.1 | | BoolQ | 69.4 | 83.3 | +-------------+----------+-------------+ [0] https://www.kaggle.com/models/google/gemma [1] https://www.microsoft.com/en-us/research/blog/phi-2-the-surp...

A caveat: my impression of Phi-2, based on my own use and others’ experiences online, is that these benchmarks do not remotely resemble reality. The model is a paper tiger that is unable to perform almost any real-world task because it’s been fed so heavily with almost exclusively synthetic data targeted towards improving benchmark performance.

Re: Gemma: New Open Models

#210

Post by behnamoh »

DJHenk wrote: post #121
impulser_ wrote: post #27

Go back 5 years and ask anyone on this site what companies do you think will be the most open about AI in the future OpenAI, Meta, or Google. I bet 10/10 people would pick OpenAI. Now today Meta and Google, both trillion dollars companies, are releasing very powerful open models with the ability to be used commercially. Ironic.

> Ironic. Not at all. When you're the underdog, it makes perfect sense to be open because you can profit from the work of the community and gain market share. Only after establishing some kind of dominance or monopoly it makes sense (profit wise) to switch to closed technology. OpenAI was open, but is now the leader and closed up. Meta and Google need to play catch up, so they are open.

This. MistralAI is also underdog and released Mitral 7b and Mixtral 8x7b, but as soon as they got traction, they closed their models (e.g., Mistral Medium).
Post reply on HN