Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemma: New Open Models

blog.google

Post reply on HN
Print view View original

161–170 of 543 posts

Re: Gemma: New Open Models

#161

Post by daemonologist »

jcuenod wrote: post #149
tosh wrote: post #45

Benchmarks for Gemma 7B seem to be in the ballpark of Mistral 7B +-------------+----------+-------------+-------------+ | Benchmark | Gemma 7B | Mistral 7B | Llama-2 7B | +-------------+----------+-------------+-------------+ | MMLU | 64.3 | 60.1 | 45.3 | | HellaSwag | 81.2 | 81.3 | 77.2 | | HumanEval | 32.3 | 30.5 | 12.8 | +-------------+----------+-------------+-------------+ via https://mistral.ai/news/announcing-…

Came here to post the same thing for Phi-2: +-------------+----------+-------------+ | Benchmark | Gemma 2B | Phi-2 2.7B | +-------------+----------+-------------+ | MMLU | 42.3 | 56.7 | | MBPP | 29.2 | 59.1 | | BoolQ | 69.4 | 83.3 | +-------------+----------+-------------+ [0] https://www.kaggle.com/models/google/gemma [1] https://www.microsoft.com/en-us/research/blog/phi-2-the-surp...

Really looking forward to the day someone puts out an open model which outperforms Flan-T5 on BoolQ.

Re: Gemma: New Open Models

#162

Post by lordswork »

alekandreev wrote: post #3

Hello on behalf of the Gemma team! We are really excited to answer any questions you may have about our models. Opinions are our own and not of Google DeepMind.

Is there any truth behind this claim that folks who worked on Gemma have left Google?

https://x.com/yar_vol/status/1760314018575634842

Re: Gemma: New Open Models

#164

Post by anshumankmr »

Are these any good? I have been trying the non pro version of Gemini, and that seems awful at code generation. I am more keen on getting access to the best model and I would pay for it if I wasn't already paying for ChatGPT 4.

Re: Gemma: New Open Models

#165

Post by scarmig »

lawxls wrote: post #159
tosh wrote: post #45

Benchmarks for Gemma 7B seem to be in the ballpark of Mistral 7B +-------------+----------+-------------+-------------+ | Benchmark | Gemma 7B | Mistral 7B | Llama-2 7B | +-------------+----------+-------------+-------------+ | MMLU | 64.3 | 60.1 | 45.3 | | HellaSwag | 81.2 | 81.3 | 77.2 | | HumanEval | 32.3 | 30.5 | 12.8 | +-------------+----------+-------------+-------------+ via https://mistral.ai/news/announcing-…

Honestly, this is more of a PR stunt to advertise the Google Dev ecosystem than a contribution to open-source. I'm not complaining, just calling it what it is. Barely an improvement over the 5-month-old Mistral model, with the same context length of 8k. And this is a release after their announcement of Gemini Pro 1.5, which had an exponential increase in context length.

Who cares if it's a PR stunt to improve developer good will? It's still a good thing, and it's now the most open model out there.

Re: Gemma: New Open Models

#166

Post by mirekrusin »

sa-code wrote: post #156
tosh wrote: post #45

Benchmarks for Gemma 7B seem to be in the ballpark of Mistral 7B +-------------+----------+-------------+-------------+ | Benchmark | Gemma 7B | Mistral 7B | Llama-2 7B | +-------------+----------+-------------+-------------+ | MMLU | 64.3 | 60.1 | 45.3 | | HellaSwag | 81.2 | 81.3 | 77.2 | | HumanEval | 32.3 | 30.5 | 12.8 | +-------------+----------+-------------+-------------+ via https://mistral.ai/news/announcing-…

Thank you. I thought it was weird for them to release a 7B model and not mention Mistral in their release.

They forgot.

Also phi-2.

Re: Gemma: New Open Models

#168

Post by wantsanagent »

The utter bullshit of these licenses has got to stop. Do not, under any circumstances, consider using these commercially.

"Google reserves the right to restrict (remotely or otherwise) usage of any of the Gemma Services that Google reasonably believes are in violation of this Agreement."

This is a kill switch that Google maintains in perpetuity over any system you build relying on these models. Our legal review of the Llama license came to the same conclusion, we cannot rely on the goodwill of Meta for any core service, and we shouldn't rely on the same from Google.

Now, perhaps less materially important, but just as infuriating is the "Prohibited Use[s]". These cover just enough to placate the most sensitive, but omit any real harms (waging war, developing weapons) that coincidentally have massive commercial value. Use the model to build a biological weapon (as an authorized govt official)? Cool. Use it to play a prank that deceives someone? Policy violation.

And of course, as the coup de grâce, they throw in a DMCA style provision to make sure you can't modify the models in any way that could cause them to violate their kid-glove precepts.

Re: Gemma: New Open Models

#170

Post by zemo »

> Open models feature free access to the model weights, but terms of use, redistribution, and variant ownership vary according to a model’s specific terms of use, which may not be based on an open-source license.

does a model being "open" say anything about how it was trained?

Post reply on HN