Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemma 2: Improving Open Language Models at a Practical Size [pdf]

storage.googleapis.com

Post reply on HN
Print view View original

81–90 of 183 posts

Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]

#81

Post by alekandreev »

luke-stanley wrote: post #48
alekandreev wrote: post #6

Hello (again) from the Gemma team! We are quite excited to push this release out and happy to answer any questions! Opinions are our own and not of Google DeepMind.

Given the goal of mitigating self-proliferation risks, have you observed a decrease in the model's ability to do things like help a user setup a local LLM with local or cloud software? How much is pre-training dataset changes, how much is tuning? How do you think about this problem, how do you solve it? Seems tricky to me.

To quote Ludovic Peran, our amazing safety lead:

Literature has identified self-proliferation as dangerous capability of models, and details about how to define it and example of form it can take have been openly discussed by GDM (https://arxiv.org/pdf/2403.13793).

Current Gemma 2 models' success rate to end-to-end challenges is null (0 out 10), so the capabilities to perform such tasks are currently limited.

Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]

#82

Post by WhitneyLand »

alekandreev wrote: post #6

Hello (again) from the Gemma team! We are quite excited to push this release out and happy to answer any questions! Opinions are our own and not of Google DeepMind.

The paper suggests on one hand Gemma is on the same Pareto curve as Llama3, while on the other hand seems to suggest it’s exceeded its efficiency.

Is this a contradiction or am I misunderstanding something?

Btw overall very impressive work great job.

Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]

#83

Post by janwas »

austinvhuang wrote: post #55
jpcapdevila wrote: post #46

Earlier quoted context omitted.

Will gemma2 be available through gemma.cpp? https://github.com/google/gemma.cpp

This is in the works in the dev branch (thanks pchx :) https://github.com/google/gemma.cpp/pull/274

:) Confirmed working. We've just pushed the dev branch to main.

Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]

#84

Post by popalchemist »

llm_trw wrote: post #66
kkkkkkk wrote: post #49

Earlier quoted context omitted.

If this were a common sentiment or rooted in reality I would imagine their stock would not be at an all time high...

I'm an early adopter. The rest of you will catch up in the next five years.

Here's a napkin for when you're finished.

Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]

#85

Post by imjonse »

refulgentis wrote: post #56
alecco wrote: post #5

Shouldn't this (2.6B/9B) be compared with Microsoft's Phi-3 mini (3.8B) instead of Mistral and Llama-3? (table 13 on page 7) vs https://arxiv.org/pdf/2404.14219 (page 6, quite better in general) The report on knowledge distillation training is interesting, though.

Picking up from there: The games in this paper and model are annoying. The 2.6B would get stomped by Phi-3, so there's no comparison. Fair enough. 2.6B vs. 3.8B is a fairly substantial size difference thats hard to intuit when its 2.6 vs 3.8 versus 2,600,000,000 and 3,800,000,000. But then we get what I'm going to "parameter creep": Mistral 7B vs. Llama 8B vs. Gemma 9B. I worried after Llama 3 went 8B that we'd start…

In the Llama 3 case I think the increase in parameters is mostly due to the input embeddings and output logits layers, reflecting the context size increase.

Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]

#86

Post by screye »

aubanel wrote: post #73

It's exceptionally strong. In LMSys Chatbot Arena, the 27B version scores above LLama-3-70B, at the level of OpenAI GPT-4 and Claude-3 Sonnet!

What's the most obvious standouts?

In my experience, smaller models tend to do well on benchmarks and fail at generalization. Phi-2 comes to mind.

Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]

#87

Post by reissbaker »

iamronaldo wrote: post #9

So it's twice the size of phi 3 and considerably worse? What am I missing

Phi-3 does well in benchmarks but underperforms IRL; for example, Phi-3-Medium gets beaten badly by Llama-3-8b on the LMSYS Chatbot Arena despite doing better on benchmarks.

Gemma's performance if anything seems understated on benchmarks: the 27b is currently ahead of Llama3-70b on the Chatbot Arena leaderboard.

Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]

#88

Post by Aerbil313 »

k__ wrote: post #67
m00x wrote: post #26

Earlier quoted context omitted.

Worse in some aspects, better in other. Small models are never going to be generalists, so having several small models allows you to pick the one that best fits your needs.

When would you use which?

Obviously another small model would be specialized in determining that.

Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]

#89

Post by dongobread »

The knowledge distillation is very interesting but generating trillions of outputs from a large teacher model seems insanely expensive. Is this really more cost efficient than just using that compute instead for training your model with more data/more epochs?

Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]

#90

Post by DebtDeflation »

dongobread wrote: post #89

The knowledge distillation is very interesting but generating trillions of outputs from a large teacher model seems insanely expensive. Is this really more cost efficient than just using that compute instead for training your model with more data/more epochs?

I'm also curious. It seems like 6 months ago everyone was afraid of "model collapse" but now synthetic training generation and teacher models are all the rage. Have we solved the problem of model collapse?
Post reply on HN