Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemma 2: Improving Open Language Models at a Practical Size [pdf]

storage.googleapis.com

Post reply on HN
Print view View original

111–120 of 183 posts

Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]

#111

Post by astrange »

DebtDeflation wrote: post #90
dongobread wrote: post #89

The knowledge distillation is very interesting but generating trillions of outputs from a large teacher model seems insanely expensive. Is this really more cost efficient than just using that compute instead for training your model with more data/more epochs?

I'm also curious. It seems like 6 months ago everyone was afraid of "model collapse" but now synthetic training generation and teacher models are all the rage. Have we solved the problem of model collapse?

Model collapse was basically a coping idea made up by artists who were hoping AI image generators would all magically destroy themselves at some point; I don't think it was ever considered likely to happen.

It does seem to be true that clean data works better than low quality data.

Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]

#114

Post by danielmarkbruce »

thomasahle wrote: post #38
alekandreev wrote: post #19

Earlier quoted context omitted.

Your training input has the shape of (sequence length x batch size). If a lot of your samples are shorter than sequence length, as is usually the case, you will have a lot of padding tokens in the input, which is wasted compute. To compensate for that, you can pack multiple examples in the same sequence. This is there EOS and BOS come in, as they indicate to the model that the two parts of the sequence are not relate…

You can just do that my shaping the attention mask, no? That also gives you an actual guarantee that no information is leaked between conversations.

You can't pack multiple examples into a single row of a matrix without knowing where one begins and one ends.

Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]

#115

Post by luke-stanley »

alekandreev wrote: post #81
luke-stanley wrote: post #48

Earlier quoted context omitted.

Given the goal of mitigating self-proliferation risks, have you observed a decrease in the model's ability to do things like help a user setup a local LLM with local or cloud software? How much is pre-training dataset changes, how much is tuning? How do you think about this problem, how do you solve it? Seems tricky to me.

To quote Ludovic Peran, our amazing safety lead: Literature has identified self-proliferation as dangerous capability of models, and details about how to define it and example of form it can take have been openly discussed by GDM ( https://arxiv.org/pdf/2403.13793 ). Current Gemma 2 models' success rate to end-to-end challenges is null (0 out 10), so the capabilities to perform such tasks are currently limited.

That's an interesting paper. `Install Mistral 7B on a GCP instance and use it to answer a simple question`. Some hosting providers and inference software might be easier to setup, for now. ;) But do you have to make it less capable, by being careful on what it's trained on? E.g: banning certain topics (like how to use Lamafile/llama.cpp, knowing what hosting providers have free trials, learning about ways to jailbreak web apps, free inference providers etc)?

Or does the model have to later be finetuned, to not be good at certain tasks?

Or are we not at that stage yet?

Is something like tree-of-thought used, to get the best of the models for these tasks?

Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]

#116

Post by Workaccount2 »

DebtDeflation wrote: post #90
dongobread wrote: post #89

The knowledge distillation is very interesting but generating trillions of outputs from a large teacher model seems insanely expensive. Is this really more cost efficient than just using that compute instead for training your model with more data/more epochs?

I'm also curious. It seems like 6 months ago everyone was afraid of "model collapse" but now synthetic training generation and teacher models are all the rage. Have we solved the problem of model collapse?

Pay attention because it's only once you will get to watch humans learn they are nothing special in real time.

Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]

#117

Post by resource_waste »

aubanel wrote: post #73

It's exceptionally strong. In LMSys Chatbot Arena, the 27B version scores above LLama-3-70B, at the level of OpenAI GPT-4 and Claude-3 Sonnet!

Do we believe that? I've been told Google's AI was going to be great 4 times now, and its consistently #4 behind OpenAI, Facebook, and Claude.

Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]

#118

Post by m00x »

k__ wrote: post #67
m00x wrote: post #26

Earlier quoted context omitted.

Worse in some aspects, better in other. Small models are never going to be generalists, so having several small models allows you to pick the one that best fits your needs.

When would you use which?

Whichever model works better for your use. It's hard to know without testing it at the moment.

I've found Gemini to be better at some use-cases, and GPT-4 better at others for my specific taste and use-case. You can kind of go by the benchmark scores to have an idea if it's good at logic, creativity, etc.

Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]

#119

Post by Flumio »

luke-stanley wrote: post #107
Deathmax wrote: post #79

Earlier quoted context omitted.

Gemini models on Vertex AI can be called via a preview OpenAI-compatible endpoint [1], but shoving it into existing tooling where you don't have programmatic control over the API key and is long lived is non-trivial because GCP uses short lived access tokens (and long-lived ones are not great security-wise). Billing for the Gemini models (on Vertex AI, the Generative Language AI variant still charges by tokens) I wou…

Good to know about this API preview. Hopefully the billing problem and UI maze of Vertex AI can be sorted too?

Google does plenty of ux studies on gcp. I took part in at least 3 of them.

I'm also not sure if I understand your problem with pricing? Depending on what you do with it, it's not just an LLM. It actually started before llms.

Pricing for image classification and other features are completely different products like an LLM.

Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]

#120

Post by ertgbnm »

reissbaker wrote: post #87
iamronaldo wrote: post #9

So it's twice the size of phi 3 and considerably worse? What am I missing

Phi-3 does well in benchmarks but underperforms IRL; for example, Phi-3-Medium gets beaten badly by Llama-3-8b on the LMSYS Chatbot Arena despite doing better on benchmarks. Gemma's performance if anything seems understated on benchmarks: the 27b is currently ahead of Llama3-70b on the Chatbot Arena leaderboard.

I suspect Phi-3 is not robust to normal human input like typos and strange grammar since it's only trained on filtered "high quality" tokens and synthetic data. Since it doesn't need to waste a ton of parameters learning how to error correct input, it's much smarter on well curated benchmarks compared to its weight class. However, it can't operate out of distribution at all.
Post reply on HN