Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemma 3 Technical Report [pdf]

storage.googleapis.com

Post reply on HN
Print view View original

161–170 of 260 posts

Re: Gemma 3 Technical Report [pdf]

#161

Post by tucnak »

alekandreev wrote: post #149
moffkalast wrote: post #132

Earlier quoted context omitted.

What's the official take on the system prompt? The technical report doesn't mention it, but the official QAT GGUFs include some form of prepending it to the first user message. Has it been trained with any system turns with tool calls and such?

We recommend using user for the system prompt as well.

I was under the impression that the purpose of "system" prompt is to encode the instruction boundary explicitly to reduce the risk of injection. Do you enforce some kind of security invariant that we could rely on? For example, does the alignment regiment include adversarial demonstrations so that out-of-order instruction-following (such as contradicting preceding) is penalised?

Re: Gemma 3 Technical Report [pdf]

#162

Post by newfocogi »

hnuser123456 wrote: post #160
vessenes wrote: post #85

Lots to be excited about here - in particular new architecture that allows subquadratic scaling of memory needs for long context; looks like 128k+ context is officially now available on a local model. The charts make it look like if you have the RAM the model is pretty good out to 350k or so(!) with RoPE. In addition, it flavor tests well on chat arena, ELO significantly above yesterday’s best open model, Qwen 2.5 72…

Gemma is made by Google, not DeepMind. edit: Sorry, forgot DeepMind was Google's AI R&D, I read it as deepseek in your comment.

Job postings for working on Gemma are under DeepMind in London: https://boards.greenhouse.io/deepmind/jobs/6590957

Re: Gemma 3 Technical Report [pdf]

#163

Post by alekandreev »

sidkshatriya wrote: post #157
alekandreev wrote: post #33

Greetings from the Gemma team! We just got Gemma 3 out of the oven and are super excited to show it to you! Please drop any questions here and we'll answer ASAP. (Opinions our own and not of Google DeepMind.) PS we are hiring: https://boards.greenhouse.io/deepmind/jobs/6590957

As per the technical report, every 5 layers you have a global attention layer. The global attention layer during training can have as many as a 128k context length during training (though I understand it is usually 32k). Q. When you are training with a context length of 128k, is the attention in the global layers dense or sparse ? If dense, would the attention memory requirement here would be O(n^2) where n is 128k f…

We never train at 128k, only 32k, changing the scaling factor at the end.

We wanted the long context recipe to be friendly for finetuning, and training at 128k is a bit of a pain we don't do it. For inference, we see inference at 128k with the 5/1 is close to RAM usage for a fully-global-layer model at 32k.

Individual attention layers are always dense.

Re: Gemma 3 Technical Report [pdf]

#164

Post by alekandreev »

mdp2021 wrote: post #106
alekandreev wrote: post #103

Earlier quoted context omitted.

That's an idea we've thought about. However, we think the open source community has already created a very impressive set of language or region-specific finetunes [1] [2]. Also there is a lot of cultural and nuance context in every language that we don't have the capacity to cover sufficiently. So for v3 we focused on creating the best foundational multilingual model. [1] https://huggingface.co/aiplanet/buddhi-indic…

And have you measured the trade-off that could come with embracing such a large number of languages and alphabets? It would be interesting to note whether you are sacrificing some response quality, or if such supposed sacrifice is interestingly negligible, or if - even more interestingly - the quality increases with the added proficiency.

Yes we have measured the tradeoff. We don't see a drop of perplexity in English when introducing multilingual, and there is a slight drop in some English language-specific evals (~1%).

Re: Gemma 3 Technical Report [pdf]

#165

Post by mrob »

tpm wrote: post #30
hargup wrote: post #5

Earlier quoted context omitted.

Gemma is their open-source series of models. Gemini is the propertierary ones. Gemini models are bigger and better. But Gemma are pretty good too.

open-weights, not open-source (sorry to be that one but open source in this case would mean you can build it yourself from provided "source", which you can't, because it's not provided)

And even "open-weights" is generous, as they're released under a proprietary license with usage restrictions, not an open-source license.

Re: Gemma 3 Technical Report [pdf]

#166

Post by mdp2021 »

Workaccount2 wrote: post #143
mdp2021 wrote: post #106

Earlier quoted context omitted.

And have you measured the trade-off that could come with embracing such a large number of languages and alphabets? It would be interesting to note whether you are sacrificing some response quality, or if such supposed sacrifice is interestingly negligible, or if - even more interestingly - the quality increases with the added proficiency.

There are enough small model teams competing that I fell confident one of them will try this, and if it just sticking to english gives a large boost, the others will be forced to follow suite. It would also kind of suck for non-english speakers, because it will just be another feather in the hat of "English eats the world".

Some numbers to try and make an idea: if I understand correctly, Gemma3 uses a fixed (in its versions by size) vocabulary 256k entries big; the smallest 1B version has ~300M embedding parameters and ~700M non-embedding parameters; the largest 27B version has ~5x embedding parameters and ~35x non-embedding parameters.

Multilingualism covering 140 languages is quite a big feat. Gemma3 apparently aims to be compact and efficient. The two goals and features put together raise questions. You wonder for example how much does such extensive multilingualism impact the above numbers, on a benchmark of similar results. It may e.g. be a general question to wonder how much multilingualism complicates an embedding space (owing e.g. to omographic collisions), and the question becomes more prominent when you crammed 140 languages in one model.

> non-english speakers

You would produce more specialized models (where it makes sense): Eng; Eng-Fra-Esp-Deu; Man-Can... For a billion weights per model it could probably be financially acceptable.

Re: Gemma 3 Technical Report [pdf]

#167

Post by idonotknowwhy »

msp26 wrote: post #119
LeoPanthera wrote: post #13

> They are designed to help prevent our models from generating harmful content, i.e., > [...] > Sexually explicit content Dear tech companies. Sexually explicit content is not harmful. Why are you all run by puritans? I don't even want to make edgy porn, I just want to be treated like an adult.

I want to use a multimodal model for manga translation, analysis, and tagging. If this gives me the "aschually as a ethical safe harmless assistant I can't ..." spiel on anything mildly mature, that would be very disappointing. I'll run a test with Berserk and see how it goes. I'm not a big believer in abliteration, it seems to always hurt performance. Safety should be handled by a separate system, no need to cripple…

The multimodal models aren't good for this. Refusals aren't the issue (they're fine with BERSERK, though occasionally they'll refuse for copyright). The issue is the tech isn't there yet.

You'll want to use custom models to segment the manga (panels, speech bubbles), OCR the text, translate (gemma punches above it's weights for this part).

That said, I've been experimenting with using Pixtral to do the analysis part with okay-ish results (providing individual panels with the character names) but it'll still mix up the characters when they're drawn differently.

> I'm not a big believer in abliteration, it seems to always hurt performance.

Agreed, it's fun to play with but it increases halucinations. And for creative writing, it makes the model write more compliant characters (they'll give in too easily during negotiations, rather than refuse, etc)

Could probably be improved with more targeted abliteration.

Re: Gemma 3 Technical Report [pdf]

#168

Post by sidkshatriya »

alekandreev wrote: post #163
sidkshatriya wrote: post #157

Earlier quoted context omitted.

As per the technical report, every 5 layers you have a global attention layer. The global attention layer during training can have as many as a 128k context length during training (though I understand it is usually 32k). Q. When you are training with a context length of 128k, is the attention in the global layers dense or sparse ? If dense, would the attention memory requirement here would be O(n^2) where n is 128k f…

We never train at 128k, only 32k, changing the scaling factor at the end. We wanted the long context recipe to be friendly for finetuning, and training at 128k is a bit of a pain we don't do it. For inference, we see inference at 128k with the 5/1 is close to RAM usage for a fully-global-layer model at 32k. Individual attention layers are always dense.

Thanks for your answer ! So in the 32k global layer, every token attends to each of the other 32k tokens ?

[Edit: You answered the question when you said that individual attention layers are always dense.]

Re: Gemma 3 Technical Report [pdf]

#169

Post by osanseviero »

LeoPanthera wrote: post #20
meetpateltech wrote: post #2

Gemma 3 is out! Multimodal (image + text), 128K context, supports 140+ languages, and comes in 1B, 4B, 12B, and 27B sizes with open weights & commercial use. Gemma 3 model overview: https://ai.google.dev/gemma/docs/core Huggingface collection: https://huggingface.co/collections/google/gemma-3-release-67... ollama: https://ollama.com/library/gemma3

Doesn't yet work in LM Studio. Barfs an error when trying to load the model. (Error 6, whatever that means. Happy I missed the first 5.)

Please make sure to update to the latest llama.cpp version

Re: Gemma 3 Technical Report [pdf]

#170

Post by aoeusnth1 »

tomthe wrote: post #22

Very cool open release. Impressive that a 27b model can be as good as the much bigger state of the art models (according to their table of Chatbot Arena, tied with O1-preview and above Sonnet 3.7). But the example image shows that this model still makes dumb errors or has a poor common sense although it read every information correctly.

Looking at every other benchmark, it's significantly behind typical big models from a year ago (Claude 3.0, Gemini 1.5, GPT 4.0). I think Google must have extensive LMArena-focused RLHF tuning for their models to juice their scores.
Post reply on HN