Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemma: New Open Models

blog.google

Post reply on HN
Print view View original

371–380 of 543 posts

Re: Gemma: New Open Models

#371

Post by chatmasta »

sotasota wrote: post #369
epistasis wrote: post #352

Earlier quoted context omitted.

Of all the very very very many things that Google models get wrong, not understanding nationality and skin tone distributions seems to be a very weird one to focus on. Why are there three links to this question? And why are people so upset over it? Very odd, seems like it is mostly driven by political rage.

Because the wrongness is intentional.

Exactly. Sure this particular example is driven by political rage, but the underlying issue is that the maintainers of these models are altering them to conform to an agenda. It's not even surprising that people choose to focus on the political rage aspect of it, because that same political rage is the source of the agenda in the first place. It's a concerning precedent to set, because what other non-political modifications might be in the model?

Re: Gemma: New Open Models

#372

Post by sergiotapia »

protomolecule wrote: post #330
ramoz wrote: post #279

Earlier quoted context omitted.

I was wondering if these models would perform in such a way, given this week's X/twitter storm over Gemini generated images. E.g. https://x.com/debarghya_das/status/1759786243519615169?s=20 https://x.com/MiceynComplex/status/1759833997688107301?s=20 https://x.com/AravSrinivas/status/1759826471655452984?s=20

Regarding the last one: there 1.5 million immigrants in Norway with total population 5.4 million. Gemini isn't very wrong, is it?

bro you know exactly what the request meant. GOOGLE knew exactly what the request meant, and had to _train_ it to do something worse. Come on now.

If I ask for a Bolivian woman, I expect a colla or a camba. Not a japanese woman, despite Santa Cruz having a very large japanese population.

Re: Gemma: New Open Models

#373

Post by epistasis »

sotasota wrote: post #369
epistasis wrote: post #352

Earlier quoted context omitted.

Of all the very very very many things that Google models get wrong, not understanding nationality and skin tone distributions seems to be a very weird one to focus on. Why are there three links to this question? And why are people so upset over it? Very odd, seems like it is mostly driven by political rage.

Because the wrongness is intentional.

Is it intentional? You think they intentionally made it not understand skin tone distribution by country? I would believe it if there was proof, but with all the other things it gets wrong it's weird to jump to that conclusion.

There's way too much politics in these things. I'm tired of people pushing on the politics rather than pushing for better tech.

Re: Gemma: New Open Models

#374

Post by thrdbndndn »

robswc wrote: post #284
robbiep wrote: post #282

Earlier quoted context omitted.

I find myself shocked that people ask questions of the world from these models, as though pulping every text and its component words relationships and deriving statistical relationships between them should reliably deliver useful information. Don’t get me wrong, I’ve used LLMs and been amazed by their output, but the p-zombie statistical model has no idea what it is saying back to you and the idea that we should trus…

I don't have this problem with any other model. I've had really long conversations with ChatGPT on road trips and it has never gone off the rails like Gemini seems to do.

ChatGPT the only model I did not have such problem.

Any local models can go off the rail very easily and more importantly, they're very bad at following very specific instructions.

Re: Gemma: New Open Models

#375

Post by GaggiX »

voxgen wrote: post #359
GaggiX wrote: post #220

Earlier quoted context omitted.

I don't see the number 49152 reported in the config.json, what line are you referring to? I just see the intermediate_size of 24576 (so 8x). EDIT: I didn't read the comment correctly, you have noticed the same thing.

The *GLU-based activations functions like GEGLU and SwiGLU use 2 input values to produce 1 output value, which makes these numbers weird. In each value pair, one goes through the GELU/SiLU activation function and is then multiplied by the other "gate" value. In the report, "hidden dim" matches the number of GEGLU inputs. In the config, "intermediate_size" matches the number of GEGLU outputs. Most *GLU models so far h…

Makes perfect sense thx

Re: Gemma: New Open Models

#376

Post by lalaithion »

espadrine wrote: post #291
lalaithion wrote: post #254

Earlier quoted context omitted.

What does tokenization look like in 256k vs 32k?

It mostly means that there are tokens dedicated to rarer sequences of characters, even in foreign languages (note that Gemma is not intended to be good multilingually): “説明書” (instruction manual) has its own token, and so does “Nixon”, “آباد” (a city suffix, I believe), and the HTML sequence "\"><!--".

I understand the theory, I was looking for an example of the same text tokenized with the two different vocabularies.

Re: Gemma: New Open Models

#377

Post by lalaithion »

visarga wrote: post #360
lalaithion wrote: post #254

Earlier quoted context omitted.

What does tokenization look like in 256k vs 32k?

Text encodes in fewer tokens, and language coverage is better.

I understand the theory, I was looking for an example of the same text tokenized with the two different vocabularies.

Re: Gemma: New Open Models

#378

Post by res0nat0r »

catchnear4321 wrote: post #358
simonw wrote: post #229

The terms of use: https://ai.google.dev/gemma/terms and https://ai.google.dev/gemma/prohibited_use_policy Something that caught my eye in the terms: > Google may update Gemma from time to time, and you must make reasonable efforts to use the latest version of Gemma. One of the biggest benefits of running your own model is that it can protect you from model updates that break your carefully tested prompts, so I’m not…

reasonable effort - meaning if their changes meaningfully impact my usage, negatively, it would be unreasonable to ask me to upgrade. sounds good. this is not financial advice and ianal.

Isn't this just lawyer speak for "we update our model a lot, and we've never signed off on saying we're going to support every previous release we've ever published, and may turn them off at any time, don't complain about it when we do."

Re: Gemma: New Open Models

#379

Post by espadrine »

andy_xor_andrew wrote: post #357
espadrine wrote: post #29

I notice a few divergences to common models: - The feedforward hidden size is 16x the d_model, unlike most models which are typically 4x; - The vocabulary size is 10x (256K vs. Mistral’s 32K); - The training token count is tripled (6T vs. Llama2's 2T) Apart from that, it uses the classic transformer variations: MQA, RoPE, RMSNorm. How big was the batch size that it could be trained so fast? https://huggingface.co/mis…

> The training token count is tripled (6T vs. Llama2's 2T) Damn, 6T? That's a lot! Given that this model seems to roughly match Mistral (according to the numbers from Google), this makes me think we have saturated the 7B parameter space, and couldn't possibly make it much better unless new techniques are discovered.

Hard to say definitively. Mistral’s token embeddings only account for <2% of the 7B parameters, while Gemma’s larger token vocabulary vampirized over 10%, leaving less space for the more important parts of the network. It is a somewhat surprising tradeoff given that it was pretrained towards an English bias.

Re: Gemma: New Open Models

#380

Post by espadrine »

lalaithion wrote: post #376
espadrine wrote: post #291

Earlier quoted context omitted.

It mostly means that there are tokens dedicated to rarer sequences of characters, even in foreign languages (note that Gemma is not intended to be good multilingually): “説明書” (instruction manual) has its own token, and so does “Nixon”, “آباد” (a city suffix, I believe), and the HTML sequence "\"><!--".

I understand the theory, I was looking for an example of the same text tokenized with the two different vocabularies.

Do you have an example text in mind?

You can use this playground to test it out: https://huggingface.co/spaces/Xenova/the-tokenizer-playgroun...

Post reply on HN