Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemma: New Open Models

blog.google

Post reply on HN
Print view View original

291–300 of 543 posts

Re: Gemma: New Open Models

#291

Post by espadrine »

lalaithion wrote: post #254
espadrine wrote: post #29

I notice a few divergences to common models: - The feedforward hidden size is 16x the d_model, unlike most models which are typically 4x; - The vocabulary size is 10x (256K vs. Mistral’s 32K); - The training token count is tripled (6T vs. Llama2's 2T) Apart from that, it uses the classic transformer variations: MQA, RoPE, RMSNorm. How big was the batch size that it could be trained so fast? https://huggingface.co/mis…

What does tokenization look like in 256k vs 32k?

It mostly means that there are tokens dedicated to rarer sequences of characters, even in foreign languages (note that Gemma is not intended to be good multilingually): “説明書” (instruction manual) has its own token, and so does “Nixon”, “آباد” (a city suffix, I believe), and the HTML sequence "\"><!--".

Re: Gemma: New Open Models

#292

Post by sorokod »

robswc wrote: post #281
cooper_ganglia wrote: post #275

Earlier quoted context omitted.

I wonder if they have a system prompt to promote diversity in outputs that touch on race at all? I’ve seen several instances of people requesting a photo of a specific people, and it adds in more people to diversify. Not inherently bad, but it is if it forces it to provide incorrect answers like in your example.

That's what I don't understand. I asked it why it assumed Native Americans were in Japan and it said: > I assumed [...] various ethnicities, including Indigenous American, due to the diversity present in Japan throughout history. However, this overlooked [...] I focused on providing diverse representations without adequately considering the specific historical context. I see no reason why this sort of thing won't ext…

Pretty funny as Japan is known to be one of the least ethnically diverse countries in the world.

Re: Gemma: New Open Models

#293

Post by aetherson »

robswc wrote: post #257

I personally can't take any models from google seriously. I was asking it about the Japanese Heian period and it told me such nonsensical information you would have thought it was a joke or parody. Some highlights were "Native American women warriors rode across the grassy plains of Japan, carrying Yumi" and "A diverse group of warriors, including a woman of European descent wielding a katana, stand together in camar…

Were you asking Gemma about this, or Gemini? What were your prompts?

Re: Gemma: New Open Models

#294

Post by margorczynski »

cooper_ganglia wrote: post #275
robswc wrote: post #257

I personally can't take any models from google seriously. I was asking it about the Japanese Heian period and it told me such nonsensical information you would have thought it was a joke or parody. Some highlights were "Native American women warriors rode across the grassy plains of Japan, carrying Yumi" and "A diverse group of warriors, including a woman of European descent wielding a katana, stand together in camar…

I wonder if they have a system prompt to promote diversity in outputs that touch on race at all? I’ve seen several instances of people requesting a photo of a specific people, and it adds in more people to diversify. Not inherently bad, but it is if it forces it to provide incorrect answers like in your example.

> Not inherently bad

It is, it's consistently doing something the user didn't asked to and in most cases doesn't want. In many cases the model is completely unusable.

Re: Gemma: New Open Models

#295

Post by charcircuit »

ramoz wrote: post #279
robswc wrote: post #257

I personally can't take any models from google seriously. I was asking it about the Japanese Heian period and it told me such nonsensical information you would have thought it was a joke or parody. Some highlights were "Native American women warriors rode across the grassy plains of Japan, carrying Yumi" and "A diverse group of warriors, including a woman of European descent wielding a katana, stand together in camar…

I was wondering if these models would perform in such a way, given this week's X/twitter storm over Gemini generated images. E.g. https://x.com/debarghya_das/status/1759786243519615169?s=20 https://x.com/MiceynComplex/status/1759833997688107301?s=20 https://x.com/AravSrinivas/status/1759826471655452984?s=20

Those are most likely due to the system prompt which tries to reduce bias (but ends introducing bias in the opposite direction for some prompts as you can see) so I wouldn't expect to see that happen with an open model where you can control the entire system prompt

Re: Gemma: New Open Models

#296

Post by whymauri »

robbiep wrote: post #282
robswc wrote: post #257

I personally can't take any models from google seriously. I was asking it about the Japanese Heian period and it told me such nonsensical information you would have thought it was a joke or parody. Some highlights were "Native American women warriors rode across the grassy plains of Japan, carrying Yumi" and "A diverse group of warriors, including a woman of European descent wielding a katana, stand together in camar…

I find myself shocked that people ask questions of the world from these models, as though pulping every text and its component words relationships and deriving statistical relationships between them should reliably deliver useful information. Don’t get me wrong, I’ve used LLMs and been amazed by their output, but the p-zombie statistical model has no idea what it is saying back to you and the idea that we should trus…

I mean, I use GPT-4 on the daily as part of my work and it reliably delivers useful information. It's actually the exception for me if it provides garbage or incorrect information about code.

Re: Gemma: New Open Models

#297

Post by spiantino »

trisfromgoogle wrote: post #106
turnsout wrote: post #102

Earlier quoted context omitted.

What is the license? I couldn’t find it on the 1P site or Kaggle.

You can find the terms on our website, ai.google.dev/gemma: https://ai.google.dev/gemma/terms

out of curiosity, why is this a "terms" and not a license? I'm used to reading and understanding the software as coming with a license to use it. Do the terms give us license to use this explicitly?

Re: Gemma: New Open Models

#298

Post by callalex »

p1esk wrote: post #193
mustafabisic1 wrote: post #10

The fact Gemma team is in the comments section answering questions is praiseworthy to me :)

https://twitter.com/yar_vol/status/1760314018575634842

The link is broken. On HN (or any forum really) it is expected for a brief description of the content to be provided when posting a link. Links die all the time, but forum posts don’t have to die with them.

Re: Gemma: New Open Models

#299

Post by vorticalbox »

espadrine wrote: post #53
vorticalbox wrote: post #40

Earlier quoted context omitted.

are there plans to release an official GGUF version to use with llama.ccp?

It is already part of the release on Huggingface: https://huggingface.co/google/gemma-7b/blob/main/gemma-7b.gg... It is a pretty clean release! I had some 500 issues with Kaggle validating my license approval, so you might too, but after a few attempts I could access the model.

I didn't see this when searching thanks

Re: Gemma: New Open Models

#300

Post by mvdtnz »

robbiep wrote: post #282
robswc wrote: post #257

I personally can't take any models from google seriously. I was asking it about the Japanese Heian period and it told me such nonsensical information you would have thought it was a joke or parody. Some highlights were "Native American women warriors rode across the grassy plains of Japan, carrying Yumi" and "A diverse group of warriors, including a woman of European descent wielding a katana, stand together in camar…

I find myself shocked that people ask questions of the world from these models, as though pulping every text and its component words relationships and deriving statistical relationships between them should reliably deliver useful information. Don’t get me wrong, I’ve used LLMs and been amazed by their output, but the p-zombie statistical model has no idea what it is saying back to you and the idea that we should trus…

People ask these kinds of questions because tech companies and the media have been calling these things (rather ridiculously) "AI".
Post reply on HN