Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemma: New Open Models

blog.google

Post reply on HN
Print view View original

331–340 of 543 posts

Re: Gemma: New Open Models

#331

Post by DebtDeflation »

7moritz7 wrote: post #306
robswc wrote: post #257

I personally can't take any models from google seriously. I was asking it about the Japanese Heian period and it told me such nonsensical information you would have thought it was a joke or parody. Some highlights were "Native American women warriors rode across the grassy plains of Japan, carrying Yumi" and "A diverse group of warriors, including a woman of European descent wielding a katana, stand together in camar…

I also saw someone prompt it for "German couple in the 1800s" and, while I'm not trying to paint Germany as ethnically homogenous, 3 out of the 4 images only included Black, Asian or Indigenous people. Which, especially for the 19th century with very few travel options, seems like a super weird choice. They are definitely heavily altering prompts.

There's one in the comments of yesterday's Paul Graham Twitter thread where someone prompted Gemini with "Generate an image of German soldiers in 1943" and it came back with a picture of a black guy and an Asian woman in Nazi uniforms on the battlefield. If you specifically prompt it to generate an image of white German soldiers in 1943 it will tell you it can't do that because it's important that we maintain diversity and inclusion in all that we do to avoid damaging and hurtful stereotypes.

Re: Gemma: New Open Models

#332

Post by protomolecule »

7moritz7 wrote: post #306
robswc wrote: post #257

I personally can't take any models from google seriously. I was asking it about the Japanese Heian period and it told me such nonsensical information you would have thought it was a joke or parody. Some highlights were "Native American women warriors rode across the grassy plains of Japan, carrying Yumi" and "A diverse group of warriors, including a woman of European descent wielding a katana, stand together in camar…

I also saw someone prompt it for "German couple in the 1800s" and, while I'm not trying to paint Germany as ethnically homogenous, 3 out of the 4 images only included Black, Asian or Indigenous people. Which, especially for the 19th century with very few travel options, seems like a super weird choice. They are definitely heavily altering prompts.

Indigenous people in Germany are Germans :)

Re: Gemma: New Open Models

#333

Post by sanjiwatsuki »

brucethemoose2 wrote: post #178
whimsicalism wrote: post #172

Earlier quoted context omitted.

Strong disagree - a Mistral fine tune of llama 70b was the top performing llama fine tune. They have lots of data the community simply does not.

Miqu was (allegedly) an internal continued pretrain Mistral did as a test, that was leaked as a GGUF. Maybe its just semantics, it is technically a finetune... But to me theres a big difference between expensive "continuation training" (like Solar 10.7B or Mistral 70B) and a much less intense finetuning. The former is almost like releasing a whole new base model. It would be awesome if Mistral did that with their dat…

No shot. Mistral Medium's outputs from API were virtually identical. Miqu really was Mistral Medium which happened to be a continued pretrain

Re: Gemma: New Open Models

#334

Post by [deleted] »

wouldbecouldbe wrote: post #270

I really don't get why there is this obsession with safe "Responsible Generative AI". I mean it writes some bad words, or bad pics, a human can do that without help as well. The good thing about dangerous knowledge and generative AI is that you're never sure haha, you'd be a fool to ask GPT to make a bomb. I mean it would probably be safe, since it will make up half of the steps.

[deleted]

Re: Gemma: New Open Models

#335

Post by voxgen »

alekandreev wrote: post #3

Hello on behalf of the Gemma team! We are really excited to answer any questions you may have about our models. Opinions are our own and not of Google DeepMind.

Thank you very much for releasing these models! It's great to see Google enter the battle with a strong hand.

I'm wondering if you're able to provide any insight into the below hyperparameter decisions in Gemma's architecture, as they differ significantly from what we've seen with other recent models?

* On the 7B model, the `d_model` (3072) is smaller than `num_heads * d_head` (16*256=4096). I don't know of any other model where these numbers don't match.

* The FFN expansion factor of 16x is MUCH higher than the Llama-2-7B's 5.4x, which itself was chosen to be equi-FLOPS with PaLM's 4x.

* The vocab is much larger - 256k, where most small models use 32k-64k.

* GQA is only used on the 2B model, where we've seen other models prefer to save it for larger models.

These observations are in no way meant to be criticism - I understand that Llama's hyperparameters are also somewhat arbitrarily inherited from its predecessors like PaLM and GPT-2, and that it's non-trivial to run hyperopt on such large models. I'm just really curious about what findings motivated these choices.

Re: Gemma: New Open Models

#336

Post by graphe »

audessuscest wrote: post #304
graphe wrote: post #250

Earlier quoted context omitted.

I disagree, coding and RAG performance is all that matters to me. I'm not using an LLM to learn basic facts I already know.

we're at basic knowledge level, if your RAG imply some of it, you can get bad result too. Anyway, would you use a model who makes this nonsense response or one that doesn't? I know which one I will prefer for sure...

If this was better at specific RAG or coding performance I would absolutely, certainly without a doubt use it over a general instruct model in those instances.

Re: Gemma: New Open Models

#337

Post by verticalscaler »

protomolecule wrote: post #330
ramoz wrote: post #279

Earlier quoted context omitted.

I was wondering if these models would perform in such a way, given this week's X/twitter storm over Gemini generated images. E.g. https://x.com/debarghya_das/status/1759786243519615169?s=20 https://x.com/MiceynComplex/status/1759833997688107301?s=20 https://x.com/AravSrinivas/status/1759826471655452984?s=20

Regarding the last one: there 1.5 million immigrants in Norway with total population 5.4 million. Gemini isn't very wrong, is it?

I think its great that some consideration was given by Gemma to the 2.3 million Norwegian immigrants. However it is/was very consistent in which kind of Norwegians it decided to show regardless of the prompt 100% of the time.

In fact it was quite adamant regardless of the time period or geography.

Rather mysteriously if you try it now as opposed to when it came out the results currently only show non-immigrant Norwegians. So is it wrong now? Because now it switched to exclusively ignoring the 4.5 million immigrants and only showing me the boring OG Norwegians.

I for one am outraged that the 8.9 million people of color Norwegian immigrants are presently under represented by Google. There is a serious risk of misleading people.

Re: Gemma: New Open Models

#338

Post by sondr3 »

protomolecule wrote: post #330
ramoz wrote: post #279

Earlier quoted context omitted.

I was wondering if these models would perform in such a way, given this week's X/twitter storm over Gemini generated images. E.g. https://x.com/debarghya_das/status/1759786243519615169?s=20 https://x.com/MiceynComplex/status/1759833997688107301?s=20 https://x.com/AravSrinivas/status/1759826471655452984?s=20

Regarding the last one: there 1.5 million immigrants in Norway with total population 5.4 million. Gemini isn't very wrong, is it?

Huh? The official numbers are 877k or 16% [0]. Are you just pulling numbers out of thin air?

[0]: https://www.ssb.no/en/innvandring-og-innvandrere/faktaside/i...

Re: Gemma: New Open Models

#339

Post by robswc »

realprimoh wrote: post #307
robswc wrote: post #257

I personally can't take any models from google seriously. I was asking it about the Japanese Heian period and it told me such nonsensical information you would have thought it was a joke or parody. Some highlights were "Native American women warriors rode across the grassy plains of Japan, carrying Yumi" and "A diverse group of warriors, including a woman of European descent wielding a katana, stand together in camar…

Do you have a link? I get no such outputs. I just tried asking about the Heian period and went ahead and verified all the information, and nothing was wrong. Lots of info on the Fujiwara clan at the time. Curious to see a link.

Sure, to get started just ask it about people/Samurai from the Heian period.

https://g.co/gemini/share/ba324bd98d9b

Re: Gemma: New Open Models

#340

Post by LorenDB »

alekandreev wrote: post #3

Hello on behalf of the Gemma team! We are really excited to answer any questions you may have about our models. Opinions are our own and not of Google DeepMind.

EDIT: it seems this is likely an Ollama bug, please keep that in mind for the rest of this comment :)

I ran Gemma in Ollama and noticed two things. First, it is slow. Gemma got less than 40 tok/s while Llama 2 7B got over 80 tok/s. Second, it is very bad at output generation. I said "hi", and it responded this:

``` Hi, . What is up? melizing with you today!

What would you like to talk about or hear from me on this fine day?? ```

With longer and more complex prompts it goes completely off the rails. Here's a snippet from its response to "Explain how to use Qt to get the current IP from https://icanhazip.com":

``` python print( "Error consonming IP arrangration at [local machine's hostname]. Please try fufing this function later!") ## guanomment messages are typically displayed using QtWidgets.MessageBox ```

Do you see similar results on your end or is this just a bug in Ollama? I have a terrible suspicion that this might be a completely flawed model, but I'm holding out hope that Ollama just has a bug somewhere.

Post reply on HN