Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemma: New Open Models

blog.google

Post reply on HN
Print view View original

471–480 of 543 posts

Re: Gemma: New Open Models

#471

Post by kweingar »

robswc wrote: post #395
epistasis wrote: post #373

Earlier quoted context omitted.

Is it intentional? You think they intentionally made it not understand skin tone distribution by country? I would believe it if there was proof, but with all the other things it gets wrong it's weird to jump to that conclusion. There's way too much politics in these things. I'm tired of people pushing on the politics rather than pushing for better tech.

I mean, I asked it for a samurai from a specific Japanese time period and it gave me a picture of a "non-binary indigenous American woman" (its words, not mine) so I think there is something intentional going on.

I would be very surprised if it said "nonbinary indigenous American woman" considering that nonbinary and woman are different categories

Re: Gemma: New Open Models

#472

Post by kweingar »

verticalscaler wrote: post #464
willsmith72 wrote: post #451

Earlier quoted context omitted.

Yeah that is just absurd. Google has been burnt before, e.g. classifying black people as gorillas in 2015, so I can understand their fear when they have so much to lose, but clearly they've gone way too far the other way and are going to have to do a lot to regain people's trust. For now, Gemini is a play toy https://www.bbc.com/news/technology-33347866.amp

Completely unrelated, enough excuses. This is not some sort of mistake or overcorrection, it is by explicit overt design. These cowards will never regain my trust. I won't hire or work with or for googlers or any DEI people ever.

> I won't hire or work with or for googlers or any DEI people ever.

I’m sure they’ll be very sad not to work with you.

Re: Gemma: New Open Models

#473

Post by verticalscaler »

kweingar wrote: post #472
verticalscaler wrote: post #464

Earlier quoted context omitted.

Completely unrelated, enough excuses. This is not some sort of mistake or overcorrection, it is by explicit overt design. These cowards will never regain my trust. I won't hire or work with or for googlers or any DEI people ever.

> I won't hire or work with or for googlers or any DEI people ever. I’m sure they’ll be very sad not to work with you.

Of course not, I'm the wrong color or whatever the hell.

Re: Gemma: New Open Models

#475

Post by attentive »

tosh wrote: post #45

Benchmarks for Gemma 7B seem to be in the ballpark of Mistral 7B +-------------+----------+-------------+-------------+ | Benchmark | Gemma 7B | Mistral 7B | Llama-2 7B | +-------------+----------+-------------+-------------+ | MMLU | 64.3 | 60.1 | 45.3 | | HellaSwag | 81.2 | 81.3 | 77.2 | | HumanEval | 32.3 | 30.5 | 12.8 | +-------------+----------+-------------+-------------+ via https://mistral.ai/news/announcing-…

In my subjective tests it's not even close to Mistral. While my local gemma is quantized, so is mistral.

But I also tried gemma on huggingface.co/chat which I assume isn't quantized.

Re: Gemma: New Open Models

#476

Post by Aiedail »

This is commendable, but there's room for improvement. Up until now, SOTA-level "open-source" LLM models (LLaMA, Mistral, etc.) have usually only made their inference code and model architecture public. While these elements are not insignificant, they are somewhat trivial when compared to the training code and training datasets, as these two factors largely determine the performance of the model. This is not open at all. It goes without saying that sharing the training datasets and process with other AI researchers is crucial. This transparency would not only help to improve the model(for others could contribute to it) but also contribute to the whole community, as they usually advertised. Otherwise, it will be difficult for these efforts to truly promote the development of LLM.

Re: Gemma: New Open Models

#477

Post by robswc »

kweingar wrote: post #471
robswc wrote: post #395

Earlier quoted context omitted.

I mean, I asked it for a samurai from a specific Japanese time period and it gave me a picture of a "non-binary indigenous American woman" (its words, not mine) so I think there is something intentional going on.

I would be very surprised if it said "nonbinary indigenous American woman" considering that nonbinary and woman are different categories

You're right, my mind inserted "woman" to go with the picture:

https://gemini.google.com/share/ba324bd98d9b

At least it would never make such a heinous mistake like that :)

Re: Gemma: New Open Models

#478

Post by Jonanin »

It is surprising how willing Google is to stretch the truth in the marketing for its AI initiatives. Although it's being compared against 7B models, Gemma "7B" is actually much more than 8B parameters in total.

Re: Gemma: New Open Models

#480

Post by nl »

sa-code wrote: post #156
tosh wrote: post #45

Benchmarks for Gemma 7B seem to be in the ballpark of Mistral 7B +-------------+----------+-------------+-------------+ | Benchmark | Gemma 7B | Mistral 7B | Llama-2 7B | +-------------+----------+-------------+-------------+ | MMLU | 64.3 | 60.1 | 45.3 | | HellaSwag | 81.2 | 81.3 | 77.2 | | HumanEval | 32.3 | 30.5 | 12.8 | +-------------+----------+-------------+-------------+ via https://mistral.ai/news/announcing-…

Thank you. I thought it was weird for them to release a 7B model and not mention Mistral in their release.

The release page has comparisons to Mistral everywhere: https://ai.google.dev/gemma
Post reply on HN