Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemma: New Open Models

blog.google

Post reply on HN
Print view View original

271–280 of 543 posts

Re: Gemma: New Open Models

#271

Post by moffkalast »

impulser_ wrote: post #27

Go back 5 years and ask anyone on this site what companies do you think will be the most open about AI in the future OpenAI, Meta, or Google. I bet 10/10 people would pick OpenAI. Now today Meta and Google, both trillion dollars companies, are releasing very powerful open models with the ability to be used commercially. Ironic.

> what companies do you think will be the most open about AI in the future OpenAI, Meta, or Google.

The funny part is that the real answer is: Some random French company is running circles around them all.

I mean who the hell just drops a torrent magnet link onto twitter for the best state of the art LLM base model for its size class, and with a completely open license. No corporate grandstanding, no benchmark overpromises, no theatrics. That was unfathomably based of Mistral.

Re: Gemma: New Open Models

#272

Post by stale2002 »

wantsanagent wrote: post #168

The utter bullshit of these licenses has got to stop. Do not, under any circumstances, consider using these commercially. "Google reserves the right to restrict (remotely or otherwise) usage of any of the Gemma Services that Google reasonably believes are in violation of this Agreement." This is a kill switch that Google maintains in perpetuity over any system you build relying on these models. Our legal review of th…

Wait, you actually care about the license and read it?

It's seems like you aren't up to date.

Most of the startup space is entirely ignoring all these licenses. If the weights are available, it is being used commerically without regards to any licensing.

And everyone is getting away with it and nobody is being sued.

Good luck trying to keep up if you aren't doing the same!

Feels free to hamstring yourself though if you like.

Re: Gemma: New Open Models

#273

Post by refulgentis »

wouldbecouldbe wrote: post #270

I really don't get why there is this obsession with safe "Responsible Generative AI". I mean it writes some bad words, or bad pics, a human can do that without help as well. The good thing about dangerous knowledge and generative AI is that you're never sure haha, you'd be a fool to ask GPT to make a bomb. I mean it would probably be safe, since it will make up half of the steps.

I guess what I'd tell you is, there's a lot of fools in this world.

Re: Gemma: New Open Models

#274

Post by TheHypnotist »

graphe wrote: post #250
audessuscest wrote: post #111

Earlier quoted context omitted.

Does this model also thinks german were black 200 years ago ? Or is afraid to answer basic stuff ? because if this is the case no one will care about that model.

I disagree, coding and RAG performance is all that matters to me. I'm not using an LLM to learn basic facts I already know.

How do you ragebait for premium pearl clutching?

Re: Gemma: New Open Models

#275

Post by cooper_ganglia »

robswc wrote: post #257

I personally can't take any models from google seriously. I was asking it about the Japanese Heian period and it told me such nonsensical information you would have thought it was a joke or parody. Some highlights were "Native American women warriors rode across the grassy plains of Japan, carrying Yumi" and "A diverse group of warriors, including a woman of European descent wielding a katana, stand together in camar…

I wonder if they have a system prompt to promote diversity in outputs that touch on race at all? I’ve seen several instances of people requesting a photo of a specific people, and it adds in more people to diversify. Not inherently bad, but it is if it forces it to provide incorrect answers like in your example.

Re: Gemma: New Open Models

#276

Post by itomatik »

FergusArgyll wrote: post #153
tosh wrote: post #45

Benchmarks for Gemma 7B seem to be in the ballpark of Mistral 7B +-------------+----------+-------------+-------------+ | Benchmark | Gemma 7B | Mistral 7B | Llama-2 7B | +-------------+----------+-------------+-------------+ | MMLU | 64.3 | 60.1 | 45.3 | | HellaSwag | 81.2 | 81.3 | 77.2 | | HumanEval | 32.3 | 30.5 | 12.8 | +-------------+----------+-------------+-------------+ via https://mistral.ai/news/announcing-…

the real gold will be when this gets finetuned. (maybe by mistral...)

how does one finetune llama (or any other LLM) using mistral?

is the flow like this?

- take small dataset

- generate bigger dataset using mistral (how this is this done?)

- run LoRA to fine tune gemma extended dataset.

Re: Gemma: New Open Models

#277

Post by refulgentis »

rfw300 wrote: post #209
jcuenod wrote: post #149

Earlier quoted context omitted.

Came here to post the same thing for Phi-2: +-------------+----------+-------------+ | Benchmark | Gemma 2B | Phi-2 2.7B | +-------------+----------+-------------+ | MMLU | 42.3 | 56.7 | | MBPP | 29.2 | 59.1 | | BoolQ | 69.4 | 83.3 | +-------------+----------+-------------+ [0] https://www.kaggle.com/models/google/gemma [1] https://www.microsoft.com/en-us/research/blog/phi-2-the-surp...

A caveat: my impression of Phi-2, based on my own use and others’ experiences online, is that these benchmarks do not remotely resemble reality. The model is a paper tiger that is unable to perform almost any real-world task because it’s been fed so heavily with almost exclusively synthetic data targeted towards improving benchmark performance.

Hear hear! I don't understand why it has persistent mindshare, it's not even trained for chat. Meanwhile StableLM 3B runs RAG in my browser, on my iPhone, on my Pixel ..

Re: Gemma: New Open Models

#278

Post by phh »

rfw300 wrote: post #209
jcuenod wrote: post #149

Earlier quoted context omitted.

Came here to post the same thing for Phi-2: +-------------+----------+-------------+ | Benchmark | Gemma 2B | Phi-2 2.7B | +-------------+----------+-------------+ | MMLU | 42.3 | 56.7 | | MBPP | 29.2 | 59.1 | | BoolQ | 69.4 | 83.3 | +-------------+----------+-------------+ [0] https://www.kaggle.com/models/google/gemma [1] https://www.microsoft.com/en-us/research/blog/phi-2-the-surp...

A caveat: my impression of Phi-2, based on my own use and others’ experiences online, is that these benchmarks do not remotely resemble reality. The model is a paper tiger that is unable to perform almost any real-world task because it’s been fed so heavily with almost exclusively synthetic data targeted towards improving benchmark performance.

Fun that's not my experience of Phi-2. I use it for non-creative context, but function calling, and I find as reliable as much bigger models (no fine-tuning just constraining JSON + CoT). Phi-2 unquantized vs Mixtral Q8, Mixtral is not definitely better but much slower and RAM-hungry.

Re: Gemma: New Open Models

#279

Post by ramoz »

robswc wrote: post #257

I personally can't take any models from google seriously. I was asking it about the Japanese Heian period and it told me such nonsensical information you would have thought it was a joke or parody. Some highlights were "Native American women warriors rode across the grassy plains of Japan, carrying Yumi" and "A diverse group of warriors, including a woman of European descent wielding a katana, stand together in camar…

I was wondering if these models would perform in such a way, given this week's X/twitter storm over Gemini generated images.

E.g.

https://x.com/debarghya_das/status/1759786243519615169?s=20

https://x.com/MiceynComplex/status/1759833997688107301?s=20

https://x.com/AravSrinivas/status/1759826471655452984?s=20

Re: Gemma: New Open Models

#280

Post by willy_k »

dougmwne wrote: post #218
willy_k wrote: post #195

Earlier quoted context omitted.

Is “the opposite of censored” accurate for something that’s default and considerably easier to access mode of operation won’t say many things for sociopolitical reasons? Able to be un censored, sure, but the extent of that is debatable as well.

There is no default and easy access mode. These are raw model weights and only enthusiasts and researchers will download the necessary packages to run it locally. Much more likely is that some popular fine tunes show up on hugging face for more general access.

I agree that there probably will be “uncensored” fine tuned models that become available, my point was just that it’s not accurate to call Gemma “the opposite of censored” because there is a somewhat involved step that needs to be taken before it even appears uncensored. It’s also likely missing a lot of useful context that was removed from the training set and not meaningfully replaced during fine-tuning, and besides that any fine tuned “uncensored” model will be based on Gemma, not Google’s Gemma itself.

IMO “Opposite of uncensored” suggests a model whose original form eagerly gives out controversial / typically censored information, not a model that is censored but able to be fine tuned away from censorship.

Post reply on HN