Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemma: New Open Models

blog.google

Post reply on HN
Print view View original

221–230 of 543 posts

Re: Gemma: New Open Models

#221

Post by brucethemoose2 »

sqreept wrote: post #194

Tried inference with the 7B model and without flash attention this is soooooo slow. With flash attention the fine-tunning requires A100 or H100. Also the inference doesn't always stop generating resulting in garbage being added to the response.

> Also the inference doesn't always stop generating resulting in garbage being added to the response.

That sounds like a chat format misconfiguration.

This could partially be Google's fault, as they used yet another novel prompting format.

Also, for sane inference speed on H100s, you'll have to wait for architecture support from the optimized frameworks. Vanilla transformers is beyond awful even with FA2.

Re: Gemma: New Open Models

#225

Post by brucethemoose2 »

anshumankmr wrote: post #164

Are these any good? I have been trying the non pro version of Gemini, and that seems awful at code generation. I am more keen on getting access to the best model and I would pay for it if I wasn't already paying for ChatGPT 4.

You should be looking at Deepseek's coding models, and finetunes of those.

I run 33B on my desktop, and find it to be sufficient for many tasks.

Re: Gemma: New Open Models

#226

Post by brucethemoose2 »

minimaxir wrote: post #217
ericskiff wrote: post #70

Has anyone found the context length for these models yet? So far I haven't seen it mentioned in their write-up or the model card

For posterity, an easy way to find the context length of a LLM hosted on Hugging Face is to look at the max_position_embeddings in the config.json, which shows the 8192 mentioned in another comment. (although in this case you need to sign the agreement first)

There are some exceptions, like Mistral 0.1 (which is technically 32K according to the config but practically 8K because the sliding window is awful) and InternLM (which (at least initially) used auto rope scaling to extend the context as part of the model's architecture).

Re: Gemma: New Open Models

#228

Post by brucethemoose2 »

alekandreev wrote: post #60
zitterbewegung wrote: post #6

Earlier quoted context omitted.

Do you have a plan of releasing higher parameter models?

We have many great things in research and development phases, so stay tuned. I’m hopeful we can share more in the coming weeks and month!

That is awesome!

I hope y'all consider longer context models as well.

Also, are ya'll looking alternative architectures like Mamba? Being "first" with a large Mamba model would cement your architectural choices/framework support like llama did for Meta.

Re: Gemma: New Open Models

#229

Post by simonw »

The terms of use: https://ai.google.dev/gemma/terms and https://ai.google.dev/gemma/prohibited_use_policy

Something that caught my eye in the terms:

> Google may update Gemma from time to time, and you must make reasonable efforts to use the latest version of Gemma.

One of the biggest benefits of running your own model is that it can protect you from model updates that break your carefully tested prompts, so I’m not thrilled by that particular clause.

Re: Gemma: New Open Models

#230

Post by tgtweak »

simonw wrote: post #229

The terms of use: https://ai.google.dev/gemma/terms and https://ai.google.dev/gemma/prohibited_use_policy Something that caught my eye in the terms: > Google may update Gemma from time to time, and you must make reasonable efforts to use the latest version of Gemma. One of the biggest benefits of running your own model is that it can protect you from model updates that break your carefully tested prompts, so I’m not…

I don't think there's a way they can enforce that reasonably. There's no connection to the mothership to report back what version is being used or license keys at runtime...

Seems more like a "if we discover something unsafe you should update your model and we aren't liable if you don't" than something that would make your model stop working.

Post reply on HN