Gemma 2: Improving Open Language Models at a Practical Size [pdf]
storage.googleapis.com
Gemma 2: Improving Open Language Models at a Practical Size [pdf]
1–10 of 183 posts
Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]
#2The 9B and 27B versions are available for Ollama: https://ollama.com/library/gemma2
Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]
#3> Table 4 | Relevant formatting control tokens used for Gemma models
> User turn: user
> Model turn: model
> Start of conversation turn:
> End of conversation turn:
> Beginning of sequence:
> End of sequence:
You know I keep wondering why and tokens are even a thing in general. No model is tuned to keep generating multiple turns after its equivalent is sent, and what's the point of when you're parsing the entire context anyway. If it's an attempt to ignore text before it... then why is that text there? Just remove it from context, you're throwing away compute.
Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]
#4> Table 4 | Relevant formatting control tokens used for Gemma models > User turn: user > Model turn: model > Start of conversation turn: > End of conversation turn: > Beginning of sequence: > End of sequence: You know I keep wondering why and tokens are even a thing in general. No model is tuned to keep generating multiple turns after its equivalent is sent, and what's the point of when you're parsing the entire cont…
think about training.
Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]
#5Shouldn't this (2.6B/9B) be compared with Microsoft's Phi-3 mini (3.8B) instead of Mistral and Llama-3?
(table 13 on page 7) vs https://arxiv.org/pdf/2404.14219 (page 6, quite better in general)
The report on knowledge distillation training is interesting, though.
Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]
#6Hello (again) from the Gemma team! We are quite excited to push this release out and happy to answer any questions!
Opinions are our own and not of Google DeepMind.
Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]
#7I gave up hope on r"Gem[ma|ini]" long time ago. I don't believe that Google can't produce good LLMs because of its massive company size; Microsoft is also a giant company (more market cap than Google) but it keeps surprising us with the ϕ models.
I think Google just lacks the vision to understand what makes a good LLM. Theoretical contributions by research teams are valuable, but the real-world is built around engineering ideas that may lack the "purity" and elegance of theory but damn it they work.
Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]
#8Hello (again) from the Gemma team! We are quite excited to push this release out and happy to answer any questions! Opinions are our own and not of Google DeepMind.
How is Gemma-2 licensed?
Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]
#9So it's twice the size of phi 3 and considerably worse? What am I missing
Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]
#10Hello (again) from the Gemma team! We are quite excited to push this release out and happy to answer any questions! Opinions are our own and not of Google DeepMind.
No question. Thanks for thinking of 27B.