Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemma 2: Improving Open Language Models at a Practical Size [pdf]

storage.googleapis.com

Post reply on HN
Print view View original

11–20 of 183 posts

Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]

#11

Post by johnfn »

behnamoh wrote: post #7

I gave up hope on r"Gem[ma|ini]" long time ago. I don't believe that Google can't produce good LLMs because of its massive company size; Microsoft is also a giant company (more market cap than Google) but it keeps surprising us with the ϕ models. I think Google just lacks the vision to understand what makes a good LLM. Theoretical contributions by research teams are valuable, but the real-world is built around engine…

Maybe you gave up before Google released Gemini Advanced? This viewpoint seemed more accurate before it was related, but Gemini Advanced is the third best LLM as rated here [1]. In fact, had second place until a few days ago when Claude 3.5 came out.

[1]: https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar...

Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]

#12

Post by [deleted] »

behnamoh wrote: post #7

I gave up hope on r"Gem[ma|ini]" long time ago. I don't believe that Google can't produce good LLMs because of its massive company size; Microsoft is also a giant company (more market cap than Google) but it keeps surprising us with the ϕ models. I think Google just lacks the vision to understand what makes a good LLM. Theoretical contributions by research teams are valuable, but the real-world is built around engine…

[deleted]

Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]

#13

Post by alecco »

behnamoh wrote: post #7

I gave up hope on r"Gem[ma|ini]" long time ago. I don't believe that Google can't produce good LLMs because of its massive company size; Microsoft is also a giant company (more market cap than Google) but it keeps surprising us with the ϕ models. I think Google just lacks the vision to understand what makes a good LLM. Theoretical contributions by research teams are valuable, but the real-world is built around engine…

I wonder if Google is making Deepmind people switch from their cool original research to doing LLMs like everybody else. Having their scale in money and data, I would hire new teams of engineers who want to do LLMs and leave the Deepmind researchers do their thing. Not killing the goose that lays golden eggs.

Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]

#14

Post by moffkalast »

alekandreev wrote: post #6

Hello (again) from the Gemma team! We are quite excited to push this release out and happy to answer any questions! Opinions are our own and not of Google DeepMind.

The 4k sliding window context seems like a controversial choice after Mistral 7B mostly failed at showing any benefits from it. What was the rationale behind that instead of just going for full 8k or 16k?

Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]

#15

Post by msoad »

Phi-3 blow this out of the water.

                      Benchmark  |  Gemma 2 (9B)  |  Phi-3 Small (7B)
    -----------------------------|----------------|-------------------
                  MMLU (5-Shot)  |       63.6     |       75.7
             HellaSwag (5-Shot)  |       49.8     |       77.0
                  ANLI (7-Shot)  |       48.7     |       58.1
           GSM-8K (8-Shot; CoT)  |       59.8     |       89.6
                 MedQA (2-Shot)  |       49.6     |       65.4
               AGIEval (0-Shot)  |       42.1     |       45.1
              TriviaQA (5-Shot)  |       72.3     |       58.1
                Arc-C (10-Shot)  |       78.3     |       90.7
                Arc-E (10-Shot)  |       91.4     |       97.0
                  PIQA (5-Shot)  |       78.1     |       86.9
                SociQA (5-Shot)  |       65.5     |       79.2
    BigBench-Hard (3-Shot; CoT)  |       59.6     |       79.1
            WinoGrande (5-Shot)  |       55.6     |       81.5
           OpenBookQA (10-Shot)  |       78.6     |       88.0
                 BoolQ (2-Shot)  |       66.0     |       84.8
        CommonSenseQA (10-Shot)  |       76.2     |       80.0
      TruthfulQA (10-Shot; MC2)  |       52.1     |       70.2
             HumanEval (0-Shot)  |       34.1     |       61.0
                  MBPP (3-Shot)  |       51.5     |       71.7

Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]

#16

Post by ertgbnm »

iamronaldo wrote: post #9

So it's twice the size of phi 3 and considerably worse? What am I missing

They used two non-mutually exclusive techniques. Phi-3 is mostly a curriculum training breakthrough. By filtering training set for high quality tokens and training on synthetic data, they were able to achieve great results. Gemma-2 is a distillation breakthrough. By training LLMs with guidance from larger teacher LLMs, they were able to achieve great results too.

Porque no los dos?

Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]

#17

Post by moffkalast »

danielmarkbruce wrote: post #4
moffkalast wrote: post #3

> Table 4 | Relevant formatting control tokens used for Gemma models > User turn: user > Model turn: model > Start of conversation turn: > End of conversation turn: > Beginning of sequence: > End of sequence: You know I keep wondering why and tokens are even a thing in general. No model is tuned to keep generating multiple turns after its equivalent is sent, and what's the point of when you're parsing the entire cont…

think about training.

I suppose it would act as a concrete separator when instruct tuning, but lots of prompt templates don't use it, especially older ones like Alpaca. Maybe it leads to more overall coherence?

Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]

#18

Post by philipkglass »

alecco wrote: post #5

Shouldn't this (2.6B/9B) be compared with Microsoft's Phi-3 mini (3.8B) instead of Mistral and Llama-3? (table 13 on page 7) vs https://arxiv.org/pdf/2404.14219 (page 6, quite better in general) The report on knowledge distillation training is interesting, though.

It's such a wide range of model sizes that I could see why they compare with Llama 3 70b as well as Llama 3 8b (tables 12, 13). I agree that the Phi-3 series is a stronger competitor for knowledge extraction/summarizing and would make a good comparison. My current favorite for such tasks, on a VRAM-limited workstation, is Phi-3 medium (phi3:14b-instruct).

Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]

#19

Post by alekandreev »

moffkalast wrote: post #3

> Table 4 | Relevant formatting control tokens used for Gemma models > User turn: user > Model turn: model > Start of conversation turn: > End of conversation turn: > Beginning of sequence: > End of sequence: You know I keep wondering why and tokens are even a thing in general. No model is tuned to keep generating multiple turns after its equivalent is sent, and what's the point of when you're parsing the entire cont…

Your training input has the shape of (sequence length x batch size). If a lot of your samples are shorter than sequence length, as is usually the case, you will have a lot of padding tokens in the input, which is wasted compute.

To compensate for that, you can pack multiple examples in the same sequence. This is there EOS and BOS come in, as they indicate to the model that the two parts of the sequence are not related.

Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]

#20

Post by mixtureoftakes »

Good realease but the annoying part is they're very unclear about which types of models they are comparing. They provide benchmark comparisons for the base models only and arena comparisons for instruct only? Was that intentional? Why would you ever do that? This makes things unnecessary complicated imo and the only payoff is a short term win for google on paper.

Guess I'll just fully test it for my own tasks to know for sure

Post reply on HN