51–60 of 183 posts
Phi-3 blow this out of the water. Benchmark | Gemma 2 (9B) | Phi-3 Small (7B) -----------------------------|----------------|------------------- MMLU (5-Shot) | 63.6 | 75.7 HellaSwag (5-Shot) | 49.8 | 77.0 ANLI (7-Shot) | 48.7 | 58.1 GSM-8K (8-Shot; CoT) | 59.8 | 89.6 MedQA (2-Shot) | 49.6 | 65.4 AGIEval (0-Shot) | 42.1 | 45.1 TriviaQA (5-Shot) | 72.3 | 58.1 Arc-C (10-Shot) | 78.3 | 90.7 Arc-E (10-Shot) | 91.4 | 97.0…
Hello (again) from the Gemma team! We are quite excited to push this release out and happy to answer any questions! Opinions are our own and not of Google DeepMind.
You can try 27b at www.aistudio,google.com. Send in your favorite prompts, and we hope you like the responses.
The 9B and 27B versions are available for Ollama: https://ollama.com/library/gemma2
https://aistudio.google.com/app/prompts/new_chat?model=gemma...
So far it seems pretty strong for its size.
Hello (again) from the Gemma team! We are quite excited to push this release out and happy to answer any questions! Opinions are our own and not of Google DeepMind.
Will gemma2 be available through gemma.cpp? https://github.com/google/gemma.cpp
Shouldn't this (2.6B/9B) be compared with Microsoft's Phi-3 mini (3.8B) instead of Mistral and Llama-3? (table 13 on page 7) vs https://arxiv.org/pdf/2404.14219 (page 6, quite better in general) The report on knowledge distillation training is interesting, though.
The 2.6B would get stomped by Phi-3, so there's no comparison.
Fair enough. 2.6B vs. 3.8B is a fairly substantial size difference thats hard to intuit when its 2.6 vs 3.8 versus 2,600,000,000 and 3,800,000,000.
But then we get what I'm going to "parameter creep": Mistral 7B vs. Llama 8B vs. Gemma 9B. I worried after Llama 3 went 8B that we'd start seeing games with parameters, but, thought I was being silly.
Nice! Can you explain what you mean by "simulate training beyond the number of available tokens"? Why does using distillation from a larger model simulate training with more tokens?
There are two new chatbots on Chatbot Arena, called "late-june-chatbot" and "im-just-another-late-june-chatbot". Both of them report that they are Gemma if you ask. I'm assuming it's these two models, but AFAIK there has been no official announcement.
Earlier quoted context omitted.
Your training input has the shape of (sequence length x batch size). If a lot of your samples are shorter than sequence length, as is usually the case, you will have a lot of padding tokens in the input, which is wasted compute. To compensate for that, you can pack multiple examples in the same sequence. This is there EOS and BOS come in, as they indicate to the model that the two parts of the sequence are not relate…
You can just do that my shaping the attention mask, no? That also gives you an actual guarantee that no information is leaked between conversations.
Hello (again) from the Gemma team! We are quite excited to push this release out and happy to answer any questions! Opinions are our own and not of Google DeepMind.