21–30 of 183 posts
Hello (again) from the Gemma team! We are quite excited to push this release out and happy to answer any questions! Opinions are our own and not of Google DeepMind.
How is Gemma-2 licensed?
I gave up hope on r"Gem[ma|ini]" long time ago. I don't believe that Google can't produce good LLMs because of its massive company size; Microsoft is also a giant company (more market cap than Google) but it keeps surprising us with the ϕ models. I think Google just lacks the vision to understand what makes a good LLM. Theoretical contributions by research teams are valuable, but the real-world is built around engine…
I wonder if Google is making Deepmind people switch from their cool original research to doing LLMs like everybody else. Having their scale in money and data, I would hire new teams of engineers who want to do LLMs and leave the Deepmind researchers do their thing. Not killing the goose that lays golden eggs.
And when we continue fine-tune.how much and what type of data we learn it on, I'm pretty sure for a smart agent who is not a knowledgeable expert but primarily a agent (understand what and how) this will get smaller and easier to run everywhere.
Earlier quoted context omitted.
think about training.
I suppose it would act as a concrete separator when instruct tuning, but lots of prompt templates don't use it, especially older ones like Alpaca. Maybe it leads to more overall coherence?
If you have a bunch of small prompts/answers, you can fit them into bigger batches if you use start/stop tokens.
I gave up hope on r"Gem[ma|ini]" long time ago. I don't believe that Google can't produce good LLMs because of its massive company size; Microsoft is also a giant company (more market cap than Google) but it keeps surprising us with the ϕ models. I think Google just lacks the vision to understand what makes a good LLM. Theoretical contributions by research teams are valuable, but the real-world is built around engine…
Claude's reign has begun, and I'd say it has a solid enough lead for at least another two weeks of dominance before it's dethroned.
So it's twice the size of phi 3 and considerably worse? What am I missing
Small models are never going to be generalists, so having several small models allows you to pick the one that best fits your needs.
Do I have to manually sanitize the input before I give it to the model?
I gave up hope on r"Gem[ma|ini]" long time ago. I don't believe that Google can't produce good LLMs because of its massive company size; Microsoft is also a giant company (more market cap than Google) but it keeps surprising us with the ϕ models. I think Google just lacks the vision to understand what makes a good LLM. Theoretical contributions by research teams are valuable, but the real-world is built around engine…
Phi-3 blow this out of the water. Benchmark | Gemma 2 (9B) | Phi-3 Small (7B) -----------------------------|----------------|------------------- MMLU (5-Shot) | 63.6 | 75.7 HellaSwag (5-Shot) | 49.8 | 77.0 ANLI (7-Shot) | 48.7 | 58.1 GSM-8K (8-Shot; CoT) | 59.8 | 89.6 MedQA (2-Shot) | 49.6 | 65.4 AGIEval (0-Shot) | 42.1 | 45.1 TriviaQA (5-Shot) | 72.3 | 58.1 Arc-C (10-Shot) | 78.3 | 90.7 Arc-E (10-Shot) | 91.4 | 97.0…
Phi-3 blow this out of the water. Benchmark | Gemma 2 (9B) | Phi-3 Small (7B) -----------------------------|----------------|------------------- MMLU (5-Shot) | 63.6 | 75.7 HellaSwag (5-Shot) | 49.8 | 77.0 ANLI (7-Shot) | 48.7 | 58.1 GSM-8K (8-Shot; CoT) | 59.8 | 89.6 MedQA (2-Shot) | 49.6 | 65.4 AGIEval (0-Shot) | 42.1 | 45.1 TriviaQA (5-Shot) | 72.3 | 58.1 Arc-C (10-Shot) | 78.3 | 90.7 Arc-E (10-Shot) | 91.4 | 97.0…