This is a great release! If you are looking to try it locally with a great interface, I am working on an app [1] and I just pushed an update to support Gemma2. 1: https://msty.app
Any plans on adding this to Chocolatey for Windows download?
Gemma 2: Improving Open Language Models at a Practical Size [pdf]
131–140 of 183 posts
Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]
#132Earlier quoted context omitted.
Phi-3 does well in benchmarks but underperforms IRL; for example, Phi-3-Medium gets beaten badly by Llama-3-8b on the LMSYS Chatbot Arena despite doing better on benchmarks. Gemma's performance if anything seems understated on benchmarks: the 27b is currently ahead of Llama3-70b on the Chatbot Arena leaderboard.
I suspect Phi-3 is not robust to normal human input like typos and strange grammar since it's only trained on filtered "high quality" tokens and synthetic data. Since it doesn't need to waste a ton of parameters learning how to error correct input, it's much smarter on well curated benchmarks compared to its weight class. However, it can't operate out of distribution at all.
Personally vibe checking Phi-3-Medium is worse in my experience, no matter how well you spell — it just isn't good at all compared to Llama3-8b, despite being significantly larger in param count. I suspect the "high quality tokens" were "high quality" in the sense that they resembled tokens one might encounter in benchmarks, and not "high quality" in the sense of representing human-like input/output.
Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]
#133Do we know if Gemma models are fundamentally different from the ones hosted as Gemini? Gemini 1.5 flash seems to produce good results for the price and performance.
Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]
#134It's exceptionally strong. In LMSys Chatbot Arena, the 27B version scores above LLama-3-70B, at the level of OpenAI GPT-4 and Claude-3 Sonnet!
I think this is just due to better non-English training data.
It's 15 ELO under Llama-3-70B on english hard prompts and 41 ELO under Llama-3-70B (the latter is actually stat sig) for general English.
Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]
#135Earlier quoted context omitted.
If anyone is interested in evaling Gemma locally, this can be done pretty easily using ollama[0] and promptfoo[1] with the following config: prompts: - 'Answer this coding problem in Python: {{ask}}' providers: - ollama:chat:gemma2:9b - ollama:chat:llama3:8b tests: - vars: ask: function to find the nth fibonacci number - vars: ask: calculate pi to the nth digit - # ... One small thing I've always appreciated about Ge…
In Ollama, Gemma:9b works fine, but 27b seems to be producing a lot of nonsense for me. Asking for a bit of python or JavaScript code rapidly devolves into producing code-like gobbledegook, extending for hundreds of lines.
The tokenizer in llama.cpp probably needs fixing then or it has some other bug.
Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]
#136It's exceptionally strong. In LMSys Chatbot Arena, the 27B version scores above LLama-3-70B, at the level of OpenAI GPT-4 and Claude-3 Sonnet!
I'd encourage people to test for themselves (and to let the Chatbot Arena scores to settle) before getting caught up in too much hype. I just did a personal eval and I found gemma-2-27b-it (tested on AI Studio) performed far worse in my testing than Llama 3 70B, especially for reasoning and basic world understanding queries.
Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]
#137Earlier quoted context omitted.
If anyone is interested in evaling Gemma locally, this can be done pretty easily using ollama[0] and promptfoo[1] with the following config: prompts: - 'Answer this coding problem in Python: {{ask}}' providers: - ollama:chat:gemma2:9b - ollama:chat:llama3:8b tests: - vars: ask: function to find the nth fibonacci number - vars: ask: calculate pi to the nth digit - # ... One small thing I've always appreciated about Ge…
In Ollama, Gemma:9b works fine, but 27b seems to be producing a lot of nonsense for me. Asking for a bit of python or JavaScript code rapidly devolves into producing code-like gobbledegook, extending for hundreds of lines.
27b is working fine for me, hosted on ollama w/ continue.dev in VSCode.
Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]
#138Earlier quoted context omitted.
Any gemma-2-9b or 27b 4 bit GGUF's on HuggingFace yet? Thanks!
Actually for the 9B model, this has 4-bit quantised weights (and others): https://huggingface.co/bartowski/gemma-2-9b-it-GGUF Still no 27B 4-bit GGUF quants on HF yet! I'm monitoring this search: https://huggingface.co/models?library=gguf&sort=trending&sea...
Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]
#139It's exceptionally strong. In LMSys Chatbot Arena, the 27B version scores above LLama-3-70B, at the level of OpenAI GPT-4 and Claude-3 Sonnet!
I'd encourage people to test for themselves (and to let the Chatbot Arena scores to settle) before getting caught up in too much hype. I just did a personal eval and I found gemma-2-27b-it (tested on AI Studio) performed far worse in my testing than Llama 3 70B, especially for reasoning and basic world understanding queries.
Same. I tried 27B and found it to be not even close to llama3-70b.
Even llama-8b did better in some of my tests than Gemma 27b.
Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]
#140It's exceptionally strong. In LMSys Chatbot Arena, the 27B version scores above LLama-3-70B, at the level of OpenAI GPT-4 and Claude-3 Sonnet!
I'd encourage people to test for themselves (and to let the Chatbot Arena scores to settle) before getting caught up in too much hype. I just did a personal eval and I found gemma-2-27b-it (tested on AI Studio) performed far worse in my testing than Llama 3 70B, especially for reasoning and basic world understanding queries.
I also prefer to use "Coding" or "Hard Prompts (Overall)" instead of default "Overall" in Chatbot Arena scores to determine the actual performance level of LLMs. Seems much more align to my vibe test in terms reasoning. I guess the "Overall" contains a lot of creative tasks, which is not what I use the most in the daily tasks.