Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemma 2: Improving Open Language Models at a Practical Size [pdf]

storage.googleapis.com

Post reply on HN
Print view View original

121–130 of 183 posts

Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]

#121

Post by groby_b »

astrange wrote: post #111
DebtDeflation wrote: post #90

Earlier quoted context omitted.

I'm also curious. It seems like 6 months ago everyone was afraid of "model collapse" but now synthetic training generation and teacher models are all the rage. Have we solved the problem of model collapse?

Model collapse was basically a coping idea made up by artists who were hoping AI image generators would all magically destroy themselves at some point; I don't think it was ever considered likely to happen. It does seem to be true that clean data works better than low quality data.

You're confusing it with data poisoning.

Model collapse itself is(was?) a fairly serious research topic: https://arxiv.org/abs/2305.17493

We've by now reached a "probably not inevitable" - https://arxiv.org/abs/2404.01413 argues there's a finite upper bound to error - but I'd also point out that that paper assumes training data cardinality increases with the number of training generations and is strictly accumulative.

To a first order, that means you better have a pre-2022 dataset to get started, and have archived it well.

but it's probably fair to say current SOTA is still more or less "it's neither impossible nor inevitable".

Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]

#122

Post by lhl »

ankeshanand wrote: post #78
luke-stanley wrote: post #60

Earlier quoted context omitted.

It's fairly easy to pay OpenAI or Mistral money to use their API's. Figuring out how Google Cloud Vertex works and how it's billed is more complicated. Azure and AWS are similar in how complex they are to use for this. Could Google Cloud please provide an OpenAI compatible API and service? I know it's a different department. But it'd make using your models way easier. It often feels like Google Cloud has no UX or end…

If you're an individual developer and not an enterprise, just go straight to Google AIStudio or GeminiAPI instead: https://aistudio.google.com/app/apikey . It's dead simple getting an API key and calling with a rest client.

Sadly, while gemma-2-27b-it is available (as a Preview model) on the AI Studio playground, it didn't show up via API on list_models() for me.

Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]

#123

Post by zone411 »

np_space wrote: post #103
alekandreev wrote: post #6

Hello (again) from the Gemma team! We are quite excited to push this release out and happy to answer any questions! Opinions are our own and not of Google DeepMind.

Are Gemma-2 models available via API yet? Looks to me like it's not yet on vertexai

"Soon" https://x.com/LechMazur/status/1806366744706998732

Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]

#124

Post by kouteiheika »

refulgentis wrote: post #56
alecco wrote: post #5

Shouldn't this (2.6B/9B) be compared with Microsoft's Phi-3 mini (3.8B) instead of Mistral and Llama-3? (table 13 on page 7) vs https://arxiv.org/pdf/2404.14219 (page 6, quite better in general) The report on knowledge distillation training is interesting, though.

Picking up from there: The games in this paper and model are annoying. The 2.6B would get stomped by Phi-3, so there's no comparison. Fair enough. 2.6B vs. 3.8B is a fairly substantial size difference thats hard to intuit when its 2.6 vs 3.8 versus 2,600,000,000 and 3,800,000,000. But then we get what I'm going to "parameter creep": Mistral 7B vs. Llama 8B vs. Gemma 9B. I worried after Llama 3 went 8B that we'd start…

There was no parameter creep with Llama. Llama 8B is actually a ~7B model comparable to Mistral 7B if you strip away multilingual embeddings and match what Mistral 7B supports.

Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]

#125

Post by anxman »

alekandreev wrote: post #65
luke-stanley wrote: post #60

Earlier quoted context omitted.

It's fairly easy to pay OpenAI or Mistral money to use their API's. Figuring out how Google Cloud Vertex works and how it's billed is more complicated. Azure and AWS are similar in how complex they are to use for this. Could Google Cloud please provide an OpenAI compatible API and service? I know it's a different department. But it'd make using your models way easier. It often feels like Google Cloud has no UX or end…

Happy to pass on any feedback to our Google Cloud friends. :)

I also hate the billing. It feels like configuring AWS more than calling APIs.

Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]

#126

Post by renewiltord »

chown wrote: post #42

This is a great release! If you are looking to try it locally with a great interface, I am working on an app [1] and I just pushed an update to support Gemma2. 1: https://msty.app

What the heck, this looks cool! How have I missed it. Gonna give it a whirl.

Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]

#127

Post by luke-stanley »

Flumio wrote: post #119
luke-stanley wrote: post #107

Earlier quoted context omitted.

Good to know about this API preview. Hopefully the billing problem and UI maze of Vertex AI can be sorted too?

Google does plenty of ux studies on gcp. I took part in at least 3 of them. I'm also not sure if I understand your problem with pricing? Depending on what you do with it, it's not just an LLM. It actually started before llms. Pricing for image classification and other features are completely different products like an LLM.

They should do a whole lot more then! Ideally they'd have effective impact. It's a busy mess on GCP. If they wanted to compete well, they should do much better with UX design, especially for onboarding. Compare how easy setting up a Mistral account is with GCP to do some generative LLM in a Python script. GCP is a maze. Did you make an account to reply to this? I'm curious what you do with GCP? Are you a heavy user?

Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]

#128

Post by [deleted] »

typpo wrote: post #95
aubanel wrote: post #73

It's exceptionally strong. In LMSys Chatbot Arena, the 27B version scores above LLama-3-70B, at the level of OpenAI GPT-4 and Claude-3 Sonnet!

If anyone is interested in evaling Gemma locally, this can be done pretty easily using ollama[0] and promptfoo[1] with the following config: prompts: - 'Answer this coding problem in Python: {{ask}}' providers: - ollama:chat:gemma2:9b - ollama:chat:llama3:8b tests: - vars: ask: function to find the nth fibonacci number - vars: ask: calculate pi to the nth digit - # ... One small thing I've always appreciated about Ge…

[deleted]

Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]

#129

Post by roywiggins »

typpo wrote: post #95
aubanel wrote: post #73

It's exceptionally strong. In LMSys Chatbot Arena, the 27B version scores above LLama-3-70B, at the level of OpenAI GPT-4 and Claude-3 Sonnet!

If anyone is interested in evaling Gemma locally, this can be done pretty easily using ollama[0] and promptfoo[1] with the following config: prompts: - 'Answer this coding problem in Python: {{ask}}' providers: - ollama:chat:gemma2:9b - ollama:chat:llama3:8b tests: - vars: ask: function to find the nth fibonacci number - vars: ask: calculate pi to the nth digit - # ... One small thing I've always appreciated about Ge…

In Ollama, Gemma:9b works fine, but 27b seems to be producing a lot of nonsense for me. Asking for a bit of python or JavaScript code rapidly devolves into producing code-like gobbledegook, extending for hundreds of lines.

Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]

#130

Post by aubanel »

resource_waste wrote: post #117
aubanel wrote: post #73

It's exceptionally strong. In LMSys Chatbot Arena, the 27B version scores above LLama-3-70B, at the level of OpenAI GPT-4 and Claude-3 Sonnet!

Do we believe that? I've been told Google's AI was going to be great 4 times now, and its consistently #4 behind OpenAI, Facebook, and Claude.

LMSys Chatbot Arena is a crowd-sourced ranking with an ELO system: basically users a presented with 2 hidden models, they get the answers of the 2 models when presenting their request, and they vote which one performed bests, which realized one marche and updates the ELO scores. This is the closest thing that we have to a gold truth for LLM evaluation: and Gemma2-27B performs extremely well in Chatbot Arena ELO.
Post reply on HN