Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemma 2: Improving Open Language Models at a Practical Size [pdf]

storage.googleapis.com

Post reply on HN
Print view View original

161–170 of 183 posts

Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]

#161

Post by moffkalast »

magicalhippo wrote: post #141
kkkkkkk wrote: post #49

Earlier quoted context omitted.

If this were a common sentiment or rooted in reality I would imagine their stock would not be at an all time high...

Ironically I was just thinking earlier today how the most valuable Google products to me are YouTube and Android... and that's it. I gave up on Chrome a decade ago, going back to Firefox. I don't use Google for search anymore, I do use Gmail but I also got Protonmail so could easily migrate the Gmail traffic there. A lot of non-techies I know have complained for some time how Google search sucks, and while a lot use…

Is it really possible to even disrupt Youtube? It's been a constant in our lives for the past 20 years and is basically a historical record by now. By a rough estimate, they have to keep buying over 1% of the total world production of HDD drives just to stay on top of the new data being uploaded. Google has completely destroyed it, placing more ads than videos on it, making it unusable without an adblocker and people still use it, it's that core to everyone's lives. It's like a public utility.

Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]

#162

Post by mdrzn »

moffkalast wrote: post #97
alekandreev wrote: post #81

Earlier quoted context omitted.

To quote Ludovic Peran, our amazing safety lead: Literature has identified self-proliferation as dangerous capability of models, and details about how to define it and example of form it can take have been openly discussed by GDM ( https://arxiv.org/pdf/2403.13793 ). Current Gemma 2 models' success rate to end-to-end challenges is null (0 out 10), so the capabilities to perform such tasks are currently limited.

Turns out LLM alignment is super easy, barely an inconvenience.

Wow wow wow.... wow.

Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]

#163

Post by luke-stanley »

Flumio wrote: post #158
luke-stanley wrote: post #127

Earlier quoted context omitted.

They should do a whole lot more then! Ideally they'd have effective impact. It's a busy mess on GCP. If they wanted to compete well, they should do much better with UX design, especially for onboarding. Compare how easy setting up a Mistral account is with GCP to do some generative LLM in a Python script. GCP is a maze. Did you make an account to reply to this? I'm curious what you do with GCP? Are you a heavy user?

I create new accounts because I use hn too much. I use gcp professional every day and always found it quite intuitive. Did plenty of image classification with vertex ai too

Why would you make new accounts because you use HN too much? Doesn't make sense to me. Anyhow if you use GCP every day, you're going to have learned it's weird clunky behaviour. GCP's main problem is that they've steadily become a sprawling mess of complexity, which is in big contrast to quite a few LLM specific cloud services that are happy to take peoples money without extra complexity?

Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]

#164

Post by logankilpatrick »

bapcon wrote: post #69
luke-stanley wrote: post #60

Earlier quoted context omitted.

It's fairly easy to pay OpenAI or Mistral money to use their API's. Figuring out how Google Cloud Vertex works and how it's billed is more complicated. Azure and AWS are similar in how complex they are to use for this. Could Google Cloud please provide an OpenAI compatible API and service? I know it's a different department. But it'd make using your models way easier. It often feels like Google Cloud has no UX or end…

I have to agree with all of this. I tried switching to Gemini, but the lack of clear billing/quotas, horrible documentation, and even poor implementation of status codes on failed requests have led me to stick with OpenAI. I don't know who writes Google's documentation or does the copyediting for their console, but it is hard to adapt. I have spent hours troubleshooting, only to find out it's because the documentatio…

We are working hard to improve this across ai.google.dev (Gemini API), Hang tight!

Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]

#165

Post by alecco »

refulgentis wrote: post #56
alecco wrote: post #5

Shouldn't this (2.6B/9B) be compared with Microsoft's Phi-3 mini (3.8B) instead of Mistral and Llama-3? (table 13 on page 7) vs https://arxiv.org/pdf/2404.14219 (page 6, quite better in general) The report on knowledge distillation training is interesting, though.

Picking up from there: The games in this paper and model are annoying. The 2.6B would get stomped by Phi-3, so there's no comparison. Fair enough. 2.6B vs. 3.8B is a fairly substantial size difference thats hard to intuit when its 2.6 vs 3.8 versus 2,600,000,000 and 3,800,000,000. But then we get what I'm going to "parameter creep": Mistral 7B vs. Llama 8B vs. Gemma 9B. I worried after Llama 3 went 8B that we'd start…

Phi-3 3.8B seems to perform much better on almost every test than Gemma 2 9B. It is comparable.

Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]

#166

Post by Flumio »

luke-stanley wrote: post #163
Flumio wrote: post #158

Earlier quoted context omitted.

I create new accounts because I use hn too much. I use gcp professional every day and always found it quite intuitive. Did plenty of image classification with vertex ai too

Why would you make new accounts because you use HN too much? Doesn't make sense to me. Anyhow if you use GCP every day, you're going to have learned it's weird clunky behaviour. GCP's main problem is that they've steadily become a sprawling mess of complexity, which is in big contrast to quite a few LLM specific cloud services that are happy to take peoples money without extra complexity?

Not being logged in feels like a bigger hurdle to comment and check if someone responded to it.

It's a shitty solution to a stupid problem ;)

But I did mention that vertex AI is more than just hosting llms though

Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]

#167

Post by viridian »

imjonse wrote: post #156
raffraffraff wrote: post #155

> We use the same data filtering techniques as Gemma 1. Specifically, we filter the pre- training dataset to reduce the risk of unwanted or unsafe utterances. Hmmm. I'd love to know what qualifies as "unsafe".

It will refuse to describe the process of making napalm using only double entendres.

I don't understand the point of this sort of censorship when I can go to google, ask how to make napalm, and get a million results telling me to dissolve styrofoam in gasoline.

I've seen documentaries and science shows on cable TV that demonstrate basic facts like this, or how the IRA produced IEDs, or how molotov cocktails were made in the spanish civil war.

The information is beyond easy to access, and has been for decades.

Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]

#168

Post by jmcv »

chown wrote: post #42

This is a great release! If you are looking to try it locally with a great interface, I am working on an app [1] and I just pushed an update to support Gemma2. 1: https://msty.app

Just downloaded, looks great. Love the synced split view.

But I'm not seeing Gemma 2 or Claude 3.5 Sonnet even though it's announced on your landing page.

Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]

#169

Post by lhl »

aubanel wrote: post #73

It's exceptionally strong. In LMSys Chatbot Arena, the 27B version scores above LLama-3-70B, at the level of OpenAI GPT-4 and Claude-3 Sonnet!

Just saw this, might get lost in the noise, but just for posterity, apparently the Gemma 2 models were specifically RL’d to index on Chat Arena performance: https://x.com/natolambert/status/1806384821826109597

(Relevant sections of the paper highlighted.)

Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]

#170

Post by refulgentis »

alecco wrote: post #165
refulgentis wrote: post #56

Earlier quoted context omitted.

Picking up from there: The games in this paper and model are annoying. The 2.6B would get stomped by Phi-3, so there's no comparison. Fair enough. 2.6B vs. 3.8B is a fairly substantial size difference thats hard to intuit when its 2.6 vs 3.8 versus 2,600,000,000 and 3,800,000,000. But then we get what I'm going to "parameter creep": Mistral 7B vs. Llama 8B vs. Gemma 9B. I worried after Llama 3 went 8B that we'd start…

Phi-3 3.8B seems to perform much better on almost every test than Gemma 2 9B. It is comparable.

I agree.

The implication in my post is "if the reason was size, it's invalidated later"

Post reply on HN