Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemma 2: Improving Open Language Models at a Practical Size [pdf]

storage.googleapis.com

Post reply on HN
Print view View original

151–160 of 183 posts

Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]

#151

Post by aashu_dwivedi »

bayesianbot wrote: post #143
chown wrote: post #42

This is a great release! If you are looking to try it locally with a great interface, I am working on an app [1] and I just pushed an update to support Gemma2. 1: https://msty.app

Looks cool even though closed source makes me wary. Trying to save Anthropic API key on Arch Linux doesn't do anything and there's a message "If you're experiencing problems saving API keys especially on Linux, contact Discord", if it's so common problem maybe you should have a link with possible fixes? Adding another Discord server and searching for answers for a question that clearly has been asked often enough fee…

What does closed source mean in this context? The weights are open and the model architecture has to be open for people to use it for inference.

Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]

#152

Post by astrange »

groby_b wrote: post #121
astrange wrote: post #111

Earlier quoted context omitted.

Model collapse was basically a coping idea made up by artists who were hoping AI image generators would all magically destroy themselves at some point; I don't think it was ever considered likely to happen. It does seem to be true that clean data works better than low quality data.

You're confusing it with data poisoning. Model collapse itself is(was?) a fairly serious research topic: https://arxiv.org/abs/2305.17493 We've by now reached a "probably not inevitable" - https://arxiv.org/abs/2404.01413 argues there's a finite upper bound to error - but I'd also point out that that paper assumes training data cardinality increases with the number of training generations and is strictly accumulative…

Oh, no, they definitely believe both are going to happen and ChatGPT is just going to stop working because it'll see itself on the internet. It goes with the common belief that LLMs learn from what you type into them.

> To a first order, that means you better have a pre-2022 dataset to get started, and have archived it well.

I think that will always be available, or at least, a dataset with the distribution you want will be available.

Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]

#153

Post by janwas »

thot_experiment wrote: post #142
luke-stanley wrote: post #70

Earlier quoted context omitted.

Actually for the 9B model, this has 4-bit quantised weights (and others): https://huggingface.co/bartowski/gemma-2-9b-it-GGUF Still no 27B 4-bit GGUF quants on HF yet! I'm monitoring this search: https://huggingface.co/models?library=gguf&sort=trending&sea...

I'm curious about the quantization quality claims in the table there. Is this a Gemma 2 specific thing (more subtlety in the weights somehow?). In my testing and testing I've seen elsewhere at least for llama3 8B (and some less rigorous testing with other models) q_8 -> q4_K_M are basically indistinguishable from one another?

Yes, PPL and certain benchmarks do not detect differences from quantization. But recent work gives cause for concern, e.g., https://arxiv.org/pdf/2310.01382, https://arxiv.org/pdf/2405.18137.

Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]

#154

Post by onel »

aashu_dwivedi wrote: post #151
bayesianbot wrote: post #143

Earlier quoted context omitted.

Looks cool even though closed source makes me wary. Trying to save Anthropic API key on Arch Linux doesn't do anything and there's a message "If you're experiencing problems saving API keys especially on Linux, contact Discord", if it's so common problem maybe you should have a link with possible fixes? Adding another Discord server and searching for answers for a question that clearly has been asked often enough fee…

What does closed source mean in this context? The weights are open and the model architecture has to be open for people to use it for inference.

I think he was referring to Msty which is closed-source

Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]

#156

Post by imjonse »

raffraffraff wrote: post #155

> We use the same data filtering techniques as Gemma 1. Specifically, we filter the pre- training dataset to reduce the risk of unwanted or unsafe utterances. Hmmm. I'd love to know what qualifies as "unsafe".

It will refuse to describe the process of making napalm using only double entendres.

Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]

#157

Post by dandanua »

canyon289 wrote: post #53
alekandreev wrote: post #6

Hello (again) from the Gemma team! We are quite excited to push this release out and happy to answer any questions! Opinions are our own and not of Google DeepMind.

I also work at Google and on Gemma (so same disclaimers) You can try 27b at www.aistudio,google.com. Send in your favorite prompts, and we hope you like the responses.

Why is AIStudio not available in Ukraine? I have no problem with using Gemini web UI or other LLM providers from Ukraine, but this Google API constrain is strange.

Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]

#158

Post by Flumio »

luke-stanley wrote: post #127
Flumio wrote: post #119

Earlier quoted context omitted.

Google does plenty of ux studies on gcp. I took part in at least 3 of them. I'm also not sure if I understand your problem with pricing? Depending on what you do with it, it's not just an LLM. It actually started before llms. Pricing for image classification and other features are completely different products like an LLM.

They should do a whole lot more then! Ideally they'd have effective impact. It's a busy mess on GCP. If they wanted to compete well, they should do much better with UX design, especially for onboarding. Compare how easy setting up a Mistral account is with GCP to do some generative LLM in a Python script. GCP is a maze. Did you make an account to reply to this? I'm curious what you do with GCP? Are you a heavy user?

I create new accounts because I use hn too much.

I use gcp professional every day and always found it quite intuitive.

Did plenty of image classification with vertex ai too

Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]

#159

Post by jmorgan »

thot_experiment wrote: post #149
jmorgan wrote: post #148

Earlier quoted context omitted.

Sorry about this – working on fixing the issue with hitting the context limit. Gemma 2 supports a 8192 context limit – which can be selected if you provide the `num_ctx` parameter in the API or via `ollama run` with `/set parameter num_ctx 8192`

Thanks! If you have a moment can you give me a quick explainer on what happens when you hit the context limit in ollama? I had assumed that ollama would just trunc the context to whatever is set in the model, but I guess this isn't the case?

Currently when the context limit is hit, there's a halving of the context window (or a "context shift") to allow inference to continue – this is helpful for smaller (e.g. 1-2k) context windows.

However, not all models (especially newer ones) respond well to this, which makes sense. We're working on changing the behavior in Ollama's API to be more similar to OpenAI, Anthropic and similar APIs so that when the context limit is hit, the API returns a "limit" finish/done reason. Hope this is helpful!

Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]

#160

Post by rldjbpin »

for me, the 2.5B model for gemma (now 1) was very interesting as that was the first major offering at this size level.

for basic llm tasks that most people would use on their daily lives (simple rag on your own data), it did the job for the most part (unless you need a lot of context maybe).

on paper the newer one shows significant improvement with slightly larger size, but i hope HumanEval regression is not going to matter for most people.

Post reply on HN