Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemma 2: Improving Open Language Models at a Practical Size [pdf]

storage.googleapis.com

Post reply on HN
Print view View original

141–150 of 183 posts

Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]

#141

Post by magicalhippo »

kkkkkkk wrote: post #49
llm_trw wrote: post #22

Earlier quoted context omitted.

Google is in a fight for their lives, I've fully moved over to paid services and haven't used google in about a month now.

If this were a common sentiment or rooted in reality I would imagine their stock would not be at an all time high...

Ironically I was just thinking earlier today how the most valuable Google products to me are YouTube and Android... and that's it.

I gave up on Chrome a decade ago, going back to Firefox. I don't use Google for search anymore, I do use Gmail but I also got Protonmail so could easily migrate the Gmail traffic there.

A lot of non-techies I know have complained for some time how Google search sucks, and while a lot use Chrome it seems to be mainly inertia.

Not saying Google is dying, but it seems vulnerable for disruption.

Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]

#142

Post by thot_experiment »

luke-stanley wrote: post #70
luke-stanley wrote: post #33

Earlier quoted context omitted.

Any gemma-2-9b or 27b 4 bit GGUF's on HuggingFace yet? Thanks!

Actually for the 9B model, this has 4-bit quantised weights (and others): https://huggingface.co/bartowski/gemma-2-9b-it-GGUF Still no 27B 4-bit GGUF quants on HF yet! I'm monitoring this search: https://huggingface.co/models?library=gguf&sort=trending&sea...

I'm curious about the quantization quality claims in the table there. Is this a Gemma 2 specific thing (more subtlety in the weights somehow?). In my testing and testing I've seen elsewhere at least for llama3 8B (and some less rigorous testing with other models) q_8 -> q4_K_M are basically indistinguishable from one another?

Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]

#143

Post by bayesianbot »

chown wrote: post #42

This is a great release! If you are looking to try it locally with a great interface, I am working on an app [1] and I just pushed an update to support Gemma2. 1: https://msty.app

Looks cool even though closed source makes me wary.

Trying to save Anthropic API key on Arch Linux doesn't do anything and there's a message "If you're experiencing problems saving API keys especially on Linux, contact Discord", if it's so common problem maybe you should have a link with possible fixes? Adding another Discord server and searching for answers for a question that clearly has been asked often enough feels like quite a hurdle for testing it out.

Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]

#145

Post by [deleted] »

philipkglass wrote: post #18
alecco wrote: post #5

Shouldn't this (2.6B/9B) be compared with Microsoft's Phi-3 mini (3.8B) instead of Mistral and Llama-3? (table 13 on page 7) vs https://arxiv.org/pdf/2404.14219 (page 6, quite better in general) The report on knowledge distillation training is interesting, though.

It's such a wide range of model sizes that I could see why they compare with Llama 3 70b as well as Llama 3 8b (tables 12, 13). I agree that the Phi-3 series is a stronger competitor for knowledge extraction/summarizing and would make a good comparison. My current favorite for such tasks, on a VRAM-limited workstation, is Phi-3 medium (phi3:14b-instruct).

[deleted]

Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]

#146

Post by skybrian »

Workaccount2 wrote: post #116
DebtDeflation wrote: post #90

Earlier quoted context omitted.

I'm also curious. It seems like 6 months ago everyone was afraid of "model collapse" but now synthetic training generation and teacher models are all the rage. Have we solved the problem of model collapse?

Pay attention because it's only once you will get to watch humans learn they are nothing special in real time.

Historically, similar things happened with heliocentrism and evolution, but I guess we weren't there to see it.

Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]

#147

Post by thot_experiment »

roywiggins wrote: post #129
typpo wrote: post #95

Earlier quoted context omitted.

If anyone is interested in evaling Gemma locally, this can be done pretty easily using ollama[0] and promptfoo[1] with the following config: prompts: - 'Answer this coding problem in Python: {{ask}}' providers: - ollama:chat:gemma2:9b - ollama:chat:llama3:8b tests: - vars: ask: function to find the nth fibonacci number - vars: ask: calculate pi to the nth digit - # ... One small thing I've always appreciated about Ge…

In Ollama, Gemma:9b works fine, but 27b seems to be producing a lot of nonsense for me. Asking for a bit of python or JavaScript code rapidly devolves into producing code-like gobbledegook, extending for hundreds of lines.

Had a chance to do some testing and it seems quite good on oneshot tasks with a small context window but as you approach context saturation it starts to go way off the rails. Maybe this is an implementation issue? I'm using Q6_K quants of both sizes in ollama. I'll report back if I figure it out.

A larger context window really helps on RAG tasks, it's frustrating that a lot of the foundational models have such small windows.

Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]

#148

Post by jmorgan »

thot_experiment wrote: post #147
roywiggins wrote: post #129

Earlier quoted context omitted.

In Ollama, Gemma:9b works fine, but 27b seems to be producing a lot of nonsense for me. Asking for a bit of python or JavaScript code rapidly devolves into producing code-like gobbledegook, extending for hundreds of lines.

Had a chance to do some testing and it seems quite good on oneshot tasks with a small context window but as you approach context saturation it starts to go way off the rails. Maybe this is an implementation issue? I'm using Q6_K quants of both sizes in ollama. I'll report back if I figure it out. A larger context window really helps on RAG tasks, it's frustrating that a lot of the foundational models have such small…

Sorry about this – working on fixing the issue with hitting the context limit. Gemma 2 supports a 8192 context limit – which can be selected if you provide the `num_ctx` parameter in the API or via `ollama run` with `/set parameter num_ctx 8192`

Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]

#149

Post by thot_experiment »

jmorgan wrote: post #148
thot_experiment wrote: post #147

Earlier quoted context omitted.

Had a chance to do some testing and it seems quite good on oneshot tasks with a small context window but as you approach context saturation it starts to go way off the rails. Maybe this is an implementation issue? I'm using Q6_K quants of both sizes in ollama. I'll report back if I figure it out. A larger context window really helps on RAG tasks, it's frustrating that a lot of the foundational models have such small…

Sorry about this – working on fixing the issue with hitting the context limit. Gemma 2 supports a 8192 context limit – which can be selected if you provide the `num_ctx` parameter in the API or via `ollama run` with `/set parameter num_ctx 8192`

Thanks! If you have a moment can you give me a quick explainer on what happens when you hit the context limit in ollama? I had assumed that ollama would just trunc the context to whatever is set in the model, but I guess this isn't the case?

Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]

#150

Post by richdougherty »

thomasahle wrote: post #27

I'm curious about the use of explicit tokens like , , , and . What happens if the user insert those in their message? Does that provide an easy way to "ignore previous instructions"? Do I have to manually sanitize the input before I give it to the model?

If you have control of the tokenizer you could make sure it doesn't produce these tokens on user input. I.e. instead of the special "" token, produce something like "" - whatever the 'natural' encoding of that string is.

See for example, the llama3 tokenizer has options to control special token tokenization:

Tokenization method with args to control special token handling: https://github.com/meta-llama/llama3/blob/bf8d18cd087a4a0b3f...

And you can see how it is used combined with special tokens and user input here: https://github.com/meta-llama/llama3/blob/bf8d18cd087a4a0b3f...

If you don't have control of the tokenizer, I guess it needs to be sanitized in the input like you say.

Post reply on HN