Gemma 2: Improving Open Language Models at a Practical Size [pdf] • Hacker News Board

Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]

#181

Post by 0x7cfe » Mon, Jul 01, 2024, 6:45 AM UTC

bugglebeetle wrote: post #135

roywiggins wrote: post #129

Earlier quoted context omitted.

In Ollama, Gemma:9b works fine, but 27b seems to be producing a lot of nonsense for me. Asking for a bit of python or JavaScript code rapidly devolves into producing code-like gobbledegook, extending for hundreds of lines.

The tokenizer in llama.cpp probably needs fixing then or it has some other bug.

Definitely. I tried gemma2:27B model with phrases like "translate the following sentence to language X" and it even failed to understand the task and spat out completely irrelevant things, like math formulas.

OTOH, smaller model did it perfectly.

Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]

#182

Post by lhl » Mon, Jul 01, 2024, 6:41 PM UTC

occamrazor wrote: post #174

lhl wrote: post #169

Earlier quoted context omitted.

Just saw this, might get lost in the noise, but just for posterity, apparently the Gemma 2 models were specifically RL’d to index on Chat Arena performance: https://x.com/natolambert/status/1806384821826109597 (Relevant sections of the paper highlighted.)

On prompts only, with answers presumably from the teacher model (Gemini). It was not trained or RLHFd on Arena replies or user preferences.

Yes, answers were distilled from a much stronger model. On the one hand, you can argue that this is exactly what the LMSYS, WildBench etc datasets are for (to improve performance/alignment on real-world use cases), but on the other hand, it's clear that training on the questions (most of which are repeatedly used by the (largely non-representative of general population) users of the ChatArena for comparing/testing models) makes ChatArena ELO less useful as a model comparison tool and artificially elevates Gemma 2's ChatArena score relative to its OOD performance.

At the end of the day, by optimizing for leaderboard scoring, it makes the leaderboard ranking less useful as a benchmark (Goodhart's law strikes again). The Gemma team obviously isn't the only one doing it, but it's important to be clear-eyed about the consequences.

Re: Gemma 2: Improving Open Language Models at a Practical Size [pdf]

#183

Post by Grimblewald » Tue, Jul 02, 2024, 2:29 AM UTC

smcleod wrote: post #180

Grimblewald wrote: post #179

Earlier quoted context omitted.

8K is a sizable window, sure larger 'exists' but also advertised context windows and functional context windows are not the same thing. I would rather a model that can 'only' handle 8k tokens but handles 8k as well as it handles 1k compared to a model that 'can' handle 32k, but realistically, output for contexts beyond 1k are garbage.

Deepseek Coder v2 and Qwen2 are both great at 32k context. Can’t tell the difference between those models at 8k and 32k fully utilised. The difference in quality between them and 8k models when doing codegen is night and day. Not to mention that many of the little 8k models also have sliding window at 4k which essentially makes them 4k models.

I agree, they're exceptional models, however this can not be said of all models that boast a large context window.