Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemma: New Open Models

blog.google

Post reply on HN
Print view View original

171–180 of 543 posts

Re: Gemma: New Open Models

#171

Post by mochomocha »

sa-code wrote: post #156
tosh wrote: post #45

Benchmarks for Gemma 7B seem to be in the ballpark of Mistral 7B +-------------+----------+-------------+-------------+ | Benchmark | Gemma 7B | Mistral 7B | Llama-2 7B | +-------------+----------+-------------+-------------+ | MMLU | 64.3 | 60.1 | 45.3 | | HellaSwag | 81.2 | 81.3 | 77.2 | | HumanEval | 32.3 | 30.5 | 12.8 | +-------------+----------+-------------+-------------+ via https://mistral.ai/news/announcing-…

Thank you. I thought it was weird for them to release a 7B model and not mention Mistral in their release.

The technical report (linked in the 2nd paragraph of the blog post) mentions it, and compares against it: https://storage.googleapis.com/deepmind-media/gemma/gemma-re...

Re: Gemma: New Open Models

#172

Post by whimsicalism »

brucethemoose2 wrote: post #157
FergusArgyll wrote: post #153

Earlier quoted context omitted.

the real gold will be when this gets finetuned. (maybe by mistral...)

TBH the community has largely outrun Mistral's own finetuning. The 7B model in particular is such a popular target because its so practical to train.

Strong disagree - a Mistral fine tune of llama 70b was the top performing llama fine tune. They have lots of data the community simply does not.

Re: Gemma: New Open Models

#173

Post by smarterclayton »

gaogao wrote: post #51
gaogao wrote: post #44

Earlier quoted context omitted.

Mostly to boost research and commercial usage around JAX/Gemini is my read. Any internal research using Gemma is now more easily externally reproducible, external research and frameworks are easier to translate over, goodwill especially from researchers.

There's also less of a special sauce for text models itself these days with the propietary being more on the pre-training data and training stack (e.g. how to get 10k GPUs/TPUs running together smoothly). Multi-modal models (or adjacent like Sora) are less likely to be open sourced in the immediate term.

There is a lot of work to make the actual infrastructure and lower level management of lots and lots of GPUs/TPUs open as well - my team focuses on making the infrastructure bit at least a bit more approachable on GKE and Kubernetes.

https://github.com/GoogleCloudPlatform/ai-on-gke/tree/main

and

https://github.com/google/xpk (a bit more focused on HPC, but includes AI)

and

https://github.com/stas00/ml-engineering (not associated with GKE, but describes training with SLURM)

The actual training is still a bit of a small pool of very experienced people, but it's getting better. And every day serving models gets that much faster - you can often simply draft on Triton and TensorRT-LLM or vLLM and see significant wins month to month.

Re: Gemma: New Open Models

#174

Post by crossroadsguy »

lawxls wrote: post #159
tosh wrote: post #45

Benchmarks for Gemma 7B seem to be in the ballpark of Mistral 7B +-------------+----------+-------------+-------------+ | Benchmark | Gemma 7B | Mistral 7B | Llama-2 7B | +-------------+----------+-------------+-------------+ | MMLU | 64.3 | 60.1 | 45.3 | | HellaSwag | 81.2 | 81.3 | 77.2 | | HumanEval | 32.3 | 30.5 | 12.8 | +-------------+----------+-------------+-------------+ via https://mistral.ai/news/announcing-…

Honestly, this is more of a PR stunt to advertise the Google Dev ecosystem than a contribution to open-source. I'm not complaining, just calling it what it is. Barely an improvement over the 5-month-old Mistral model, with the same context length of 8k. And this is a release after their announcement of Gemini Pro 1.5, which had an exponential increase in context length.

That’s about the point of having a developer ecosystem, isn’t it?

Re: Gemma: New Open Models

#175

Post by kiraaa »

lawxls wrote: post #159
tosh wrote: post #45

Benchmarks for Gemma 7B seem to be in the ballpark of Mistral 7B +-------------+----------+-------------+-------------+ | Benchmark | Gemma 7B | Mistral 7B | Llama-2 7B | +-------------+----------+-------------+-------------+ | MMLU | 64.3 | 60.1 | 45.3 | | HellaSwag | 81.2 | 81.3 | 77.2 | | HumanEval | 32.3 | 30.5 | 12.8 | +-------------+----------+-------------+-------------+ via https://mistral.ai/news/announcing-…

Honestly, this is more of a PR stunt to advertise the Google Dev ecosystem than a contribution to open-source. I'm not complaining, just calling it what it is. Barely an improvement over the 5-month-old Mistral model, with the same context length of 8k. And this is a release after their announcement of Gemini Pro 1.5, which had an exponential increase in context length.

mistral 7b v0.2 supports 32k

Re: Gemma: New Open Models

#178

Post by brucethemoose2 »

whimsicalism wrote: post #172
brucethemoose2 wrote: post #157

Earlier quoted context omitted.

TBH the community has largely outrun Mistral's own finetuning. The 7B model in particular is such a popular target because its so practical to train.

Strong disagree - a Mistral fine tune of llama 70b was the top performing llama fine tune. They have lots of data the community simply does not.

Miqu was (allegedly) an internal continued pretrain Mistral did as a test, that was leaked as a GGUF.

Maybe its just semantics, it is technically a finetune... But to me theres a big difference between expensive "continuation training" (like Solar 10.7B or Mistral 70B) and a much less intense finetuning. The former is almost like releasing a whole new base model.

It would be awesome if Mistral did that with their data, but thats very different than releasing a Gemma Instruct finetune.

Re: Gemma: New Open Models

#180

Post by candiddevmike »

wantsanagent wrote: post #168

The utter bullshit of these licenses has got to stop. Do not, under any circumstances, consider using these commercially. "Google reserves the right to restrict (remotely or otherwise) usage of any of the Gemma Services that Google reasonably believes are in violation of this Agreement." This is a kill switch that Google maintains in perpetuity over any system you build relying on these models. Our legal review of th…

Could you share what models you consider to be OK for commercialization?
Post reply on HN