Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google releases Gemma 4 open models

deepmind.google

Post reply on HN
Print view View original

211–220 of 507 posts

Re: Google releases Gemma 4 open models

#211

Post by [deleted] »

pentagrama wrote: post #190
danielhanchen wrote: post #2

Thinking / reasoning + multimodal + tool calling. We made some quants at https://huggingface.co/collections/unsloth/gemma-4 for folks to run them - they work really well! Guide for those interested: https://unsloth.ai/docs/models/gemma-4 Also note to use temperature = 1.0, top_p = 0.95, top_k = 64 and the EOS is " ". " thought\n" is also used for the thinking trace!

Hey, I tried to use Unsloth to run Gemma 4 locally but got stuck during the setup on Windows 11. At some point it asked me to create a password, and right after that it threw an error. Here’s a screenshot: https://imgur.com/a/sCMmqht This happened after running the PowerShell setup, where it installed several things like NVIDIA components, VS Code, and Python. At the end, PowerShell tell me to open a http://localhost…

[deleted]

Re: Google releases Gemma 4 open models

#212

Post by danielhanchen »

ryandrake wrote: post #203
danielhanchen wrote: post #115

Earlier quoted context omitted.

Thank you! I presume 24B is somewhat faster since it's only 4B activated - 31B is quite a large dense model so more accurate!

This is one of the more confusing aspects of experimenting with local models as a noob. Given my GPU, which model should I use, which quantization of that model should I pick (unsloth tends to offer over a dozen!) and what context size should I use? Overestimate any of these, and the model just won't load and you have to trial-and-error your way to finding a good combination. The red/yellow/green indicators on huggin…

Definitely Unsloth Studio can help - we recommend specific quants (like Gemma-4) and also auto calculate the context length etc!

Re: Google releases Gemma 4 open models

#213

Post by pama »

nabakin wrote: post #196
melodyogonna wrote: post #181

Earlier quoted context omitted.

I reviewed the TensorRT-LLM commit history from the past few days and couldn't find any updates regarding Gemma 4 support. By contrast, here is the reference for MAX: https://github.com/modular/modular/commit/57728b23befed8f3b4...

If OP meant they have the fastest implementation of Gemma 4 on Blackwell at the moment, I guess that is technically true. I doubt that will hold up when TensorRT-LLM finishes their implementation though.

How is the sglang performance on Blackwell for this model?

Re: Google releases Gemma 4 open models

#214

Post by evilelectron »

polishdude20 wrote: post #132
evilelectron wrote: post #57

Earlier quoted context omitted.

Daniel, your work is changing the world. More power to you. I setup a pipeline for inference with OCR, full text search, embedding and summarization of land records dating back 1800s. All powered by the GGUF's you generate and llama.cpp. People are so excited that they can now search the records in multiple languages that a 1 minute wait to process the document seems nothing. Thank you!

Hey in really interested in your pipeline techniques. I've got some pdfs I need to get processed but processing them in the cloud with big providers requires redaction. Wondering if a local model or a self hosted one would work just as well.

I run llama.cpp with Qwen3-VL-8B-Instruct-Q4_K_S.gguf with mmproj-F16.gguf for OCR and translation. I also run llama.cpp with Qwen3-Embedding-0.6B-GGUF for embeddings. Drupal 11 with ai_provider_ollama and custom provider ai_provider_llama (heavily derived from ai_provider_ollama) with PostreSQL and pgvector.

People on site scan the documents and upload them for archival. The directory monitor looks for new files in the archive directories and once a new file is available, it is uploaded to Drupal. Once a new content is created in Drupal, Drupal triggers the translation and embedding process through llama.cpp. Qwen3-VL-8B is also used for chat and RAG. Client is familiar with Drupal and CMS in general and wanted to stay in a similar environment. If you are starting new I would recommend looking at docling.

Re: Google releases Gemma 4 open models

#215

Post by retinaros »

simonw wrote: post #106
wordpad wrote: post #83

Earlier quoted context omitted.

Do you think it's just part of their training set now?

If it's part of their training set why do the 2B and 4B models produce such terrible SVGs?

because generating nice looking svg requires handling code, shapes, long context, reasoning and at 2b you most likely will break the syntax of the file 9 times out of 10 if you train for that. or you will need to go for simpler pelicans. might not be worth to ft on a 2b. but on their top tier open model it is definitly worth it. even not directly but just crawling a github would make it train on your pelicans.

Re: Google releases Gemma 4 open models

#216

Post by ryandrake »

danielhanchen wrote: post #212
ryandrake wrote: post #203

Earlier quoted context omitted.

This is one of the more confusing aspects of experimenting with local models as a noob. Given my GPU, which model should I use, which quantization of that model should I pick (unsloth tends to offer over a dozen!) and what context size should I use? Overestimate any of these, and the model just won't load and you have to trial-and-error your way to finding a good combination. The red/yellow/green indicators on huggin…

Definitely Unsloth Studio can help - we recommend specific quants (like Gemma-4) and also auto calculate the context length etc!

Will have to try it out. I always thought that was more for fine-tuning and less for inference.

Re: Google releases Gemma 4 open models

#218

Post by petu »

zaat wrote: post #95
danielhanchen wrote: post #2

Thinking / reasoning + multimodal + tool calling. We made some quants at https://huggingface.co/collections/unsloth/gemma-4 for folks to run them - they work really well! Guide for those interested: https://unsloth.ai/docs/models/gemma-4 Also note to use temperature = 1.0, top_p = 0.95, top_k = 64 and the EOS is " ". " thought\n" is also used for the thinking trace!

Thank you for your work. You have an answer on your page regarding "Should I pick 26B-A4B or 31B?", but can you please clarify if, assuming 24GB vRAM, I should pick a full precision smaller model or 4 bit larger model?

Try 26B first. 31B seems to have very heavy KV cache (maybe bugged in llama.cpp at the moment; 16K takes up 4.9GB).

edit: 31B cache is not bugged, there's static SWA cost of 3.6GB.. so IQ4_XS at 15.2GB seems like reasonable pair, but even then barely enough for 64K for 24GB VRAM. Maybe 8 bit KV quantization is fine now after https://github.com/ggml-org/llama.cpp/pull/21038 got merged, so 100K+ is possible.

> I should pick a full precision smaller model or 4 bit larger model?

4 bit larger model. You have to use quant either way -- even if by full precision you mean 8 bit, it's gonna be 26GB + overhead + chat context.

Try UD-Q4_K_XL.

Re: Google releases Gemma 4 open models

#219

Post by danielhanchen »

ryandrake wrote: post #216
danielhanchen wrote: post #212

Earlier quoted context omitted.

Definitely Unsloth Studio can help - we recommend specific quants (like Gemma-4) and also auto calculate the context length etc!

Will have to try it out. I always thought that was more for fine-tuning and less for inference.

Oh yes sadly we partially mis-communicated haha - there's both and synthetic data generation + exporting!

Re: Google releases Gemma 4 open models

#220

Post by danielhanchen »

petu wrote: post #218
zaat wrote: post #95

Earlier quoted context omitted.

Thank you for your work. You have an answer on your page regarding "Should I pick 26B-A4B or 31B?", but can you please clarify if, assuming 24GB vRAM, I should pick a full precision smaller model or 4 bit larger model?

Try 26B first. 31B seems to have very heavy KV cache (maybe bugged in llama.cpp at the moment; 16K takes up 4.9GB). edit: 31B cache is not bugged, there's static SWA cost of 3.6GB.. so IQ4_XS at 15.2GB seems like reasonable pair, but even then barely enough for 64K for 24GB VRAM. Maybe 8 bit KV quantization is fine now after https://github.com/ggml-org/llama.cpp/pull/21038 got merged, so 100K+ is possible. > I should…

Yes UD-Q4_K_XL works well! :)
Post reply on HN