Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemma 4 12B: A unified, encoder-free multimodal model

blog.google

Post reply on HN
Print view View original

81–90 of 421 posts

Re: Gemma 4 12B: A unified, encoder-free multimodal model

#81

Post by robgough »

djyde wrote: post #36

What are the use cases for these small models? Is there anyone using models of this scale in their daily life who could share their experience?

I've got a home-built dictation app that uses a local model to clear up the text and fix grammar. It was super easy to build. I’m extending it to capture meeting notes and summarise too. All on-device.

I saw a little app the other day, I think someone posted on here, that looks at your screenshot and renames the file based off the contents of the file.

There's tons of little examples like that. For a lot of use cases, you really don't need the frontier models.

Re: Gemma 4 12B: A unified, encoder-free multimodal model

#82

Post by NekkoDroid »

vehemenz wrote: post #73
digdugdirk wrote: post #50

I do enjoy the immediate out of touch signaling with the "runs on your 16gb vram laptop" line. Because everyone has a laptop with 16gb vram, or can just pop out and buy a new one, right?

This comment has me a bit confused. Consumers were complaining about the standard 8GB with the early 2020 refresh of MacBook Pros, many OSes ago. Sure, it might be workable for many tasks (as evidenced by the recent sales of the MacBook Neo), but users with a mere 8GB shouldn't have expectations of LLM performance. Even 16GB feels like a stretch.

I think you are mixing up RAM and VRAM.

Re: Gemma 4 12B: A unified, encoder-free multimodal model

#83

Post by zozbot234 »

philipkglass wrote: post #63
lxgr wrote: post #60

Am I missing something or are the Ollama versions of this ( https://ollama.com/library/gemma4/tags ) text-only for now?

Since ollama has diverged from llama.cpp, it will take a bit of time for ollama to support multi-modality. If you're using plain llama.cpp it looks like a PR has already merged for this model with vision and audio support: https://github.com/ggml-org/llama.cpp/pull/24077

They've actually gone back to (a lightly patched) llama.cpp with the 0.30 release a few weeks ago, and have now vendored-in an up to date release. Needless to say this is great news for both projects!

Re: Gemma 4 12B: A unified, encoder-free multimodal model

#84

Post by make3 »

mchinen wrote: post #72
minimaxir wrote: post #2

The big story here is the encoder-free part, which I still don't fully understand. > Vision: We replaced Gemma 4’s vision encoder with a lightweight embedding module consisting of a single matrix multiplication, positional embedding and normalizations. That's technically encoding, just without using a dedicated model for it like SigLIP? The Developer's Guide elaborates, it's still a 35M layer which I am curious is ro…

The audio side is even more interesting, as it seems they totally got rid of positional embedding are just doing a single linear transform to match the LLM input dimension and that's it. > Audio: We simplified audio processing even further. We removed the audio encoder entirely and projected the raw audio signal into the same dimensional space as text tokens.

I guarantee you there's positional information one way or another. they just don't mention it because positional embeddings are extremely cheap computationally, not worth mentioning

Re: Gemma 4 12B: A unified, encoder-free multimodal model

#85

Post by spott »

randomNumber7 wrote: post #33

> Novel unified architecture: No multimodal encoders. The vision and audio inputs flow directly into the LLM backbone. I would be interested in how this actually works. I couldn't find a description of the model architecture (and I did check the links in the Google blog)

https://newsletter.maartengrootendorst.com/p/a-visual-guide-... (in a link from here: https://developers.googleblog.com/gemma-4-12b-the-developer-..., which was linked in the text of the post, but not the linkdump at the end).

Re: Gemma 4 12B: A unified, encoder-free multimodal model

#87

Post by woadwarrior01 »

minimaxir wrote: post #2

The big story here is the encoder-free part, which I still don't fully understand. > Vision: We replaced Gemma 4’s vision encoder with a lightweight embedding module consisting of a single matrix multiplication, positional embedding and normalizations. That's technically encoding, just without using a dedicated model for it like SigLIP? The Developer's Guide elaborates, it's still a 35M layer which I am curious is ro…

There are many priors to encoder-free VLMs. I specifically remember the EVE series of models from ~2 years.

https://github.com/baaivision/EVE

Re: Gemma 4 12B: A unified, encoder-free multimodal model

#89

Post by claysmithr »

digdugdirk wrote: post #50

I do enjoy the immediate out of touch signaling with the "runs on your 16gb vram laptop" line. Because everyone has a laptop with 16gb vram, or can just pop out and buy a new one, right?

I have 24 gb unified memory so it’s a good model for me
Post reply on HN