Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemma 4 12B: A unified, encoder-free multimodal model

blog.google

Post reply on HN
Print view View original

71–80 of 421 posts

Re: Gemma 4 12B: A unified, encoder-free multimodal model

#71

Post by rao-v »

minimaxir wrote: post #2

The big story here is the encoder-free part, which I still don't fully understand. > Vision: We replaced Gemma 4’s vision encoder with a lightweight embedding module consisting of a single matrix multiplication, positional embedding and normalizations. That's technically encoding, just without using a dedicated model for it like SigLIP? The Developer's Guide elaborates, it's still a 35M layer which I am curious is ro…

Encoder free is huge for running on SBCs etc. often the encoding time is a significant fraction of generation time if you are using a VLM as a all purpose vision model

Re: Gemma 4 12B: A unified, encoder-free multimodal model

#72

Post by mchinen »

minimaxir wrote: post #2

The big story here is the encoder-free part, which I still don't fully understand. > Vision: We replaced Gemma 4’s vision encoder with a lightweight embedding module consisting of a single matrix multiplication, positional embedding and normalizations. That's technically encoding, just without using a dedicated model for it like SigLIP? The Developer's Guide elaborates, it's still a 35M layer which I am curious is ro…

The audio side is even more interesting, as it seems they totally got rid of positional embedding are just doing a single linear transform to match the LLM input dimension and that's it.

> Audio: We simplified audio processing even further. We removed the audio encoder entirely and projected the raw audio signal into the same dimensional space as text tokens.

Re: Gemma 4 12B: A unified, encoder-free multimodal model

#73

Post by vehemenz »

digdugdirk wrote: post #50

I do enjoy the immediate out of touch signaling with the "runs on your 16gb vram laptop" line. Because everyone has a laptop with 16gb vram, or can just pop out and buy a new one, right?

This comment has me a bit confused.

Consumers were complaining about the standard 8GB with the early 2020 refresh of MacBook Pros, many OSes ago. Sure, it might be workable for many tasks (as evidenced by the recent sales of the MacBook Neo), but users with a mere 8GB shouldn't have expectations of LLM performance. Even 16GB feels like a stretch.

Re: Gemma 4 12B: A unified, encoder-free multimodal model

#74

Post by literalAardvark »

rootusrootus wrote: post #30
ethanpil wrote: post #13

What's Google's business case for releasing open models? Don't get me wrong, I am grateful and appreciative of these releases. I'm trying to understand how it fits into their bigger picture as a for profit company? Are they not helping competitors build on the novel technology they have developed? Is it simply goodwill and/or marketing? Or am I missing something strategic?

Neutering OpenAI and Anthropic would be my guess. Commoditized LLMs won't hurt Google nearly as much as it hurts the LLM-only companies, and so accelerating the inevitable just helps knock out potential future competition in areas where Google -does- make a lot of money now.

I think this plays a part, but the truth is that Google doesn't need to do that, Chinese open models are already doing that by themselves.

So perhaps another part is just Google showing that they can indeed play at the big boys table.

Re: Gemma 4 12B: A unified, encoder-free multimodal model

#75

Post by gen220 »

zozbot234 wrote: post #68
gen220 wrote: post #65

Earlier quoted context omitted.

A big part of the frontier labs abilities to charge 80% gross margins on inference is having the cornered resource of frontier models. If that inference becomes popular and valuable enough that those companies make billions of dollars in profit, those companies could use that profit to fund the building of alternative products and platforms that dis-intermediate google's relationship with the customer. Google already…

A 12B-sized model is a far cry from "frontier inference". That's more like DeepSeek V4 Pro territory which is a 1.6T model. Or for multi-modal models, Kimi 2.6 which is 1T.

at risk of quoting myself... :)

> By offering frontier inference closer to cost *and* open-sourcing everything that's sub-frontier

It's two prongs! One prong is that their frontier inference pricing is significantly cheaper/closer-to-at-cost as Anthropic's.

The subject of this thread is the other prong: offering compelling models that are sub-frontier and self-hostable.

Self-hosting models and at-cost frontier models are the high-end and low-end disruptions, respectively, to Ant/OAI/etc.'s business models.

Re: Gemma 4 12B: A unified, encoder-free multimodal model

#76

Post by boutell »

zozbot234 wrote: post #68
gen220 wrote: post #65

Earlier quoted context omitted.

A big part of the frontier labs abilities to charge 80% gross margins on inference is having the cornered resource of frontier models. If that inference becomes popular and valuable enough that those companies make billions of dollars in profit, those companies could use that profit to fund the building of alternative products and platforms that dis-intermediate google's relationship with the customer. Google already…

A 12B-sized model is a far cry from "frontier inference". That's more like DeepSeek V4 Pro territory which is a 1.6T model. Or for multi-modal models, Kimi 2.6 which is 1T.

You're right that it's not literally frontier. But like recent Qwen releases, it is a lot more capable than anybody thought models of this size could be a year ago, like capable enough to set a ceiling on what you can charge for AI for certain applications. Others still clearly justify a stronger model, but this trend may continue, etc.

Re: Gemma 4 12B: A unified, encoder-free multimodal model

#77

Post by embedding-shape »

sambaumann wrote: post #70
embedding-shape wrote: post #40

Earlier quoted context omitted.

MLX is quite literally macOS-specific technology, for other platforms you want non-MLX. I was sure "MLX" stood for "Metal-something-something" but can't find any reference to that somehow, anywho, "Metal" is hardware-accelerated graphics on Apple platforms FWIW. Edit: about the actual release on Ollama, if you're on non-Apple hardware you probably want the NVFP4 variant ("gemma4:12b-nvfp4") which was uploaded 45 minu…

I still get "this model requires macOS" when trying to pull that one

I don't use Ollama myself anymore, but seems others been having similar issues for quite some time, maybe one of these fit your environment exactly? https://github.com/ollama/ollama/issues?q=is%3Aissue%20state...

Re: Gemma 4 12B: A unified, encoder-free multimodal model

#78

Post by spott »

minimaxir wrote: post #2

The big story here is the encoder-free part, which I still don't fully understand. > Vision: We replaced Gemma 4’s vision encoder with a lightweight embedding module consisting of a single matrix multiplication, positional embedding and normalizations. That's technically encoding, just without using a dedicated model for it like SigLIP? The Developer's Guide elaborates, it's still a 35M layer which I am curious is ro…

This is just early fusion basically.

FAIR did this 2 years ago now: https://arxiv.org/abs/2405.09818

I've been waiting for something like this to be released since then.

The annoying thing is that chameleon was multi-modal out based on the same principles, but this model is just inputs... (I'm curious how they did pre-training without having multi-modal outputs as well. I wonder if they just chopped them off rather than support image output).

Re: Gemma 4 12B: A unified, encoder-free multimodal model

#79

Post by pferdone »

matja wrote: post #43
minimaxir wrote: post #2

The big story here is the encoder-free part, which I still don't fully understand. > Vision: We replaced Gemma 4’s vision encoder with a lightweight embedding module consisting of a single matrix multiplication, positional embedding and normalizations. That's technically encoding, just without using a dedicated model for it like SigLIP? The Developer's Guide elaborates, it's still a 35M layer which I am curious is ro…

One side-effect, is that the separate .mmproj file (Multi-Modal Projection encoder) is no longer needed, when using the model with llama.cpp etc.

But do I have the option to run it 'text only'?

Re: Gemma 4 12B: A unified, encoder-free multimodal model

#80

Post by properbrew »

djyde wrote: post #36

What are the use cases for these small models? Is there anyone using models of this scale in their daily life who could share their experience?

I think small models have a very good niche for specific tasks. I utilise a fine tuned Phi-4 model (smaller than this one) that fits in about 3.5gb of RAM (not vram) for the document processing side of things for the desktop app I develop (a bit of a shameless plug - whistle-enterprise.com).

If you have a very specific idea for local model use you can find a way to make it work very well, you don't even need to have a graphics card or NPU chip. You just have to be extremely constrained in how it's used. I think as a generic chatbot they're not great, I'd use a hosted SOTA model and I'm a big fan of local LLMs myself.

Post reply on HN