I'm actually thinking how much this is bett3r (besides multimedia) over prismml's 1.5bit model based on qwen2.5 or sth.
Gemma 4 12B: A unified, encoder-free multimodal model
231–240 of 421 posts
Re: Gemma 4 12B: A unified, encoder-free multimodal model
#232Earlier quoted context omitted.
I just build llama.cpp from scratch on the PR that has MTP drafters. https://github.com/ggml-org/llama.cpp/pull/23398 Please don't use Ollama, it's a bad actor in the OSS community.
I don't have the energy to build stuff all the time, that's a rabbit-hole side tunnel I don't really want to get into. I have larger concerns in my life that are more urgent than developing that side of things. But I've moved on from Ollama for the time being, though I am mainly interested to see what the Gemma 4 MTP speeds are like on my M1 Max, so I may test it. I am quite impressed with the tools in LM Studio, whi…
Totally understandable. YMMV but I found the llama.cpp build process to work on the first try on my machine, and it only takes a couple minutes, which definitely isn't my usual expectation or experience. I was very pleasantly surprised. Their web-ui is also getting very polished while still doing a great job of letting you tweak all the weird settings.
Re: Gemma 4 12B: A unified, encoder-free multimodal model
#233The big story here is the encoder-free part, which I still don't fully understand. > Vision: We replaced Gemma 4’s vision encoder with a lightweight embedding module consisting of a single matrix multiplication, positional embedding and normalizations. That's technically encoding, just without using a dedicated model for it like SigLIP? The Developer's Guide elaborates, it's still a 35M layer which I am curious is ro…
Either Google changed the text or you editorialised it a tiny bit - just for all others that got excited, they mean 16GB V RAM. So a premium graphics card requiring a >2500€ device is the minimum to run this. Still progress, but not quite democratic yet. Weird though that Google might be cannibalising it's own AI subscription service?
I've bought a laptop for 2500€ device is necessary, though I'm certain it would yield better and faster results.
Re: Gemma 4 12B: A unified, encoder-free multimodal model
#234We are now entering the closed loop game. Google doesn't need anyone else to accelerate their models. This is their bread and butter. I'm both shocked but also not surprised that they continue to develop such efficiencies. Honestly it's like silicon and CPU architecture advancement. We kept shrinking it and shrinking it and it kept getting more and more powerful and here we are with AI and it's only going to be 100x…
Nope, lol.
Large models still are quite far ahead, don't be fooled that even Gemma:31b (which is better than the 12b overall) is anywhere close to big models.
There is definitely room for optimization, but fundamentally, for complex tasks, you need visible small gradients for accuracy that allow the model to be trained on (and consequently be followed during inference). For example, if you specify in instructions not to write code but ask coding question, Gemma will still write code. Whereas Gemini/Claude will pick up on that and follow your instructions better.
Re: Gemma 4 12B: A unified, encoder-free multimodal model
#235Earlier quoted context omitted.
I guess Google implements more / stronger guard rails than Alibaba and thus confuses these small models. At least this was my impression with Gemma3 models where it often said that the image contains some nudity / sex scenes and therefore it cannot give a description of the image. Never understood the point of this behavior....
The biggest problem with all the Google models has always been RLHF, particularly safety training. They take a good, smart model and make it behave like a corporate person that has been to far to many forced anti-{sexism, racism...} seminars so that it is now living in fear of saying something that could be construed as wrong by some moral standard.
This is almost certainly not true.
If it was, they wouldn't need to be using the classifiers they are using to warn Gemini about problematic prompts.
Re: Gemma 4 12B: A unified, encoder-free multimodal model
#236What's Google's business case for releasing open models? Don't get me wrong, I am grateful and appreciative of these releases. I'm trying to understand how it fits into their bigger picture as a for profit company? Are they not helping competitors build on the novel technology they have developed? Is it simply goodwill and/or marketing? Or am I missing something strategic?
A big part of the frontier labs abilities to charge 80% gross margins on inference is having the cornered resource of frontier models. If that inference becomes popular and valuable enough that those companies make billions of dollars in profit, those companies could use that profit to fund the building of alternative products and platforms that dis-intermediate google's relationship with the customer. Google already…
Don't think its that.
Basically with upcoming spark laptops, the smaller models will likely get fine tuned to interface with google services. Then, Google can essentially make Chromebook software include those models, which is the same use case as android.
And you better believe that they will be collecting user data and building advertising models.
Re: Gemma 4 12B: A unified, encoder-free multimodal model
#237Earlier quoted context omitted.
I guess Google implements more / stronger guard rails than Alibaba and thus confuses these small models. At least this was my impression with Gemma3 models where it often said that the image contains some nudity / sex scenes and therefore it cannot give a description of the image. Never understood the point of this behavior....
The biggest problem with all the Google models has always been RLHF, particularly safety training. They take a good, smart model and make it behave like a corporate person that has been to far to many forced anti-{sexism, racism...} seminars so that it is now living in fear of saying something that could be construed as wrong by some moral standard.
[flagged]
Re: Gemma 4 12B: A unified, encoder-free multimodal model
#238Just tried this out. Jesus Christ. Google does some things so well.
I mean, they did invent the technology. It's actually kinda surprising they're not the leader in the space. They kinda got Kodaked (though the story is still playing out, and I guess they're still somewhat competitive in the space even if Anthropic and OpenAI are the leaders).
Re: Gemma 4 12B: A unified, encoder-free multimodal model
#239I'm waiting for FP8 quant, preferably from Google.
Re: Gemma 4 12B: A unified, encoder-free multimodal model
#240What are the use cases for these small models? Is there anyone using models of this scale in their daily life who could share their experience?
I've used Gemma for reviewing and categorizing my writing online over several years (~5 million words across a forum for an OSS project I work on, HN, reddit, etc.), experimenting with training LoRAs (again, on my own writing, since I don't have to worry about ethically sourcing the data if it's all mine), and I'm currently using it to perform web searches and extract data about a specific type of business. It's plen…
So one of thr things you're using it for is to generate leads to spam businesses with unwanted LLM produced marketing materials it sounds like.
Wow LLMs are changing the world, what a utopia.