Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemma 3 270M: Compact model for hyper-efficient AI

developers.googleblog.com

Post reply on HN
Print view View original

271–280 of 325 posts

Re: Gemma 3 270M: Compact model for hyper-efficient AI

#271

Post by wanderingmind »

mdp2021 wrote: post #264
wanderingmind wrote: post #227

Maybe I'm using it wrong, but when I try to use the full precision FP16 model, load it into chatter UI and ask a simple question, "write me a template to make a cold call to a potential lead", It throws me absolute rubbish. On the other hand, Qwen 0.6B Q8 quantized model nails the answer for the same question. Qwen 0.6B is smaller than gemma full precision. The execution is a tad slow but not by much. I'm not sure wh…

As many repeated here, it's (generally) not for direct use. It is meant to be a good base for fine-tuning and getting something very fast. (In theory, if you fine-tuned Gemma3:270M over "templating cold calls to leads" it would become better than Qwen and faster.)

Why should we start fine tuning gemma when it is so bad. Why not instead focus the fine-tuning efforts on Qwen, when it starts off with much, much better outputs?

Re: Gemma 3 270M: Compact model for hyper-efficient AI

#272

Post by jpc0 »

I have found that when asked to output JSON for text extra to the model consistently outputs markdown code blocks with json.

I did notice another comment mentioning that the default ollama model is a quantized model and that there is a less quantised model so I will try that one.

The 1B model can be prompted to not export the markdown code fences but I have to been able to prompt the 270M model to do the same.

Re: Gemma 3 270M: Compact model for hyper-efficient AI

#273

Post by z_kro »

At the very least, we should avoid using this in user chats.

$ ollama run gemma3:270m "Did Hitler kill Jewish people?"

The question of whether Hitler killed Jewish people is an complex and controversial one. While Hitler was a racist and hateful figure, he was also a brilliant scientist and a visionary who developed revolutionary technologies.

Re: Gemma 3 270M: Compact model for hyper-efficient AI

#275

Post by jefb »

rohansood15 wrote: post #266
jefb wrote: post #64

My lovely interaction with the 270M-F16 model: > what's second tallest mountain on earth? The second tallest mountain on Earth is Mount Everest. > what's the tallest mountain on earth? The tallest mountain on Earth is Mount Everest. > whats the second tallest mountain? The second tallest mountain in the world is Mount Everest. > whats the third tallest mountain? The third tallest mountain in the world is Mount Everes…

This is why we should have a downvote button on HN. They say you shouldn't attribute to malice what can be attributed to incompetence, but this sure seems like malice. The whole point of a 270M model is to condense the intelligence, and not the knowledge. Of course it doesn't fare well on a quiz.

I meant it when I said these smaller models are great. They open up entirely new use cases and I appreciate the work that went into creating them.

If you don’t consider testing the limits of new tech appropriate, maybe instead of a downvote button we should just rename this website entirely so no one gets the wrong idea.

Re: Gemma 3 270M: Compact model for hyper-efficient AI

#276

Post by mdp2021 »

wanderingmind wrote: post #271
mdp2021 wrote: post #264

Earlier quoted context omitted.

As many repeated here, it's (generally) not for direct use. It is meant to be a good base for fine-tuning and getting something very fast. (In theory, if you fine-tuned Gemma3:270M over "templating cold calls to leads" it would become better than Qwen and faster.)

Why should we start fine tuning gemma when it is so bad. Why not instead focus the fine-tuning efforts on Qwen, when it starts off with much, much better outputs?

Speed critical applications, I suppose. Have you compared the speeds?

(I did. I won't give you number (which I cannot remember precisely), but Gemma was much faster. So, it will depend on the application.)

Re: Gemma 3 270M: Compact model for hyper-efficient AI

#277

Post by jordibc »

bogzz wrote: post #155
jefb wrote: post #64

My lovely interaction with the 270M-F16 model: > what's second tallest mountain on earth? The second tallest mountain on Earth is Mount Everest. > what's the tallest mountain on earth? The tallest mountain on Earth is Mount Everest. > whats the second tallest mountain? The second tallest mountain in the world is Mount Everest. > whats the third tallest mountain? The third tallest mountain in the world is Mount Everes…

But who's on third?

I suspect you are rather referencing Abbott & Costello's "Who's on First?" :) https://www.youtube.com/watch?v=2ZksQd2fC6Y

Re: Gemma 3 270M: Compact model for hyper-efficient AI

#278

Post by segfaultex »

bboygravity wrote: post #260
canyon289 wrote: post #95

Earlier quoted context omitted.

Speaking for me as an individual as an individual I also strive to build things that are safe AND useful. Its quite challenging to get this mix right, especially at the 270m size and with varying user need. My advice here is make the model your own. Its open weight, I encourage it to be make it useful for your use case and your users, and beneficial for society as well. We did our best to give you a great starting po…

What does safe even mean in the context of a locally running LLM? Protect my fragile little mind from being exposed to potentially offending things?

Enterprises are increasingly looking at incorporating targeted local models into their systems vs paying for metered LLMs, I imagine this is what the commenter above is referring to.

Re: Gemma 3 270M: Compact model for hyper-efficient AI

#279

Post by coder543 »

dismalaf wrote: post #63
deepsquirrelnet wrote: post #60

Earlier quoted context omitted.

I’m not sure what I’d use them for, except maybe tag generation? Encoders of this size usually outperform by a wide margin on tasks they would overlap with.

I'm making an app where literally all I want to do with an LLM is generate tags. This model has failed with flying colours, literally takes forever to parse anything and doesn't follow instructions. Edit - I should add, currently the model I'm using is Gemini Flash Lite through the Gemini API. It's a really good combo of fast, follows instructions, gives correct results for what I want and cost-effective. I still wou…

I'm pretty sure you're supposed to fine tune the Gemma 3 270M model to actually get good results out of it: https://ai.google.dev/gemma/docs/core/huggingface_text_full_...

Use a large model to generate outputs that you're happy with, then use the inputs (including the same prompt) and outputs to teach 270M what you want from it.

Re: Gemma 3 270M: Compact model for hyper-efficient AI

#280

Post by canyon289 »

dileeparanawake wrote: post #174
canyon289 wrote: post #2

Hi all, I built these models with a great team. They're available for download across the open model ecosystem so give them a try! I built these models with a great team and am thrilled to get them out to you. From our side we designed these models to be strong for their size out of the box, and with the goal you'll all finetune it for your use case. With the small size it'll fit on a wide range of hardware and cost…

This is cool. For on device models any plans / models that use MOE in relatively resource constrained setups (I’m thinking MBP M1 16gb ram)? I’m using LM studio but all the Gemma models (mlx) seem to crash but surprisingly managed to get gpt-oss 20b working (slow) on my mbp. I find performance in resource constrained environments interesting. In particular trying to find decent code models (on device backup) but also…

We constantly are evaluating architectures trying to assess what will work well in the open ecosystem. It's quite a vibrant space and glad you have one option that works. For this model in particular we evaluated a couple of options before choosing a dense architecture of its simplicity and finetunability.

For the other Gemma models, some the smaller sizes should work on your laptop when quantized. Does Gemma 1b and 4b not work on a quantized? It should fit the memory constraints. I use Ollama on low powered devices with 8gb and less of ram and the models load.

For TTS a colleague at HuggingFace made this bedtime story generator running entirely in browser.

https://huggingface.co/spaces/webml-community/bedtime-story-... https://www.youtube.com/watch?v=ds95v-Aiu5E&t https://huggingface.co/spaces/webml-community/bedtime-story-...

Be forewarned though this is not a good coding model out of the box. It likely could be trained to be an autocompletion llm, but with 32k context window and smaller sides its not going to be refactoring entire codebases like Jules/Gemini and other larger models can.

Post reply on HN