Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemma 3 270M: Compact model for hyper-efficient AI

developers.googleblog.com

Post reply on HN
Print view View original

211–220 of 325 posts

Re: Gemma 3 270M: Compact model for hyper-efficient AI

#211

Post by kwerk »

canyon289 wrote: post #176
andrewstuart wrote: post #62

Earlier quoted context omitted.

What effort do you folks take to see your models actually running on hardware such as AMD Strix Halo or Apple M3M4? I get the sense that AI is at the “hobby kit computing” stage where they used to dump all the components in a box and give you a schematic and a soldering iron and happily say “you make it work!” And that worked in the early days of computing because there was a small number of people really motivated f…

I don't think we dumped the model and say you do the rest? Myself and my colleagues spent many days transforming the weights into various open compatible formats. And its not just us there's many orgs and partners dedicating their time, resources, and companies to making all open models easy to use. I encourage you to explore the solutions provided by them. We linked some in our blog post here, and there's more. They…

Thank you. And thank you for your kindness in these threads. It’s appreciated by the people who aren’t commenting as much

Re: Gemma 3 270M: Compact model for hyper-efficient AI

#212

Post by Jedd »

gnulinux wrote: post #78
jefb wrote: post #64

My lovely interaction with the 270M-F16 model: > what's second tallest mountain on earth? The second tallest mountain on Earth is Mount Everest. > what's the tallest mountain on earth? The tallest mountain on Earth is Mount Everest. > whats the second tallest mountain? The second tallest mountain in the world is Mount Everest. > whats the third tallest mountain? The third tallest mountain in the world is Mount Everes…

Well, this is a 270M model which is like 1/3 of 1B parameters. In the grand scheme of things, it's basically a few matrix multiplications, barely anything more than that. I don't think it's meant to have a lot of knowledge, grammar, or even coherence. These input: ``` Customer Review says: ai bought your prod-duct and I wanna return becaus it no good. Prompt: Create a JSON object that extracts information about this…

> this is a 270M model which is like 1/3 of 1B parameters

Did you ask Gemma-3-270M whether 27 is closer to a quarter or a third of 100?

Re: Gemma 3 270M: Compact model for hyper-efficient AI

#213

Post by rshemet »

lemonish97 wrote: post #57
elAhmo wrote: post #10

Earlier quoted context omitted.

How do you actually run this on an iPhone?

I use PocketPal. Can run any gguf model off hf.

you can also run it on Cactus - either in Cactus Chat from the App/Play Store or by using the Cactus framework to integrate it into your own app

Re: Gemma 3 270M: Compact model for hyper-efficient AI

#214

Post by rshemet »

riedel wrote: post #194
canyon289 wrote: post #2

Hi all, I built these models with a great team. They're available for download across the open model ecosystem so give them a try! I built these models with a great team and am thrilled to get them out to you. From our side we designed these models to be strong for their size out of the box, and with the goal you'll all finetune it for your use case. With the small size it'll fit on a wide range of hardware and cost…

Would be great to have it included in the Google Edge AI gallery android app.

you can run it in Cactus Chat (download from the Play Store)

Re: Gemma 3 270M: Compact model for hyper-efficient AI

#215

Post by er69 »

Looks very interesting ! I will evaluate this for one of our use cases for a data pipeline that handles around 50 million rows everyday . Mainly need it for segmenting user events. The low memory footprint is huge plus for cost at that scale . Need to spend this weekend on fine tuning, if this disappoints will try one of qwen model

Re: Gemma 3 270M: Compact model for hyper-efficient AI

#216

Post by fallinditch »

amilios wrote: post #152
fallinditch wrote: post #151

> this model is not designed for complex conversational use cases ... but it's also the perfect choice for creative writing ...? Isn't this a contradiction? How can a model be good at creative writing if it's no good at conversation?

Not necessarily. Where do you think the overlap is between these two tasks?

Good question! The overlap is a very interesting space.

Perhaps the overlap is where human input and oversight becomes valuable: in terms of veracity, trust, relevance, interestingness, etc

Re: Gemma 3 270M: Compact model for hyper-efficient AI

#217

Post by Workaccount2 »

dmayle wrote: post #114
perching_aix wrote: post #73

Is it time for me to finally package a language model into my Lambda deployment zips and cut through the corporate red tape at my place around AI use? Update #1: Tried it. Well, dreams dashed - would now fit space wise ( I'd have wanted it to perform natural-language to command-invocation translation (or better, emit me some JSON), but it's super not willing to do that, not in the lame way I'm trying to make it do so…

Did you finetune it before trying? Docs here: https://ai.google.dev/gemma/docs/core/huggingface_text_full_...

How well does using a SOTA model for fine-tuning work? I'm sure people have tried

Re: Gemma 3 270M: Compact model for hyper-efficient AI

#218

Post by namibj »

schyzomaniac wrote: post #177
canyon289 wrote: post #2

Hi all, I built these models with a great team. They're available for download across the open model ecosystem so give them a try! I built these models with a great team and am thrilled to get them out to you. From our side we designed these models to be strong for their size out of the box, and with the goal you'll all finetune it for your use case. With the small size it'll fit on a wide range of hardware and cost…

hi, congrats for the amazing work! i love the 27b model, and i use it basically daily. however when i tried to finetune it for a task in a low resource language, unfortunately i did not succeed: lora just did not picked up the gist of the task, full finetune lead to catastrophic forgetting. may i ask four your advice, or do you have any general tips how to do that properly? thanks in advance for your help :)

lora hyper parameter change? Defaults may not be tuned for knowledge insertion , but rather for style imprinting.

Re: Gemma 3 270M: Compact model for hyper-efficient AI

#219

Post by rao-v »

canyon289 wrote: post #2

Hi all, I built these models with a great team. They're available for download across the open model ecosystem so give them a try! I built these models with a great team and am thrilled to get them out to you. From our side we designed these models to be strong for their size out of the box, and with the goal you'll all finetune it for your use case. With the small size it'll fit on a wide range of hardware and cost…

Fabulous stuff!

Oh my request … the vision head on the Gemma models is super slow on CPU inferencing (and via Vulcan), even via llama.cpp. Any chance your team can figure out a solve? Other ViTs don’t have the same problem.

Re: Gemma 3 270M: Compact model for hyper-efficient AI

#220

Post by freehorse »

andrewstuart wrote: post #62
canyon289 wrote: post #2

Hi all, I built these models with a great team. They're available for download across the open model ecosystem so give them a try! I built these models with a great team and am thrilled to get them out to you. From our side we designed these models to be strong for their size out of the box, and with the goal you'll all finetune it for your use case. With the small size it'll fit on a wide range of hardware and cost…

What effort do you folks take to see your models actually running on hardware such as AMD Strix Halo or Apple M3M4? I get the sense that AI is at the “hobby kit computing” stage where they used to dump all the components in a box and give you a schematic and a soldering iron and happily say “you make it work!” And that worked in the early days of computing because there was a small number of people really motivated f…

Running this on your mac takes less of the effort of writing this comment (assuming you have homebrew installed)

1. open terminal.app

2. run:

    brew install llama.cpp
    llama-cli -hf ggml-org/gemma-3-270m-GGUF -c 0 -fa -p "hello"
Post reply on HN