Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemma 3 270M: Compact model for hyper-efficient AI

developers.googleblog.com

Post reply on HN
Print view View original

31–40 of 325 posts

Re: Gemma 3 270M: Compact model for hyper-efficient AI

#31

Post by simonw »

marinhero wrote: post #14
simonw wrote: post #11

This model is a LOT of fun. It's absolutely tiny - just a 241MB download - and screamingly fast, and hallucinates wildly about almost everything. Here's one of dozens of results I got for "Generate an SVG of a pelican riding a bicycle". For this one it decided to write a poem: +-----------------------+ | Pelican Riding Bike | +-----------------------+ | This is the cat! | | He's got big wings and a happy tail. | | He…

Serious question but if it hallucinates about almost everything, what's the use case for it?

Fine-tuning for specific tasks. I'm hoping to see some good examples of that soon - the blog entry mentions things like structured text extraction, so maybe something like "turn this text about an event into an iCal document" might work?

Re: Gemma 3 270M: Compact model for hyper-efficient AI

#33

Post by zamadatix »

marinhero wrote: post #14
simonw wrote: post #11

This model is a LOT of fun. It's absolutely tiny - just a 241MB download - and screamingly fast, and hallucinates wildly about almost everything. Here's one of dozens of results I got for "Generate an SVG of a pelican riding a bicycle". For this one it decided to write a poem: +-----------------------+ | Pelican Riding Bike | +-----------------------+ | This is the cat! | | He's got big wings and a happy tail. | | He…

Serious question but if it hallucinates about almost everything, what's the use case for it?

I feel like the blog post, and GP comment, does a good job of explaining how it's built to be a small model easily fine tuned for narrow tasks, rather than used for general tasks out of the box. The latter is guaranteed to hallucinate heavily at this size, that doesn't mean every specific task it's fine tuned to would be. Some examples given were fine tuning it to efficiently and quickly route a query to the right place to actually be handled or tuning it to do sentiment analysis of content.

An easily fine tunable tiny model might actually be one of the better uses of local LLMs I've seen yet. Rather than try to be a small model that's great at everything it's a tiny model you can quickly tune to do one specific thing decently, extremely fast, and locally on pretty much anything.

Re: Gemma 3 270M: Compact model for hyper-efficient AI

#34

Post by simonw »

canyon289 wrote: post #2

Hi all, I built these models with a great team. They're available for download across the open model ecosystem so give them a try! I built these models with a great team and am thrilled to get them out to you. From our side we designed these models to be strong for their size out of the box, and with the goal you'll all finetune it for your use case. With the small size it'll fit on a wide range of hardware and cost…

Do you have any practical examples of fine-tuned variants of this that you can share? A description would be great, but a demo or even downloadable model weights (GGUF ideally) would be even better.

Re: Gemma 3 270M: Compact model for hyper-efficient AI

#35

Post by deadbabe »

marinhero wrote: post #14
simonw wrote: post #11

This model is a LOT of fun. It's absolutely tiny - just a 241MB download - and screamingly fast, and hallucinates wildly about almost everything. Here's one of dozens of results I got for "Generate an SVG of a pelican riding a bicycle". For this one it decided to write a poem: +-----------------------+ | Pelican Riding Bike | +-----------------------+ | This is the cat! | | He's got big wings and a happy tail. | | He…

Serious question but if it hallucinates about almost everything, what's the use case for it?

Games where you need NPCs to talk random jiberrish.

Re: Gemma 3 270M: Compact model for hyper-efficient AI

#36

Post by simonw »

jtbayly wrote: post #24

Can somebody give me a link to a tutorial on how I would go about fine-tuning this? Also, what sorts of things might I consider fine-tuning it for?

This tutorial looks useful: https://ai.google.dev/gemma/docs/core/huggingface_text_full_...

Re: Gemma 3 270M: Compact model for hyper-efficient AI

#37

Post by cgdl »

canyon289 wrote: post #2

Hi all, I built these models with a great team. They're available for download across the open model ecosystem so give them a try! I built these models with a great team and am thrilled to get them out to you. From our side we designed these models to be strong for their size out of the box, and with the goal you'll all finetune it for your use case. With the small size it'll fit on a wide range of hardware and cost…

Very cool. For the INT4 QAT model, what is the recommended precision for the activations and for the key and values stored in KV cache?

Re: Gemma 3 270M: Compact model for hyper-efficient AI

#38

Post by lucb1e »

ertgbnm wrote: post #21
simonw wrote: post #11

This model is a LOT of fun. It's absolutely tiny - just a 241MB download - and screamingly fast, and hallucinates wildly about almost everything. Here's one of dozens of results I got for "Generate an SVG of a pelican riding a bicycle". For this one it decided to write a poem: +-----------------------+ | Pelican Riding Bike | +-----------------------+ | This is the cat! | | He's got big wings and a happy tail. | | He…

He may generate useless tokens but boy can he generate ALOT of tokens.

He? I know some Gemmas and it's distinctly a female name; is Gemma a boy's name where you're from?

Re: Gemma 3 270M: Compact model for hyper-efficient AI

#39

Post by layer8 »

simonw wrote: post #11

This model is a LOT of fun. It's absolutely tiny - just a 241MB download - and screamingly fast, and hallucinates wildly about almost everything. Here's one of dozens of results I got for "Generate an SVG of a pelican riding a bicycle". For this one it decided to write a poem: +-----------------------+ | Pelican Riding Bike | +-----------------------+ | This is the cat! | | He's got big wings and a happy tail. | | He…

> It's absolutely tiny - just a 241MB download

That still requires more than 170 floppy disks for installation.

Re: Gemma 3 270M: Compact model for hyper-efficient AI

#40

Post by JLCarveth »

jtbayly wrote: post #24

Can somebody give me a link to a tutorial on how I would go about fine-tuning this? Also, what sorts of things might I consider fine-tuning it for?

This was linked at the end of Google's announcement: https://docs.unsloth.ai/basics/gemma-3-how-to-run-and-fine-t...

Not sure how much data is needed to realistically fine-tune something like this and get useful output.

Post reply on HN