Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemma 3 270M: Compact model for hyper-efficient AI

developers.googleblog.com

Post reply on HN
Print view View original

201–210 of 325 posts

Re: Gemma 3 270M: Compact model for hyper-efficient AI

#201

Post by littlestymaar »

jama211 wrote: post #146
littlestymaar wrote: post #108

Earlier quoted context omitted.

What I read above is not an evaluation on “encyclopedic knowledge” though, it's a very basic a common sense: I wouldn't mind if the model didn't know the name of the biggest mountain on earth, but if the model cannot grasp the fact that the same mountain cannot simultaneously be #1, #2 and #3, then the model feels very dumb.

It’s a language model? Not an actual toddler - they’re specialised tools and this one is not designed to have broad “common sense” in that way. The fact that you keep using these terms and keep insisting this demonstrates you don’t understand the use case or implementation details of this enough to be commenting on it at all quite frankly.

> they’re specialised tools and this one is not designed to have broad “common sense” in that way.

Except the key property of language models compared to other machine learning techniques is their ability to have this kind of common sense understanding of the meaning of natural language.

> you don’t understand the use case of this enough to be commenting on it at all quite frankly.

That's true that I don't understand the use-case for a language model that doesn't have a grasp of what first/second/third mean. Sub-1B models are supposed to be fine-tuned to be useful, but if the base model is so bad at language it can't make the difference between first and second and you need to put that in your fine-tuning as well as your business logic, why use a base model at all?

Also, this is a clear instance of moving the goalpost, as the comment I responded to was talking about how we should not expect such a small model to have “encyclopedic knowledge”, and now you are claiming we should not expect such a small language model to make sense of language…

Re: Gemma 3 270M: Compact model for hyper-efficient AI

#202

Post by all2 »

ActorNightly wrote: post #3
canyon289 wrote: post #2

Hi all, I built these models with a great team. They're available for download across the open model ecosystem so give them a try! I built these models with a great team and am thrilled to get them out to you. From our side we designed these models to be strong for their size out of the box, and with the goal you'll all finetune it for your use case. With the small size it'll fit on a wide range of hardware and cost…

How does the 270 perform with coding? I use Gemma27b currently with a custom agent wrapper and its working pretty well.

Can you talk about your agent wrapper setup? What tools, if any did you use? How effective is it at making a dumb model smart?

Re: Gemma 3 270M: Compact model for hyper-efficient AI

#203

Post by hadlock »

canyon289 wrote: post #104
megaman821 wrote: post #96

Earlier quoted context omitted.

What size of tasks can this handle? Can you do a fine-tune of Mac System Settings?

32k context window so whatever fits in there. What is a finetune of mac system settings?

It seems to dip into repeating itself pretty quickly on any task of actual complexity.

Re: Gemma 3 270M: Compact model for hyper-efficient AI

#204

Post by minimaxir »

AJRF wrote: post #101

I've got a very real world use case I use DistilBERT for - learning how to label wordpress articles. It is one of those things where it's kind of valuable (tagging) but not enough to spend loads on compute for it. The great thing is I have enough data (100k+) to fine-tune and run a meaningful classification report over. The data is very diverse, and while the labels aren't totally evenly distributed, I can deal with…

ModernBERT may be a better base model if finetuning a model for a specific use case: https://huggingface.co/blog/modernbert

Re: Gemma 3 270M: Compact model for hyper-efficient AI

#205

Post by gapeleon »

_1 wrote: post #106
canyon289 wrote: post #2

Hi all, I built these models with a great team. They're available for download across the open model ecosystem so give them a try! I built these models with a great team and am thrilled to get them out to you. From our side we designed these models to be strong for their size out of the box, and with the goal you'll all finetune it for your use case. With the small size it'll fit on a wide range of hardware and cost…

> and with the goal you'll all finetune it for your use case. What use-cases are a good fit for finetuning this model? More specific instruction following, knowledge from proprietary data, response tone?

I'm going to try training it on a codebook to see if such a small model would work for a TTS.

Re: Gemma 3 270M: Compact model for hyper-efficient AI

#206

Post by hadlock »

jefb wrote: post #64

My lovely interaction with the 270M-F16 model: > what's second tallest mountain on earth? The second tallest mountain on Earth is Mount Everest. > what's the tallest mountain on earth? The tallest mountain on Earth is Mount Everest. > whats the second tallest mountain? The second tallest mountain in the world is Mount Everest. > whats the third tallest mountain? The third tallest mountain in the world is Mount Everes…

>give me a two day itenerary for paris

Okay, here's a two-day itinerary for Paris, designed to give you a taste of the city's charm and beauty:

Day 1: Iconic Landmarks and Artistic Exploration

Morning (9:00 AM - 12:00 PM): Start your day with a visit to the Eiffel Tower. Take the elevator to the top for breathtaking panoramic views. Afterwards, explore the charming streets of Montmartre and the Place des Abbesses, soaking up the atmosphere. Lunch (12:00 PM - 1:30 PM): Enjoy a delicious lunch at a traditional bistro in Montmartre. Try a classic French meal like steak frites or a hearty onion soup. Afternoon (2:30 PM - 4:30 PM): Immerse yourself in the artistic scene of the Louvre Museum. Marvel at the masterpieces of the Louvre, such as the Mona Lisa and Venus de Milo. Afterwards, explore the serene gardens of Tuileries Palace. Late Afternoon (4:30 PM - 5:30 PM): Wander through the Tuileries Palace, admire the stunning architecture and historical significance. Day 2: History, Food, and a Touch of Parisian Elegance

Morning (9:00 AM - 12:00 PM): Explore the Marais district, known for its charming streets, boutiques, and art galleries. Visit the Place de la Concorde, a historical square, and the Musée Rodin. Lunch (12:00 PM - 1:30 PM): Enjoy a delicious lunch in the Marais. Try a traditional French bistro or a cafe with a view. Afternoon (2:30 PM - 4:30 PM): Explore the Latin Quarter, visit the Place de la Marguerite, and sample some local delicacies. Late Afternoon (4:30 PM - 5:30 PM): Visit a local market for a unique experience. Sample fresh produce, local cheeses, and other delicacies. Tips for a Smooth Itinerary:

Book your flights and accommodation in advance, especially during peak season. Wear comfortable shoes, as you'll be doing a lot of walking. Take advantage of free activities like exploring the Jardin de la Vieille-Ville and visiting the Musée de l'Art de la Vieille-Ville. Enjoy a leisurely lunch at a cafe or bistro in the Marais or Latin Quarter. Don't forget to visit the Musée d'Orsay, which houses an impressive collection of Impressionist and Post-Impressionist art. Enjoy the beautiful atmosphere of Paris!

Re: Gemma 3 270M: Compact model for hyper-efficient AI

#207

Post by danielhanchen »

simonw wrote: post #191
danielhanchen wrote: post #135

Earlier quoted context omitted.

Oh :( Maybe the settings? Could you try temperature = 1.0, top_k = 64, top_p = 0.95, min_p = 0.0

My topping only lets me set temperature and top_p but setting them to those values did seem to avoid the infinite loops, thanks.

Oh fantastic it worked! I was actually trying to see if we can auto set these within LM Studio (Ollama for eg has params, template) - not sure if you know how that can be done? :)

Re: Gemma 3 270M: Compact model for hyper-efficient AI

#208

Post by danielhanchen »

JLCarveth wrote: post #187
simonw wrote: post #54

Earlier quoted context omitted.

Good call, I'm trying that one just now in LM Studio (by clicking "Use this model -> LM Studio" on https://huggingface.co/unsloth/gemma-3-270m-it-GGUF and selecting the F16 one). (It did not do noticeably better at my pelican test). Actually it's worse than that, several of my attempts resulted in infinite loops spitting out the same text. Maybe that GGUF is a bit broken?

I ran into the same looping issue with that model.

Definitely give

temperature = 1.0, top_k = 64, top_p = 0.95, min_p = 0.0

a try, and maybe repeat_penalty = 1.1

Re: Gemma 3 270M: Compact model for hyper-efficient AI

#209

Post by danielhanchen »

nerdix wrote: post #82

Is it possible to finetune a model like this with local hardware? Every tutorial I've come across on finetuning a local LLM uses some cloud service like colab or runpod.

Yes! For eg you can install Unsloth locally via pip install unsloth :)

Re: Gemma 3 270M: Compact model for hyper-efficient AI

#210

Post by jtolmar »

canyon289 wrote: post #89
simonw wrote: post #34

Earlier quoted context omitted.

Do you have any practical examples of fine-tuned variants of this that you can share? A description would be great, but a demo or even downloadable model weights (GGUF ideally) would be even better.

We obviously need to create a pelican bicycle svg finetune ;) If you want to try this out I'd be thrilled to do it with you, I genuinely am curious how well this model can perform if specialized on that task. A couple colleagues of mine posted an example of finetuning a model to take on persona's for videogame NPCs. They have experience working with folks in the game industry and a use case like this is suitable for…

Caves Of Qud uses Markov chain generated text to great effect in some places. I think something light that's still more competent than Markov chains has a lot of potential.
Post reply on HN