Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemma 3 270M re-implemented in pure PyTorch for local tinkering

github.com

Post reply on HN
Print view View original

21–30 of 62 posts

Re: Gemma 3 270M re-implemented in pure PyTorch for local tinkering

#21

Post by Scene_Cast2 »

canyon289 wrote: post #12
WithinReason wrote: post #9

Earlier quoted context omitted.

I would like to know your thoughts on using 2/3 of such a small the model's size for embeddings. What would be different if you used a byte-level vocabulary and spent the parameter budget on transformer parameters instead? I think you would lose performance (tok/s) but might gain accuracy.

At this small scale the embeddings indeed were a big focus. Consider this thought process. The tokens themselves are a form of compression. Lets say we have the word "WaffleHouse", character level this would be 11 tokens, but with an embedder this would be perhaps 2 or 3 tokens (I didn't actually run through the tokenizer but we could verify precisely). This matters a lot for on device processing especially. So while…

How would this matrix get trained with PyTorch? I currently have a toy Transformer network - I ended up marking the matrix as sparse and using SparseAdam - gives a bit of a performance boost, but at the same time I can't use torch.compile() on the fetch from this matrix.

Re: Gemma 3 270M re-implemented in pure PyTorch for local tinkering

#23

Post by WithinReason »

canyon289 wrote: post #12
WithinReason wrote: post #9

Earlier quoted context omitted.

I would like to know your thoughts on using 2/3 of such a small the model's size for embeddings. What would be different if you used a byte-level vocabulary and spent the parameter budget on transformer parameters instead? I think you would lose performance (tok/s) but might gain accuracy.

At this small scale the embeddings indeed were a big focus. Consider this thought process. The tokens themselves are a form of compression. Lets say we have the word "WaffleHouse", character level this would be 11 tokens, but with an embedder this would be perhaps 2 or 3 tokens (I didn't actually run through the tokenizer but we could verify precisely). This matters a lot for on device processing especially. So while…

Makes sense, thank you.

Re: Gemma 3 270M re-implemented in pure PyTorch for local tinkering

#24

Post by canyon289 »

tarruda wrote: post #16
canyon289 wrote: post #3

Hey all, I created this model with a top notch team. I answered many questions last week when this hit the front page, and happy to answer more here as well. https://news.ycombinator.com/item?id=44902148 Personally I'm excited that you all have access to this model now and hope you all get value out of using them.

Thanks for your work, it is really an amazing small LM. Can you share what kind of hardware is necessary to train it, and how long it took?

Thank you!

The Gemma3 technical report contains many details on training setup https://arxiv.org/pdf/2503.19786

This was released with the initial batch of Gemma3 so it doesn't contain the 270m details, nonetheless you'll get a good idea of what it takes to build these models.

Re: Gemma 3 270M re-implemented in pure PyTorch for local tinkering

#25

Post by canyon289 »

eachro wrote: post #22

If you wanted to train it from scratch, how long would it take on a reasonable GPU setup?

The world reasonable is vague but assuming you mean something that could be run in a residential unit it would long a very long time if training from pure scratch.

This is part of the rationale for releasing this model. Now you don't have to start from scratch and finetuning is reasonable on a wide variety of hardware, including reasonable GPU setups (and smaller)

Re: Gemma 3 270M re-implemented in pure PyTorch for local tinkering

#26

Post by torben-friis »

This might be a very basic question, but as a dev whose only interaction with models is using the main commercial ones (sonnet, ChatGPT and the like), what are some usecases for these smaller local models?

What usages can be reasonable to expect from them? Are there uses out of the box or does one have to go through some custom post-training to get useful behavior?

I feel like there is a huge gap between understanding models as a user of commercial tools and the kind of discussions happening in these threads, but I’m not sure what are the in-between steps.

Re: Gemma 3 270M re-implemented in pure PyTorch for local tinkering

#27

Post by barrkel »

torben-friis wrote: post #26

This might be a very basic question, but as a dev whose only interaction with models is using the main commercial ones (sonnet, ChatGPT and the like), what are some usecases for these smaller local models? What usages can be reasonable to expect from them? Are there uses out of the box or does one have to go through some custom post-training to get useful behavior? I feel like there is a huge gap between understandin…

Summarization, very basic tool use, without needing to go across the internet and back, and zero cost because of edge compute.

Re: Gemma 3 270M re-implemented in pure PyTorch for local tinkering

#29

Post by _giorgio_ »

torben-friis wrote: post #26

This might be a very basic question, but as a dev whose only interaction with models is using the main commercial ones (sonnet, ChatGPT and the like), what are some usecases for these smaller local models? What usages can be reasonable to expect from them? Are there uses out of the box or does one have to go through some custom post-training to get useful behavior? I feel like there is a huge gap between understandin…

Maybe also secrecy and privacy.

Re: Gemma 3 270M re-implemented in pure PyTorch for local tinkering

#30

Post by owebmaster »

canyon289 wrote: post #3

Hey all, I created this model with a top notch team. I answered many questions last week when this hit the front page, and happy to answer more here as well. https://news.ycombinator.com/item?id=44902148 Personally I'm excited that you all have access to this model now and hope you all get value out of using them.

Does it have function calls? Can we use it with MCP?
Post reply on HN