Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemma 3 270M re-implemented in pure PyTorch for local tinkering

github.com

Post reply on HN
Print view View original

61–62 of 62 posts

Re: Gemma 3 270M re-implemented in pure PyTorch for local tinkering

#61

Post by 3abiton »

canyon289 wrote: post #12
WithinReason wrote: post #9

Earlier quoted context omitted.

I would like to know your thoughts on using 2/3 of such a small the model's size for embeddings. What would be different if you used a byte-level vocabulary and spent the parameter budget on transformer parameters instead? I think you would lose performance (tok/s) but might gain accuracy.

At this small scale the embeddings indeed were a big focus. Consider this thought process. The tokens themselves are a form of compression. Lets say we have the word "WaffleHouse", character level this would be 11 tokens, but with an embedder this would be perhaps 2 or 3 tokens (I didn't actually run through the tokenizer but we could verify precisely). This matters a lot for on device processing especially. So while…

Beautiful writeup! Thanks for your service!

Re: Gemma 3 270M re-implemented in pure PyTorch for local tinkering

#62

Post by dboon »

hodgehog11 wrote: post #55
dboon wrote: post #51

First, thanks for doing everything you do! I, and I’m sure countless others, genuinely benefit from you. How would you recommend someone with a strong background in undergraduate level traditional ML get into deep learning? I use that as a broad term to encompass all the knowledge needed to understand how these models work, starting from the deep learning models of a decade ago, plus the practical ability to collect…

As someone who has students that work in deep learning, I can say that it is unwise to approach deep learning in the same way as traditional ML. Most classical methods are strongly mathematically motivated and have excellent theory to accompany them. Deep learning is still alchemy; it is a matter of experience, trying things out and getting a feel for how the pieces fit together in a modular format. Once you are expe…

Thank you, that was a wonderful reply!
Post reply on HN