Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemma 3 270M re-implemented in pure PyTorch for local tinkering

github.com

Post reply on HN
Print view View original

51–60 of 62 posts

Re: Gemma 3 270M re-implemented in pure PyTorch for local tinkering

#51

Post by dboon »

First, thanks for doing everything you do! I, and I’m sure countless others, genuinely benefit from you.

How would you recommend someone with a strong background in undergraduate level traditional ML get into deep learning? I use that as a broad term to encompass all the knowledge needed to understand how these models work, starting from the deep learning models of a decade ago, plus the practical ability to collect data or build RL gyms and fine tune them.

I understand ML math well enough that I’m confident I could follow a modern white paper after a lot of effort and research. But there are so many pieces — quantizations, flash attention, Mode, batch sizes, layer sizes, model sparsity. I feel very overwhelmed trying to piece together how all of the pieces arose, and even more overwhelmed trying to figure out how one even goes about fine tuning one. I (like most people here) am extremely technical, and it’s not often I feel this way about a field.

Thanks again! Best of luck on your work

Re: Gemma 3 270M re-implemented in pure PyTorch for local tinkering

#52

Post by canyon289 »

torben-friis wrote: post #46
canyon289 wrote: post #31

Earlier quoted context omitted.

Its a crucial question. I wrote up a long answer here. Let me know it helps https://news.ycombinator.com/item?id=44913558

Thanks for the reply! It does help to figure out where in the space this model fits. I'm still a bit confused about this part: >since it needs to be shaped to match specific tasks, we did our best to design it to be a flexible starting point for LLM-style tasks and worked with partners to put it into the right frameworks and places for you all to be able to shape it to what you need it to be. What does shaping mean i…

> What does shaping mean in this case? What tools are used, what requirements are there, both in terms of hardware and knowledge?

I'll try making an analogy to another task I like which is cooking. In cooking the chef has to make decisions like what is the overall meal going to look like, but then also detailed decisions like what the main course versus side, and even more detailed what's the proportion of side dish serving to main dish, what ingredients, how long to cook something etc.

It's kind of the same with ML models, whether AI or not. When I build smaller bayesian models I make specific choices about the model architecture, which data I use, the array shape of the output etc.

The tools used here are largely jax or pytorch, often in a framework like flax, or a NN higher level package. You often then pair it with libraries that which have NN optimizers, data loaders etc. Pytorch is more batteries included than the JAX ecosystem which separates these out.

One of the best ways to get a grasp of all of this is implement some small models yourself. These pieces will start to be come more apparent and concrete, especially because as an end users you're not exposed to them, the same way most end users are not exposed to compilers.

Re: Gemma 3 270M re-implemented in pure PyTorch for local tinkering

#53

Post by Quarrel »

dboon wrote: post #51

First, thanks for doing everything you do! I, and I’m sure countless others, genuinely benefit from you. How would you recommend someone with a strong background in undergraduate level traditional ML get into deep learning? I use that as a broad term to encompass all the knowledge needed to understand how these models work, starting from the deep learning models of a decade ago, plus the practical ability to collect…

> I’m confident I could follow a modern white paper after a lot of effort and research.

Without having done it for deep learning, I'm sure it is like any other area of computer science. You get to exactly the level you're at now, and then you put in that effort following modern papers, and each one gets easier and easier. A year later you've done the literature review for your Phd. :)

Re: Gemma 3 270M re-implemented in pure PyTorch for local tinkering

#54

Post by mdaniel »

dcreater wrote: post #49
canyon289 wrote: post #3

Hey all, I created this model with a top notch team. I answered many questions last week when this hit the front page, and happy to answer more here as well. https://news.ycombinator.com/item?id=44902148 Personally I'm excited that you all have access to this model now and hope you all get value out of using them.

As a non MLE, what are the pros/cons of OP's PyTorch re-implementation?

I'm not a ML engineer, so I can speak to the "non MLE" bit from my perspective

(literal tl;dr: learning and experimentation opportunity)

1. Since it's just PyTorch, that means one can run it locally upon whatever accelerator you have that PyTorch supports. For quite a few people that includes Metal Performance Shaders: https://docs.pytorch.org/docs/stable/mps.html

I can attest that building PyTorch from git is achievable in about 15 minutes on my M1 Pro, if you really want to chase the rabbithole. Cloning PyTorch is its own special 'please. wait.', but building it is fine

2. Since it's (of the ones that I've looked at) approximately 500 lines long, it's much, much, much more digestable than a lot of the vomit that comes out of so-called production systems. Those systems usually have only heard about typed Python in passing, and they believe it is a fad that will blow over. The ones in this repo aren't stellar about it, but at 500 lines it's easily achievable to type hint the code yourself, which can serve as an excellent learning opportunity

3. PyTorch offers some fun conversion tools, also, allowing one to compare-and-contrast how it executes under Torch versus ONNX https://docs.pytorch.org/docs/stable/onnx.html>, TorchScript https://docs.pytorch.org/docs/stable/generated/torch.jit.sav...>, CoreML https://apple.github.io/coremltools/docs-guides/source/conve...>, or a bazillion other competing frameworks

4. Related, one can play around with quantization and other "inference related" concerns (e.g. https://github.com/pytorch/ao#pytorch-native-training-to-ser... )

5. Further related, one can play around with the fine-tuning mentioned elsewhere, to better understand what is and isn't possible to achieve using that process. Because the code is digestable, and the models are reasonably sized (Qwen 0.6B weighs only 1.4GB and is Apache 2), it brings FAFO opportunities in ways that gpt-oss-20b (or bigger!) won't

I do appreciate that some of what I said may skate close to "ML engineer" concerns, so obviously your situation will be different, but for me having a better grip on how these things work enables me to have better conversations with my colleagues and also helps trip my bullshit detector when someone claims they're the second coming and are going to cure cancer or whatever

Re: Gemma 3 270M re-implemented in pure PyTorch for local tinkering

#55

Post by hodgehog11 »

dboon wrote: post #51

First, thanks for doing everything you do! I, and I’m sure countless others, genuinely benefit from you. How would you recommend someone with a strong background in undergraduate level traditional ML get into deep learning? I use that as a broad term to encompass all the knowledge needed to understand how these models work, starting from the deep learning models of a decade ago, plus the practical ability to collect…

As someone who has students that work in deep learning, I can say that it is unwise to approach deep learning in the same way as traditional ML. Most classical methods are strongly mathematically motivated and have excellent theory to accompany them. Deep learning is still alchemy; it is a matter of experience, trying things out and getting a feel for how the pieces fit together in a modular format. Once you are experienced with the common building blocks, you can develop an intuition for how they might be improved.

I would start with training a basic MLP on tabular data. Then switch to CNNs: LeNet, VGG, then ResNet. Understand each of the new blocks that are incorporated into each architecture and how they improve stability and training efficiency. There are good PyTorch tutorials for these. Use these as a playground to understand what each of the training knobs do. Look at how their implicit biases induce double descent; this should give you confidence that overfitting is rarely an issue anymore. Give finetuning a try by taking a pretrained ResNet on ImageNet, adding layers to the start and end, and training only these to adapt the model to another image dataset. This should demonstrate the power of finetuning and why pretrained models are so powerful.

Next, briefly consider a tutorial on LSTMs, recognizing the exploding and vanishing gradient problems and the traditional challenges with sequential data.

Then move to transformers. Work with language first, starting from Andrej Karpathy's excellent YouTube tutorials. Train the model in full for a bit, then see about using an existing GPT2 checkpoint. Try adapting NanoGPT to a mathematical dataset as an exercise. Then take a look at llm.c to see how to really improve performance.

Finally, take a look at ViT and DETR. Use pretrained models and finetune them on smaller datasets again.

By this point, you should have a good grounding to start reading much of the surrounding literature and understand them. You should also understand that models are never built from scratch anymore, and every model is a collection of individual pieces built elsewhere for a particular purpose.

Re: Gemma 3 270M re-implemented in pure PyTorch for local tinkering

#56

Post by hodgehog11 »

dcreater wrote: post #49
canyon289 wrote: post #3

Hey all, I created this model with a top notch team. I answered many questions last week when this hit the front page, and happy to answer more here as well. https://news.ycombinator.com/item?id=44902148 Personally I'm excited that you all have access to this model now and hope you all get value out of using them.

As a non MLE, what are the pros/cons of OP's PyTorch re-implementation?

It is extremely valuable for researchers that commonly prototype theories using PyTorch on less powerful devices. Many of my colleagues run theory experiments using GPT-2 models. This allows for an easy transition to testing on a SOTA model instead.

Re: Gemma 3 270M re-implemented in pure PyTorch for local tinkering

#57

Post by khalic »

canyon289 wrote: post #3

Hey all, I created this model with a top notch team. I answered many questions last week when this hit the front page, and happy to answer more here as well. https://news.ycombinator.com/item?id=44902148 Personally I'm excited that you all have access to this model now and hope you all get value out of using them.

I'm going to have so much fun tinkering with it, thank you!!!

Re: Gemma 3 270M re-implemented in pure PyTorch for local tinkering

#59

Post by PoignardAzur »

canyon289 wrote: post #12
WithinReason wrote: post #9

Earlier quoted context omitted.

I would like to know your thoughts on using 2/3 of such a small the model's size for embeddings. What would be different if you used a byte-level vocabulary and spent the parameter budget on transformer parameters instead? I think you would lose performance (tok/s) but might gain accuracy.

At this small scale the embeddings indeed were a big focus. Consider this thought process. The tokens themselves are a form of compression. Lets say we have the word "WaffleHouse", character level this would be 11 tokens, but with an embedder this would be perhaps 2 or 3 tokens (I didn't actually run through the tokenizer but we could verify precisely). This matters a lot for on device processing especially. So while…

Does Gemma use any specific scheme to compress embeddings? Which have you considered?

For instance, it's well-known that transformer embeddings tend to form clusters. Have you considered splitting the embedding table into "cluster centroid" and "offset from centroid" tables, where the later would presumably have a smaller range and precision?

Post reply on HN