Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemma 3 270M re-implemented in pure PyTorch for local tinkering

github.com

Post reply on HN
Print view View original

41–50 of 62 posts

Re: Gemma 3 270M re-implemented in pure PyTorch for local tinkering

#41

Post by nolist_policy »

shekhar101 wrote: post #18

Can someone (or OP) point me to a recipe to fine tune a model like this for natural language tasks like complicated NER or similar workflows? I tried finetuning Gemma3 270M when it came out last week without any success. A lot of tutorials are geared towards chat applications and role playing but I feel this model could be great for usecases like mine where I am trying to extract clean up and extract data from PDFs w…

This is using the gemma-llm python library which uses JAX in the background: https://gemma-llm.readthedocs.io/en/latest/colab_finetuning....

Re: Gemma 3 270M re-implemented in pure PyTorch for local tinkering

#42

Post by ModelForge »

ladberg wrote: post #37
lsb wrote: post #10

That’s wild that with a KV cache and compilation on the Mac CPU you are faster than on an A100 GPU.

Given that the compiled version is slower than then eager version on A100, there's definitely something suboptimal happening there

No the compiled version is actually faster.

From that table, the A100 tok/sec (larger is faster) numbers are:

- Eager: 28

- Compiled: 128

And

- KV cache eager: 26

- KV cache compiled: 99

The reason that the KV cache is slower is likely because it's not GPU-optimized code. On CPU the KV cache is faster. To make it faster on GPU, you would pre-allocate the tensors on the device for example instead of `torch.cat`ting them on the fly

Re: Gemma 3 270M re-implemented in pure PyTorch for local tinkering

#43

Post by lgessler »

shekhar101 wrote: post #18

Can someone (or OP) point me to a recipe to fine tune a model like this for natural language tasks like complicated NER or similar workflows? I tried finetuning Gemma3 270M when it came out last week without any success. A lot of tutorials are geared towards chat applications and role playing but I feel this model could be great for usecases like mine where I am trying to extract clean up and extract data from PDFs w…

If you're really just doing traditional NER (identifying non-overlapping spans of tokens which refer to named entities) then you're probably better off using encoder-only (e.g. https://huggingface.co/dslim/bert-large-NER) or encoder-decoder (e.g. https://huggingface.co/dbmdz/t5-base-conll03-english) models. These models aren't making headlines anymore because they're not decoder-only, but for established NLP tasks like this which don't involve generation, I think there's still a place for them, and I'd assume that at equal parameter counts they quite significantly outperform decoder-only models at NER, depending on the nature of the dataset.

Re: Gemma 3 270M re-implemented in pure PyTorch for local tinkering

#44

Post by [deleted] »

numpad0 wrote: post #20
n0vella wrote: post #5

Do you think these very small models have some utility in the real world? Apart from learning and academic purposes of course.

It seems to be more often correct than wrong for multilingual translation tasks(source text from[1][2]). Rough, but probably useful as traveler's phrase books. You Translate to English: Рошешуа́р (фр. Rochechouart) — місто, муніципалітет у Франції, у регіоні Нова Аквітанія, департамент Верхня В'єнна. Населення — 3637 осіб (2022)[1]. Муніципалітет розташований на відстані[2] близько 360 км на південь від Парижа, 34 км…

[deleted]

Re: Gemma 3 270M re-implemented in pure PyTorch for local tinkering

#45

Post by magicalhippo »

numpad0 wrote: post #20
n0vella wrote: post #5

Do you think these very small models have some utility in the real world? Apart from learning and academic purposes of course.

It seems to be more often correct than wrong for multilingual translation tasks(source text from[1][2]). Rough, but probably useful as traveler's phrase books. You Translate to English: Рошешуа́р (фр. Rochechouart) — місто, муніципалітет у Франції, у регіоні Нова Аквітанія, департамент Верхня В'єнна. Населення — 3637 осіб (2022)[1]. Муніципалітет розташований на відстані[2] близько 360 км на південь від Парижа, 34 км…

For comparison, here's what I got from the 27B variant:

  gemma3:27b-it-qat
  Rochechouart (French: Rochechouart) is a town and commune in France, in the Nouvelle-Aquitaine region, Department of Haute-Vienne. The population is 3,637 (2022)[1]. The commune is located approximately 360 km south of Paris, 34 km west of Limoges.

  gemma3:27b-it-qat
  The Ministry of Industry and Trade proposes supporting households installing rooftop solar power systems, with a maximum support of 2-3 million VND for systems including energy storage. This support would also include interest rate subsidies on loans for a maximum of three years. This content was presented by the Ministry of Industry and Trade in the third draft of a Decision by the Prime Minister regarding support policies for households installing self-generated, self-consumed rooftop solar power systems and energy storage systems.

Re: Gemma 3 270M re-implemented in pure PyTorch for local tinkering

#46

Post by torben-friis »

canyon289 wrote: post #31
torben-friis wrote: post #26

This might be a very basic question, but as a dev whose only interaction with models is using the main commercial ones (sonnet, ChatGPT and the like), what are some usecases for these smaller local models? What usages can be reasonable to expect from them? Are there uses out of the box or does one have to go through some custom post-training to get useful behavior? I feel like there is a huge gap between understandin…

Its a crucial question. I wrote up a long answer here. Let me know it helps https://news.ycombinator.com/item?id=44913558

Thanks for the reply!

It does help to figure out where in the space this model fits. I'm still a bit confused about this part:

>since it needs to be shaped to match specific tasks, we did our best to design it to be a flexible starting point for LLM-style tasks and worked with partners to put it into the right frameworks and places for you all to be able to shape it to what you need it to be.

What does shaping mean in this case? What tools are used, what requirements are there, both in terms of hardware and knowledge?

I would like to go beyond being spoonfed by large companies' high usability products, both to improve my knowledge and not be a victim of potential future rug pulls. In the classic software world, I guess the equivalent would be someone who runs open source software navigating the extra complexity, and ocassionally collaborates with the projects.

But I don't know what that looks like in the AI world. I've gone through some courses on machine learning but learning the basics about hessian matrices and gradient descent seems as detached from the practical point I'm searching as taking a compilers class is from learning React, so I think I've been looking in the wrong places (?).

Re: Gemma 3 270M re-implemented in pure PyTorch for local tinkering

#47

Post by ladberg »

ModelForge wrote: post #42
ladberg wrote: post #37

Earlier quoted context omitted.

Given that the compiled version is slower than then eager version on A100, there's definitely something suboptimal happening there

No the compiled version is actually faster. From that table, the A100 tok/sec (larger is faster) numbers are: - Eager: 28 - Compiled: 128 And - KV cache eager: 26 - KV cache compiled: 99 The reason that the KV cache is slower is likely because it's not GPU-optimized code. On CPU the KV cache is faster. To make it faster on GPU, you would pre-allocate the tensors on the device for example instead of `torch.cat`ting th…

Ah yep read the labels backwards and meant that - ty for catching and for the explanation

Re: Gemma 3 270M re-implemented in pure PyTorch for local tinkering

#48

Post by bigyabai »

canyon289 wrote: post #3

Hey all, I created this model with a top notch team. I answered many questions last week when this hit the front page, and happy to answer more here as well. https://news.ycombinator.com/item?id=44902148 Personally I'm excited that you all have access to this model now and hope you all get value out of using them.

Thanks for making this! One of my favorite projects was having a Discord chatbot powered by the original BERT model - these 270M weights are a fine upgrade.

Re: Gemma 3 270M re-implemented in pure PyTorch for local tinkering

#49

Post by dcreater »

canyon289 wrote: post #3

Hey all, I created this model with a top notch team. I answered many questions last week when this hit the front page, and happy to answer more here as well. https://news.ycombinator.com/item?id=44902148 Personally I'm excited that you all have access to this model now and hope you all get value out of using them.

As a non MLE, what are the pros/cons of OP's PyTorch re-implementation?
Post reply on HN