Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Good results fine tuning a local LLM like Qwen 3:0.6B to categorize questions

teachmecoolstuff.com

Post reply on HN
Print view View original

21–30 of 54 posts

Re: Good results fine tuning a local LLM like Qwen 3:0.6B to categorize questions

#21

Post by nl »

djsjajah wrote: post #5
nl wrote: post #3

If you are going to go to the bother of fine tuning for trivial problems like subject classification then I think you'll find Scikit Learn with a SGDClassifier on 2-grams will do probably just as well and be under 1MB for the trained classifier. You can train it in under a minute, and it will work perfectly well on embedded devices. Small LLMs are good choices for text classification in two cases: - If you next to pr…

Not with 800 examples. If you are going to consider an ngram model, I think you are better off getting a frontier llm to write you an absurd regex.

Hmm maybe. Turns out the author trained a logistic-regression classifier on the embeddings too, but didn't report the results:

https://github.com/thelgevold/fine-tuned-classifier/blob/mai...

Re: Good results fine tuning a local LLM like Qwen 3:0.6B to categorize questions

#22

Post by thomascountz »

nextaccountic wrote: post #7

> The model invents new categories (e.g. apartments) and doesn’t stick to the provided list of allowed categories Can this specific failure mode be solved by providing a grammar that the output must adhere to? (Not sure if Qwen has this feature, it's used for eg. to ensure the output is parseable json)

Yes, you can use constrained decoding like logit masking to force all invalid tokens in the vocabulary to -inf, and effectively be removed from selection. I believe llama.cpp exposes this by accepting a formatted grammar.

Re: Good results fine tuning a local LLM like Qwen 3:0.6B to categorize questions

#23

Post by kamranjon »

zwaps wrote: post #13

Has anyone compared recently doing something like ModernBERT plus classifier vs. full or lora FT of a small LM like qwen?

I have! I recently compared Gemma 1b to ModernBERT Large for a binary classification task and ModernBERT was the clear winner. It learned faster and performed the task better by a significant margin by the end of training. It seems the bidirectional encoder only architecture works really well for classification tasks, and I think it is related to being bidirectional whereas decoder only models like Gemma (or Qwen) can only “look backwards”. I used a mixture of FFT and LoRA as well as a mixture of CE Loss and SupCon Loss.

Re: Good results fine tuning a local LLM like Qwen 3:0.6B to categorize questions

#24

Post by kamranjon »

throwa356262 wrote: post #17

Are 0.6b models useful without fine tuning? Half of the times I ask qwen 0.6b "what is 1 + 2?" it ends up in a thinking loop of "but wait, the user is asking me to ..."

A fun thing I do with Qwen 3.5 0.8b is to take a screenshot of the Hackernews homepage and ask it to give me a JSON representation of the data and it does surprisingly well. With a well structured prompt I think it could be made to be pretty reliable tool for that type of task out of the box.

Re: Good results fine tuning a local LLM like Qwen 3:0.6B to categorize questions

#26

Post by a96 »

abhashanand1501 wrote: post #11

Do small language models run on cpus or you still need a gpus to run them?

I guess that technically depends on the software used to run the model, but in general it's always been possible to run on a CPU (and may even be possible to run on TPU or something else). It's just been slower. Likewise GPU RAM vs system RAM and the bandwidths involved can make hard bottlenecks.

GPU and VRAM (or fast unified RAM) is generally the option that is both available and performant, but especially really small models also run quite well on CPU and system RAM.

Re: Good results fine tuning a local LLM like Qwen 3:0.6B to categorize questions

#28

Post by throwaw12 »

deepsquirrelnet wrote: post #6

If you want to go deeper on language models, try these project ideas: - Zero-shot encoders like tasksource or GliNER - Natural language inference: https://huggingface.co/blog/dleemiller/nli-xenc-ways-to-use - GRPO training - GEPA prompt tuning Qwen 0.6B (or GEPA, then GRPO) - Use an embedding model and train a classifier (MLP, logistic, svm) - Use a larger LLM to generate a synthetic dataset (beware of lack of divers…

may I ask where did you get the list? I am looking for ways to get involved in going little more deeper on LLMs (I have very high level understanding, but my direct work doesn't involve them, hence I am not familiar with deeper details)

Re: Good results fine tuning a local LLM like Qwen 3:0.6B to categorize questions

#29

Post by stymaar »

aesthesia wrote: post #19
nl wrote: post #8

Earlier quoted context omitted.

It can. It's something that is implemented by the thing that runs the model - eg Llama.cpp - rather than the model itself. Note that it is hard to make work if you turn thinking on because the grammar gets complicated quickly (I don't recall if Qwen 0.6B can do thinking).

Thinking shouldn't be too hard to deal with---just let the model generate freely until it hits a token, then do constrained decoding, right?

Sure, but does llama-cpp support that?

Re: Good results fine tuning a local LLM like Qwen 3:0.6B to categorize questions

#30

Post by armcat »

I mean it's always nice to play around with sLLM finetuning, but for practical purposes I would always start with a lazy learner using embeddings (something like a small Stella model), pre-embed the topics/categories, embed the question, perform a kNN using cosine distance. You can use an LLM to "expand" the topics before embedding to make them more contextual. This is usually super fast and super simple and gives you a nice baseline. Then I would add a classification head after embedding layer (with maybe some dropout + 2-3 MLP layers) and train my own classifier, and compare that to lazy learner. Only after that would I start finetuning an LLM.
Post reply on HN