Do small language models run on cpus or you still need a gpus to run them?
Good results fine tuning a local LLM like Qwen 3:0.6B to categorize questions
11–20 of 54 posts
Re: Good results fine tuning a local LLM like Qwen 3:0.6B to categorize questions
#12[flagged]
Re: Good results fine tuning a local LLM like Qwen 3:0.6B to categorize questions
#13Has anyone compared recently doing something like ModernBERT plus classifier vs. full or lora FT of a small LM like qwen?
Re: Good results fine tuning a local LLM like Qwen 3:0.6B to categorize questions
#14But why using an encoder model instead of a BERT based model? For a pure classification that should be easier to train and work quite well
Re: Good results fine tuning a local LLM like Qwen 3:0.6B to categorize questions
#15“As an example, the question “When did we replace our pool pump?” will be mapped to a category called “pool” before querying the Index database.”
Cool write up! Really appreciate it but incidentally how does this categorization help you get better retrieval results?
Re: Good results fine tuning a local LLM like Qwen 3:0.6B to categorize questions
#16“As an example, the question “When did we replace our pool pump?” will be mapped to a category called “pool” before querying the Index database.” Cool write up! Really appreciate it but incidentally how does this categorization help you get better retrieval results?
Categorization allows for retrieval strategy
Re: Good results fine tuning a local LLM like Qwen 3:0.6B to categorize questions
#17Are 0.6b models useful without fine tuning?
Half of the times I ask qwen 0.6b "what is 1 + 2?" it ends up in a thinking loop of "but wait, the user is asking me to ..."
Re: Good results fine tuning a local LLM like Qwen 3:0.6B to categorize questions
#18If you are going to go to the bother of fine tuning for trivial problems like subject classification then I think you'll find Scikit Learn with a SGDClassifier on 2-grams will do probably just as well and be under 1MB for the trained classifier. You can train it in under a minute, and it will work perfectly well on embedded devices. Small LLMs are good choices for text classification in two cases: - If you next to pr…
there are models between 2-grams and 600m param models that would be good options. i don't expect a 2-gram to do very well here. also i'm not sure why this model isn't a fine choice if it solves their problem
What would you suggest instead?
Re: Good results fine tuning a local LLM like Qwen 3:0.6B to categorize questions
#19> The model invents new categories (e.g. apartments) and doesn’t stick to the provided list of allowed categories Can this specific failure mode be solved by providing a grammar that the output must adhere to? (Not sure if Qwen has this feature, it's used for eg. to ensure the output is parseable json)
It can. It's something that is implemented by the thing that runs the model - eg Llama.cpp - rather than the model itself. Note that it is hard to make work if you turn thinking on because the grammar gets complicated quickly (I don't recall if Qwen 0.6B can do thinking).
Thinking shouldn't be too hard to deal with---just let the model generate freely until it hits a token, then do constrained decoding, right?
Re: Good results fine tuning a local LLM like Qwen 3:0.6B to categorize questions
#20[flagged]