Tangentially related, but the UK Gov Incubator for AI has quite a nifty LLM driven classification pipeline for survey answers.
Good results fine tuning a local LLM like Qwen 3:0.6B to categorize questions
31–40 of 54 posts
Re: Good results fine tuning a local LLM like Qwen 3:0.6B to categorize questions
#32If you are going to go to the bother of fine tuning for trivial problems like subject classification then I think you'll find Scikit Learn with a SGDClassifier on 2-grams will do probably just as well and be under 1MB for the trained classifier. You can train it in under a minute, and it will work perfectly well on embedded devices. Small LLMs are good choices for text classification in two cases: - If you next to pr…
Not with 800 examples. If you are going to consider an ngram model, I think you are better off getting a frontier llm to write you an absurd regex.
I would also recommend the approach of using an llm to create the examples, and then train from there.
You can even get fancy and do things like active learning with the llm taking the role of the human annotator and sending in trial statements (and you can use a cheap one for larger gen and a more expensive one for the classification).
I’d be interested in seeing how well LLMs work with writing things like code for what snorkel AI used to have (there was open source code a while back that I assume is still around somewhere, you wrote code that was a low quality set of classifiers and it trained a model around those)
Re: Good results fine tuning a local LLM like Qwen 3:0.6B to categorize questions
#33Earlier quoted context omitted.
there are models between 2-grams and 600m param models that would be good options. i don't expect a 2-gram to do very well here. also i'm not sure why this model isn't a fine choice if it solves their problem
What would you suggest instead?
A non-autoregressive transformer trained with a classification objective.
Re: Good results fine tuning a local LLM like Qwen 3:0.6B to categorize questions
#34Do small language models run on cpus or you still need a gpus to run them?
iGPUs are often slower or only as fast as CPUs when it comes to LLM text generation.
The advantage is mainly in memory bandwidth. External GPUs' internal memory is slightly faster than DDR attached to your CPU.
Other types of "AI" models do make use of the extra compute in GPUs but not LLMs.
Re: Good results fine tuning a local LLM like Qwen 3:0.6B to categorize questions
#35Do small language models run on cpus or you still need a gpus to run them?
Anything below one billion parameters you can run on the CPU at acceptable speed
For larger sizes you still can, it just becomes slower and slower. For a simple classification task (small input, tiny output, and you can constrain output to a couple tokens) you could even run something like a 4B or 8B model on the CPU
Re: Good results fine tuning a local LLM like Qwen 3:0.6B to categorize questions
#36> The model invents new categories (e.g. apartments) and doesn’t stick to the provided list of allowed categories Can this specific failure mode be solved by providing a grammar that the output must adhere to? (Not sure if Qwen has this feature, it's used for eg. to ensure the output is parseable json)
This was my thought as well. I'm surprised that it's not being used here (afaict)
Re: Good results fine tuning a local LLM like Qwen 3:0.6B to categorize questions
#37[dead]
Re: Good results fine tuning a local LLM like Qwen 3:0.6B to categorize questions
#38“As an example, the question “When did we replace our pool pump?” will be mapped to a category called “pool” before querying the Index database.” Cool write up! Really appreciate it but incidentally how does this categorization help you get better retrieval results?
Categorization allows for retrieval strategy
because you have a different vector store for each categorization?
What if the question crosses categories?
Re: Good results fine tuning a local LLM like Qwen 3:0.6B to categorize questions
#39If you are going to go to the bother of fine tuning for trivial problems like subject classification then I think you'll find Scikit Learn with a SGDClassifier on 2-grams will do probably just as well and be under 1MB for the trained classifier. You can train it in under a minute, and it will work perfectly well on embedded devices. Small LLMs are good choices for text classification in two cases: - If you next to pr…
A small transformer like BERT or variants is a better fit. It only takes a few examples, which can be generated synthetically using an LLM.
Trains quickly and classifies speedily on modern hardware.
Had a lot of fun doing stuff like this years ago, before LLMs were a thing.
Re: Good results fine tuning a local LLM like Qwen 3:0.6B to categorize questions
#40Is it just me or half these comments read like AI