If you're gonna fine-tune for a closed set classification problem like this, you could just fine-tune BERT and get a faster model with better performance.
Good results fine tuning a local LLM like Qwen 3:0.6B to categorize questions
41–50 of 54 posts
Re: Good results fine tuning a local LLM like Qwen 3:0.6B to categorize questions
#42Are 0.6b models useful without fine tuning? Half of the times I ask qwen 0.6b "what is 1 + 2?" it ends up in a thinking loop of "but wait, the user is asking me to ..."
If you don't want the thinking, you can pass `enable_thinking: false` to the `chat_template_kwargs`. If using promptfoo, this can be done via:
providers:
- # llama-server
id: openai:chat:qwen
config:
apiBaseUrl: http://localhost:7876
apiKey: "..."
passthrough:
chat_template_kwargs:
enable_thinking: false
Re: Good results fine tuning a local LLM like Qwen 3:0.6B to categorize questions
#43Earlier quoted context omitted.
What would you suggest instead?
A non-autoregressive transformer trained with a classification objective.
These are absurdly effective for this kind of task. Training is fast and straight forward. Packaging for deployment as ONNX is pretty simple as well.
Re: Good results fine tuning a local LLM like Qwen 3:0.6B to categorize questions
#44Are 0.6b models useful without fine tuning? Half of the times I ask qwen 0.6b "what is 1 + 2?" it ends up in a thinking loop of "but wait, the user is asking me to ..."
A fun thing I do with Qwen 3.5 0.8b is to take a screenshot of the Hackernews homepage and ask it to give me a JSON representation of the data and it does surprisingly well. With a well structured prompt I think it could be made to be pretty reliable tool for that type of task out of the box.
While a fun poc, surely it would be better to just use the API (see the footer)? Or just `curl | x2j | jq` and map the HTML directly to JSON?
Re: Good results fine tuning a local LLM like Qwen 3:0.6B to categorize questions
#45“As an example, the question “When did we replace our pool pump?” will be mapped to a category called “pool” before querying the Index database.” Cool write up! Really appreciate it but incidentally how does this categorization help you get better retrieval results?
Categorization allows for retrieval strategy
In a general sense, you see this crop up in SKILLS.md files and other places, as LLMs have to deal with broad contexts. People try and drill into some taxonomy in a naive fashion using plaintext as a directive, which is not particularly optimal.
I wonder if one could build a 'mixture of experts' at the model level that leveraged a variety of small models "within" a larger model...
Re: Good results fine tuning a local LLM like Qwen 3:0.6B to categorize questions
#46If you want to go deeper on language models, try these project ideas: - Zero-shot encoders like tasksource or GliNER - Natural language inference: https://huggingface.co/blog/dleemiller/nli-xenc-ways-to-use - GRPO training - GEPA prompt tuning Qwen 0.6B (or GEPA, then GRPO) - Use an embedding model and train a classifier (MLP, logistic, svm) - Use a larger LLM to generate a synthetic dataset (beware of lack of divers…
may I ask where did you get the list? I am looking for ways to get involved in going little more deeper on LLMs (I have very high level understanding, but my direct work doesn't involve them, hence I am not familiar with deeper details)
I'd been working with language models for several years before LLMs were a solution to this kind of problem. These are some ideas "off the top of my head" about how you can do classification in various ways. There's really a lot of ways to tackle it now, and a lot of trade-offs you can learn by experimenting with them.
There's even more options still, especially if you go further back toward more traditional methods. Static word vectors like GloVe or fasttext (optionally more modern equivalents like WordLlama or Model2Vec). Then there's sklearn-style stuff too. Those can be really small/fast but have more accuracy-level tradeoffs.
Re: Good results fine tuning a local LLM like Qwen 3:0.6B to categorize questions
#47Earlier quoted context omitted.
A fun thing I do with Qwen 3.5 0.8b is to take a screenshot of the Hackernews homepage and ask it to give me a JSON representation of the data and it does surprisingly well. With a well structured prompt I think it could be made to be pretty reliable tool for that type of task out of the box.
While a fun poc, surely it would be better to just use the API (see the footer)? Or just `curl | x2j | jq` and map the HTML directly to JSON?
Yes apologies, Hackernews was just an example, you can do this with any website - it’s just a simple benchmark I like to use for testing vision models.
Re: Good results fine tuning a local LLM like Qwen 3:0.6B to categorize questions
#48Earlier quoted context omitted.
Not with 800 examples. If you are going to consider an ngram model, I think you are better off getting a frontier llm to write you an absurd regex.
Hmm maybe. Turns out the author trained a logistic-regression classifier on the embeddings too, but didn't report the results: https://github.com/thelgevold/fine-tuned-classifier/blob/mai...
Expanding on this experiment using logistic regression is an interesting continuation, detailed here: https://www.teachmecoolstuff.com/viewarticle/using-logistic-...
In summary: Using logistic regression actually improves accuracy, but also performance during both runtime and during training.
Re: Good results fine tuning a local LLM like Qwen 3:0.6B to categorize questions
#49Earlier quoted context omitted.
A non-autoregressive transformer trained with a classification objective.
These are absurdly effective for this kind of task. Training is fast and straight forward. Packaging for deployment as ONNX is pretty simple as well.
As a follow up to the original article, I added a new experiment using Logistic Regression and the results are very good. It actually improves on the accuracy by a few points.
More details here: https://www.teachmecoolstuff.com/viewarticle/using-logistic-...
Re: Good results fine tuning a local LLM like Qwen 3:0.6B to categorize questions
#50Earlier quoted context omitted.
Thinking shouldn't be too hard to deal with---just let the model generate freely until it hits a token, then do constrained decoding, right?
Sure, but does llama-cpp support that?
It does and this is how I did it.
But actually getting that grammar right as well as actually making it work with the correct Jinja template to correctly enable thinking mode and parse it out was a lot more work than I expected.