Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Qwen3.5 Fine-Tuning Guide

unsloth.ai

Post reply on HN
Print view View original

51–60 of 118 posts

Re: Qwen3.5 Fine-Tuning Guide

#51

Post by arkmm »

azath92 wrote: post #9
clueless wrote: post #2

What are some sample real world cases folks are using to fine tune their own small/medium models?

Only to prompt thought on this exact question, im interested in answers: I just ran a benchmark against haiku of a very simple document classification task that at the moment we farm out to haiku in parallel. very naive same prompt system via same api AWS bedrock, and can see that the a few of the 4b models are pretty good match, and could be easily run locally or just for cheap via a hosted provider. The "how much d…

You can fine tune a small LLM with a few thousand examples in just a few hours for a few dollars. It can be a bit tricky to host, but if you share a rough idea of the volume and whether this needs to be real-time or batched, I could list some of the tradeoffs you'd think about.

Source: Consulted for a few companies to help them finetune a bunch of LLMs. Typical categorical / data extraction use cases would have ~10x fewer errors at 100x lower inference cost than using the OpenAI models at the time.

Re: Qwen3.5 Fine-Tuning Guide

#52

Post by HanClinto »

bravura wrote: post #32
antirez wrote: post #6

Fine tuning is a story that is nice to tell but that with modern LLMs makes less and less sense. Modern LLMs are so powerful that they are able to few shot learn complicated things, so a strong prompt and augmenting the generation (given the massive context window of Qwen3.5, too) is usually the best option available. There are models for which fine tuning is great, like image models: there with LoRa you can get good…

For me, trying to fine-tune a model to write "best day" prose I would accept over 80% of the time. You are correct if we are talking about knowledge. However it is bad at hyper-idiosyncratic, gritty style transfer. I first noticed the issue when asking claude code to draft email responses. The choice of register was off. ("Register in writing refers to the level of formality and tone chosen to suit a specific audienc…

How well would you say it worked? I do like the idea of taking my historical forum posts and e-mails and whatnot and training an autocomplete LLM that is specifically "my voice".

Re: Qwen3.5 Fine-Tuning Guide

#53

Post by _the_inflator »

prettyblocks wrote: post #8
antirez wrote: post #6

Fine tuning is a story that is nice to tell but that with modern LLMs makes less and less sense. Modern LLMs are so powerful that they are able to few shot learn complicated things, so a strong prompt and augmenting the generation (given the massive context window of Qwen3.5, too) is usually the best option available. There are models for which fine tuning is great, like image models: there with LoRa you can get good…

I think the biggest case for fine tuning is probably that you can take small models, fine tune them for applications that require structured output, and then run cheap inference at scale. "Frontier LLMs can do it with enough context" is not really a strong argument against fine-tuning, because they're expensive to run.

I agree.

Also for certain use cases there are constraints like embedded hardware systems with no internet access. These LLMs have to be trained to specialize for clearly defined use cases under hardware constraints.

Frontier LLMs also are rarely function in isolation instead are orchestrating a system of special units aka subsystems and agents.

While costs and effort are one thing, being able to downsize these monster LLMs through finetuning itself in the first place is extremly valuable.

Re: Qwen3.5 Fine-Tuning Guide

#54

Post by andriy_koval »

prettyblocks wrote: post #8
antirez wrote: post #6

Fine tuning is a story that is nice to tell but that with modern LLMs makes less and less sense. Modern LLMs are so powerful that they are able to few shot learn complicated things, so a strong prompt and augmenting the generation (given the massive context window of Qwen3.5, too) is usually the best option available. There are models for which fine tuning is great, like image models: there with LoRa you can get good…

I think the biggest case for fine tuning is probably that you can take small models, fine tune them for applications that require structured output, and then run cheap inference at scale. "Frontier LLMs can do it with enough context" is not really a strong argument against fine-tuning, because they're expensive to run.

> "Frontier LLMs can do it with enough context" is not really a strong argument against fine-tuning, because they're expensive to run.

I am not expert in this topic, but I am wondering if large cached context is actually cheap to run and frontier models would be cost efficient too in such setting?

Re: Qwen3.5 Fine-Tuning Guide

#56

Post by embedding-shape »

0cf8612b2e1e wrote: post #47
embedding-shape wrote: post #45

Earlier quoted context omitted.

But why would I want to results to be done faster but less reliable, vs slower and more reliable? Feels like the sort of thing you'd favor accuracy over speed, otherwise you're just degrading the quality control?

Local, offline system you control is worth a lot. Introducing an external dependency guarantees you will have downtime outside of your control.

Right, but that doesn't answer why you'd need a fast 7b LLM rather than a slightly less fast 14b LLM.

Re: Qwen3.5 Fine-Tuning Guide

#57

Post by lelanthran »

yw3410 wrote: post #49
throwaway6977 wrote: post #10

Earlier quoted context omitted.

I agree- I'm currently trying to learn how I can embed a fine tuned tiny model into my c++ game so it can provide a narrative in prose of certain game-event logs. It needs to be as tiny as possible so it doesn't take resources away from the running game.

How small a model are we talking? Don't even the smallest models which would work need gigabytes of memory?

> How small a model are we talking? Don't even the smallest models which would work need gigabytes of memory?

I dunno, for game prose I expect that a tiny highly quantized model would be sufficient (generating no more than a paragraph), so 300MB - 500MB maybe? Running on CPU not GPU is feasible too, I think.

Re: Qwen3.5 Fine-Tuning Guide

#58

Post by 0xbadcafebee »

sorenjan wrote: post #50
someotherperson wrote: post #44

Earlier quoted context omitted.

At a very high level, think fruit sorting[0] where the conveyor belt doesn't stop rolling and you need to rapidly respond, and all the way through to monitoring for things like defects in silicon wafers and root causing it. Some of these issues aren't problematic on their own, but you can aggregate data over time to see if a particular machine, material or process within a factory is degrading over time. This might n…

But that's not something you'd use an LLM for. There have been computer vision systems sorting bad peas for more than a decade[0], of course there are plenty of use cases for very fast inspection systems. But when would you use an LLM for anything like that? [0] https://www.youtube.com/watch?v=eLDxXPziztw

You would use a VLM (vision language model). The model analyzes the image and outputs text, along with general context, that can drive intelligent decisions. https://tryolabs.com/blog/llms-leveraging-computer-vision

Re: Qwen3.5 Fine-Tuning Guide

#59

Post by 0xbadcafebee »

embedding-shape wrote: post #56
0cf8612b2e1e wrote: post #47

Earlier quoted context omitted.

Local, offline system you control is worth a lot. Introducing an external dependency guarantees you will have downtime outside of your control.

Right, but that doesn't answer why you'd need a fast 7b LLM rather than a slightly less fast 14b LLM.

....because sometimes people need a faster answer? There's many possible reasons someone might need speed over accuracy. In the food sorting example, if lower accuracy means you waste more peanuts, but the speed means you get rid of more bad peanuts overall, then you get fewer complaints about bad peanuts, with a tiny amount of extra material waste.

Re: Qwen3.5 Fine-Tuning Guide

#60

Post by 0cf8612b2e1e »

embedding-shape wrote: post #56
0cf8612b2e1e wrote: post #47

Earlier quoted context omitted.

Local, offline system you control is worth a lot. Introducing an external dependency guarantees you will have downtime outside of your control.

Right, but that doesn't answer why you'd need a fast 7b LLM rather than a slightly less fast 14b LLM.

In the hypothetical fruit sorting example, if you have a hard budget of 10 msec to respond and the 7B takes 8 msec and the 14B takes 12msec, there is your imaginary answer. Regular engineering where you have to balance competing constraints instead of running the biggest available.
Post reply on HN