11–20 of 33 posts
Does someone have a good understanding how 2B models can be useful in production? What tasks are you using them for? I wonder what tasks you can fine-tune them on to produce 95-99% results (if anything).
Is there a technical reason for it or just research convenience ?
Does someone have a good understanding how 2B models can be useful in production? What tasks are you using them for? I wonder what tasks you can fine-tune them on to produce 95-99% results (if anything).
The use case for small models include sentiment and intent analysis, spam and abuse detection, and classifications of various sorts. Generally LLM are thought of as chat models but the output need not be a conversation per se.
From some article I have in my draft, experimenting with open source text embeddings:
./match venture capital
purchase 0.74005488647684
sale 0.80926752301733
place 0.81188663814236
positive sentiment 0.90793311875207
negative sentiment 0.91083707598925
time 0.9108697315425
./store sillicon valley
./match venture capital
sillicon valley 0.7245139487301
purchase 0.74005488647684
sale 0.80926752301733
place 0.81188663814236
positive sentiment 0.90793311875207
negative sentiment 0.91083707598925
time 0.9108697315425
Is there a reason why the 1.58 ones are always aimed at quite small ones? Think I’ve seen an 8B but that’s about it. Is there a technical reason for it or just research convenience ?
Custom silicon would solve that, but nobody wants to build custom silicon for a data format that will go out of fashion before the production run is done.
Is there a reason why the 1.58 ones are always aimed at quite small ones? Think I’ve seen an 8B but that’s about it. Is there a technical reason for it or just research convenience ?
I suspect because current GPU hardware can't efficiently train such low bit depth models. You end up needing activations to use 8 or 16 bits in all the data paths, and don't get any more throughput per cycle on the multiplications than you would have done with FP32. Custom silicon would solve that, but nobody wants to build custom silicon for a data format that will go out of fashion before the production run is done…
Is there a reason why the 1.58 ones are always aimed at quite small ones? Think I’ve seen an 8B but that’s about it. Is there a technical reason for it or just research convenience ?
I suspect because current GPU hardware can't efficiently train such low bit depth models. You end up needing activations to use 8 or 16 bits in all the data paths, and don't get any more throughput per cycle on the multiplications than you would have done with FP32. Custom silicon would solve that, but nobody wants to build custom silicon for a data format that will go out of fashion before the production run is done…
I know it's not chatGPT4 but I've tried other very small models that run on CPU only and had better results
I built it at home this morning and tried it, perhaps my expectations were high but I wasn't terribly impressed. I asked it for a list of ten types of data I might show on a home info display panel. It gave me three. I clarified that I wanted ten, it gave me six. Every request after that just returned the same six things. I know it's not chatGPT4 but I've tried other very small models that run on CPU only and had bet…
Maybe you can you share some comparative examples?
Does someone have a good understanding how 2B models can be useful in production? What tasks are you using them for? I wonder what tasks you can fine-tune them on to produce 95-99% results (if anything).
I'm more interested in how users are taking 95-99% to 99.99% for generation-assisted tasks. I haven't seen a review or study of techniques, even though on the ground it's pretty trivial to think of some candidates.
- Use LLM to evaluate result and retry if it doesn't match.
- let users trigger a retry
- let users edit