tangentially related: refact recently shared 4 bounties worth $9,000 to help improve their tech! https://algora.io/org/smallcloudai/bounties disclaimer: i'm a cofounder of algora, the platform enabling these bounties
[deleted]
Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval
71–80 of 111 posts
Re: Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval
#72All these LLMs are pretty general if I understand correctly. Are there any efforts to create specialized models (other than for coding)? Or, what would be even better, "extract" certain areas from existing LLMs as a way to specialize them? With the goal to drastically reduce model size to be able to run on less powerful devices.
E.g. a model specializing in chemistry doesn't need to include data on world's history or to be able to write poetry.
Re: Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval
#73One misleading thing is the notion that you need a 1-2B model to run on commodity hardware. This is not really true. Llama 7B runs with Vulkan/llama.cpp on ~8GB smartphones and ~12GB laptops. That ease is going to get much better over time, as lower RAM hardware starts dropping out of the market and the Vulkan implementations get more widespread. For users trying to run LLMs on 8GB or less machines, the AI Horde appr…
7b runs on my 4gb vram machine (8gb memory). I.e. quantization helps a lot too
Re: Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval
#74All these LLMs are pretty general if I understand correctly. Are there any efforts to create specialized models (other than for coding)? Or, what would be even better, "extract" certain areas from existing LLMs as a way to specialize them? With the goal to drastically reduce model size to be able to run on less powerful devices. E.g. a model specializing in chemistry doesn't need to include data on world's history or…
I am not an expert but it still has to learn human language/grammar/whathaveyou, and that is where scale seems to matter. Fine-tuning on a subset of knowledge after that is typically how the domain-specialisation is achieved, by my understanding.
Re: Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval
#75Another model that we'll soon forget it ever existed.
Re: Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval
#76Whats the difference between 1% and 99% of HumanEval? What does it tell really?
Re: Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval
#77One misleading thing is the notion that you need a 1-2B model to run on commodity hardware. This is not really true. Llama 7B runs with Vulkan/llama.cpp on ~8GB smartphones and ~12GB laptops. That ease is going to get much better over time, as lower RAM hardware starts dropping out of the market and the Vulkan implementations get more widespread. For users trying to run LLMs on 8GB or less machines, the AI Horde appr…
Ah, but have no fear - as lower RAM hardware starts dropping out of the market, the RAM usage of Microsoft Teams will increase to compensate!
(Not even /s - while the developers of LLM applications may have 64GB RAM in their laptops or desktops, the less-technical early adopters of LLMs running locally are likely to be power users with lower-powered laptops, much more stringent RAM limits, and numerous line-of-business applications and browser tabs contending for that RAM. Causing those applications to be swapped onto disk will almost certainly result in a degraded overall experience that could easily be blamed on the LLM application itself.)
Re: Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval
#78All these LLMs are pretty general if I understand correctly. Are there any efforts to create specialized models (other than for coding)? Or, what would be even better, "extract" certain areas from existing LLMs as a way to specialize them? With the goal to drastically reduce model size to be able to run on less powerful devices. E.g. a model specializing in chemistry doesn't need to include data on world's history or…
I am not an expert but it still has to learn human language/grammar/whathaveyou, and that is where scale seems to matter. Fine-tuning on a subset of knowledge after that is typically how the domain-specialisation is achieved, by my understanding.
Domain specialization is done by continuing the full training process. Fine tuning is more for changing the style of the output than adding new knowledge.
Re: Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval
#79The title is misleading This model is not "SOTA for the size", there are smaller models that do 10-18% better in absolute score. The text says it's SOTA "among similar models" where they probably compare with other models with permissive licensing.
AFAIK There is only one model that do better, it’s phi-1 and it’s python only, and it does not support fill-in-the-middle so you can't really use it.
Phi-1-small also scores higher with 350M parameters. It helps to be specific about what the comparison is against when claiming SOTA.
Re: Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval
#80All these LLMs are pretty general if I understand correctly. Are there any efforts to create specialized models (other than for coding)? Or, what would be even better, "extract" certain areas from existing LLMs as a way to specialize them? With the goal to drastically reduce model size to be able to run on less powerful devices. E.g. a model specializing in chemistry doesn't need to include data on world's history or…
I am not an expert but it still has to learn human language/grammar/whathaveyou, and that is where scale seems to matter. Fine-tuning on a subset of knowledge after that is typically how the domain-specialisation is achieved, by my understanding.
I was wondering about that too. Would it be possible in the future to have a more modular approach to LLMs? Have a module that is responsible for basic knowledge/language/grammar and then other more specialized modules that are added selectively.
I don't know enough about fine-tuning, not sure if the process is capable of removing "unused" parts of the model (I guess not possible, similar to un-learning).