Congrats on your achievement! I'm curious about your end goal. Do you aim to beat GitHub Copilot's performance and convince devs to use Refact for code completion instead of GitHub Copilot? I want to understand the motivation behind these different code-completion models that are not solely for academic research.
Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval
11–20 of 111 posts
Re: Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval
#12License text: https://drive.google.com/file/d/16NqKiAkzyZ55NClubCIFup8pT2j... [PDF]
See last page for restrictions
Re: Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval
#13License text: https://drive.google.com/file/d/16NqKiAkzyZ55NClubCIFup8pT2j... [PDF] See last page for restrictions
Thanks. That looks pretty relaxed on terms
Re: Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval
#14The title is misleading This model is not "SOTA for the size", there are smaller models that do 10-18% better in absolute score. The text says it's SOTA "among similar models" where they probably compare with other models with permissive licensing.
Re: Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval
#15I dont trust any benchmarks for any LLM thats not coming from FB, Google, OpenAI, Anthropic, or Microsoft. These models are so dynamic, the simple benchmark numbers never tell the whole story of the quality of the model. Take for instance, a recent posting by anyscale, claiming their fine tuning of Llama 2 was competitive with OpenAI's model. The reality being their fined tuned model is basically worthless, and was competitive along a single metric/very narrow commoditized task. Its a great way to get clicks by posting these metrics though
Re: Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval
#16License text: https://drive.google.com/file/d/16NqKiAkzyZ55NClubCIFup8pT2j... [PDF] See last page for restrictions
> In any way that violates any applicable national, federal, state, local or international law or regulation;
Darn! Foiled again! I was planning on breaking some federal laws, but the license says that I can't ;( \s
Open-RAIL license has the be the worst license in existence claiming to be "open".
> You shall undertake reasonable efforts to use the latest version of the Model.
Plea to folks releasing models: Please stop using this user-hostile and deranged license
Re: Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval
#17We’ve finished training a new code model Refact LLM which took us about a month. The main use-case is for blazing-fast code completion with fill-in-the-middle, additionally, the model could reply to chat prompts. It has much better performance than all of the code models of similar size, and almost reaches the same HumanEval as Starcoder being 10x smaller in size. With the small size, it can work with most modern GPU…
> almost reaches the same HumanEval
how can you tell that HumanEval is not leaked to your training data in some form?
Re: Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval
#18We’ve finished training a new code model Refact LLM which took us about a month. The main use-case is for blazing-fast code completion with fill-in-the-middle, additionally, the model could reply to chat prompts. It has much better performance than all of the code models of similar size, and almost reaches the same HumanEval as Starcoder being 10x smaller in size. With the small size, it can work with most modern GPU…
Is it possible to run it as an LSP so that it can be used in editors other than VSCode and JetBrains? (sorry if this question is completely mad, my understanding of how these things work is extremely limited)
Yes, it's coming up in a couple of weeks.
Re: Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval
#19For the sake of not giving Microsoft and a few other tech giants immense power over the world, I really do hope the cost and efficiency of LLMs improve dramatically, until we can get GPT-4-equivalent models trained on a few graphics cards and running offline on an iPhone. Really rooting for these kinds of projects until someone makes the breakthrough.
You may be interested in what we’re working on at Symbolica AI.
We’re using formal logic in the form of abstract rewrite systems over a causal graph to perform geometric deep learning. In theory it should be able to learn the same topological structure of data that neural networks do, but using entirely discrete operations and without the random walk inherent to stochastic gradient descent.
Current experiments are really promising, and assuming the growth curve continues as we scale up you should be able to train a GPT-4 scale LLM in a few weeks on commodity hardware (we are using a desktop with 4 4090’s currently), and be able to do both inference and continual fine tuning/online learning on device.
Re: Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval
#20Congrats on your achievement! I'm curious about your end goal. Do you aim to beat GitHub Copilot's performance and convince devs to use Refact for code completion instead of GitHub Copilot? I want to understand the motivation behind these different code-completion models that are not solely for academic research.
we want to help developers who need either on-premise or permissive code assistant, copilot has neither of this. We also wanted to lower the barriers for self-hosting, so that the model is available on most GPUs with just 3GB Ram. Plus making the code completions fast and efficient (understanding entire context, not just the previous tokens).