Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval

refact.ai

Post reply on HN
Print view View original

21–30 of 111 posts

Re: Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval

#21

Post by fnordpiglet »

_xnmw wrote: post #9

For the sake of not giving Microsoft and a few other tech giants immense power over the world, I really do hope the cost and efficiency of LLMs improve dramatically, until we can get GPT-4-equivalent models trained on a few graphics cards and running offline on an iPhone. Really rooting for these kinds of projects until someone makes the breakthrough.

I think with or without algorithmic advantages hardware will improve for local model running. There’s an immense amount of capital being invested in hardware improvement and that will absolutely trickle down.

My sincere belief is that local models is the way of the future, with flexible base models adapted via Lora and context to specific use cases. I think open source models and techniques are inexorable at this point barring some sort of regulatory moat and will rival commercial models in all but extreme cases.

Re: Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval

#22

Post by OlegKlimov1337 »

howon92 wrote: post #11

Congrats on your achievement! I'm curious about your end goal. Do you aim to beat GitHub Copilot's performance and convince devs to use Refact for code completion instead of GitHub Copilot? I want to understand the motivation behind these different code-completion models that are not solely for academic research.

You can use it in practice, that was the goal of that particular model! It's fast, runs on your own hardware if you want it to.

Re: Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval

#23

Post by adrenvi »

_xnmw wrote: post #9

For the sake of not giving Microsoft and a few other tech giants immense power over the world, I really do hope the cost and efficiency of LLMs improve dramatically, until we can get GPT-4-equivalent models trained on a few graphics cards and running offline on an iPhone. Really rooting for these kinds of projects until someone makes the breakthrough.

That could also help tech giants build even larger/more capable models cheaply. Ideally there would be a hard ceiling of LLM capability that even massive amounts of hardware couldn't exceed, allowing inexpensive hardware to catch up.

Re: Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval

#24

Post by axpy906 »

_xnmw wrote: post #9

For the sake of not giving Microsoft and a few other tech giants immense power over the world, I really do hope the cost and efficiency of LLMs improve dramatically, until we can get GPT-4-equivalent models trained on a few graphics cards and running offline on an iPhone. Really rooting for these kinds of projects until someone makes the breakthrough.

The key in that is models. Per the GPT4 leaked details, it’s not a a single model but 16 MOE mixture of experts. There’s probably quite a lot of complexity on the backend in sourcing the right model for the right query. In short, it’s probably better to focus on single models for specific tasks in the OS community as evidenced by Code Llama. Having a system like GPT4 is still difficult to replicate. Getting it to run on a consumer hardware for specific tasks like code gen at almost GPT4 level is doable.

Re: Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval

#25

Post by smoldesu »

_xnmw wrote: post #9

For the sake of not giving Microsoft and a few other tech giants immense power over the world, I really do hope the cost and efficiency of LLMs improve dramatically, until we can get GPT-4-equivalent models trained on a few graphics cards and running offline on an iPhone. Really rooting for these kinds of projects until someone makes the breakthrough.

> For the sake of not giving Microsoft and a few other tech giants immense power over the world

I agree with and appreciate the sentiment, but it feels way too late for that. These people do have and exert direct control over pretty much all of our digital devices. It's funny (or sad) that we only seem to care about this when shiny doodads like AI come around every so-often.

Re: Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval

#26

Post by drcongo »

OlegKlimov1337 wrote: post #18
drcongo wrote: post #4

Earlier quoted context omitted.

Is it possible to run it as an LSP so that it can be used in editors other than VSCode and JetBrains? (sorry if this question is completely mad, my understanding of how these things work is extremely limited)

Yes, it's coming up in a couple of weeks.

Great, thanks. I'll keep an eye out.

Re: Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval

#27

Post by mholubowski »

Hey, I have a genuine question:

What is the point of a new model that isn’t better than the best possible model (example: OpenAI GPT-4)?

What’s the point in having a smaller model? Who cares?

—-

This is a real, genuine question that I don’t have a clear answer to. Excuse my ignorance, plz enlighten your boi.

Re: Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval

#28

Post by mrob »

umutisik wrote: post #14

The title is misleading This model is not "SOTA for the size", there are smaller models that do 10-18% better in absolute score. The text says it's SOTA "among similar models" where they probably compare with other models with permissive licensing.

"Permissive" usually refers to Free Software or Open Source licenses without copyleft requirements. OpenRAIL is a proprietary license because it imposes usage restrictions, contrary to both the Free Software and Open Source definitions.

Re: Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval

#29

Post by SparkyMcUnicorn »

ldjkfkdsjnv wrote: post #15

I dont trust any benchmarks for any LLM thats not coming from FB, Google, OpenAI, Anthropic, or Microsoft. These models are so dynamic, the simple benchmark numbers never tell the whole story of the quality of the model. Take for instance, a recent posting by anyscale, claiming their fine tuning of Llama 2 was competitive with OpenAI's model. The reality being their fined tuned model is basically worthless, and was c…

The community has fine-tuned some really good llama models (much better than llama-chat), but I get what you're saying.

I've been testing the best performing models on the huggingface leaderboard lately. Some of them are really impressive, and others are so bad that I second guess the prompt format or if the benchmarked model is actually the same one I'm testing.

Re: Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval

#30

Post by famouswaffles »

axpy906 wrote: post #24
_xnmw wrote: post #9

For the sake of not giving Microsoft and a few other tech giants immense power over the world, I really do hope the cost and efficiency of LLMs improve dramatically, until we can get GPT-4-equivalent models trained on a few graphics cards and running offline on an iPhone. Really rooting for these kinds of projects until someone makes the breakthrough.

The key in that is models. Per the GPT4 leaked details, it’s not a a single model but 16 MOE mixture of experts. There’s probably quite a lot of complexity on the backend in sourcing the right model for the right query. In short, it’s probably better to focus on single models for specific tasks in the OS community as evidenced by Code Llama. Having a system like GPT4 is still difficult to replicate. Getting it to run…

>There’s probably quite a lot of complexity on the backend in sourcing the right model for the right query.

This isn't how Sparse MoE models work. There isn't really any complexity like that. And different models will or can pick each token.

Sparse models aren't an ensemble of models.

Post reply on HN