Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval

refact.ai

Post reply on HN
Print view View original

51–60 of 111 posts

Re: Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval

#51

Post by breadsniffer01 »

SparkyMcUnicorn wrote: post #29
ldjkfkdsjnv wrote: post #15

I dont trust any benchmarks for any LLM thats not coming from FB, Google, OpenAI, Anthropic, or Microsoft. These models are so dynamic, the simple benchmark numbers never tell the whole story of the quality of the model. Take for instance, a recent posting by anyscale, claiming their fine tuning of Llama 2 was competitive with OpenAI's model. The reality being their fined tuned model is basically worthless, and was c…

The community has fine-tuned some really good llama models (much better than llama-chat), but I get what you're saying. I've been testing the best performing models on the huggingface leaderboard lately. Some of them are really impressive, and others are so bad that I second guess the prompt format or if the benchmarked model is actually the same one I'm testing.

Which models were really bad?

Re: Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval

#53

Post by stainablesteel »

_xnmw wrote: post #9

For the sake of not giving Microsoft and a few other tech giants immense power over the world, I really do hope the cost and efficiency of LLMs improve dramatically, until we can get GPT-4-equivalent models trained on a few graphics cards and running offline on an iPhone. Really rooting for these kinds of projects until someone makes the breakthrough.

to be fair, if that is achieved then the massive models that tech giants produce will probably be phenomenal

Re: Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval

#54

Post by k__ »

pawelduda wrote: post #41
taywrobel wrote: post #19

Earlier quoted context omitted.

You may be interested in what we’re working on at Symbolica AI. We’re using formal logic in the form of abstract rewrite systems over a causal graph to perform geometric deep learning. In theory it should be able to learn the same topological structure of data that neural networks do, but using entirely discrete operations and without the random walk inherent to stochastic gradient descent. Current experiments are re…

Sounds cool, but what are the drawbacks?

It doesn't exist at scale yet.

Re: Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval

#55

Post by TuringNYC »

mholubowski wrote: post #27

Hey, I have a genuine question: What is the point of a new model that isn’t better than the best possible model (example: OpenAI GPT-4)? What’s the point in having a smaller model? Who cares? —- This is a real, genuine question that I don’t have a clear answer to. Excuse my ignorance, plz enlighten your boi.

The other answers are great, but to add more

- You can run it behind an air-gap, where your systems are disconnected from the world.

- You can run it on the edge with low or no internet connectivity

- You do not need to worry about breaching geographic data restrictions, e.g.: medical data from Country X cannot leave Country X

Re: Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval

#56

Post by vikp »

This post is misleading, in a way that is hard to do accidentally.

  - They compare the performance of this model to the worst 7B code llama model.  The base code llama 7B python model scores 38.4% on humaneval, versus the non-python model, which only scores 33%.
  - They compare their instruct tuned model to non-instruct-tuned models.  Instruction tuning can add 20% or more to humaneval performance.  For example, WizardLM 7B scores 55% on humaneval [1], and I've trained a 7B model that scores 62% [2].
  - For another example of instruction tuning, Stablecode instruct tuned benchmarks at 26%, not the 20% they cite for the base model [3]
  - Starcoder, when prompted properly, scores 40% on humaneval [4]
  - They do not report their base model performance (as far as I can tell)
This is interesting work, and a good contribution, but it's important to compare similar models.

[1] https://github.com/nlpxucan/WizardLM

[2] https://huggingface.co/vikp/llama_coder

[3] https://stability.ai/blog/stablecode-llm-generative-ai-codin...

[4] https://github.com/huggingface/blog/blob/main/starcoder.md

Re: Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval

#57

Post by [deleted] »

_xnmw wrote: post #9

For the sake of not giving Microsoft and a few other tech giants immense power over the world, I really do hope the cost and efficiency of LLMs improve dramatically, until we can get GPT-4-equivalent models trained on a few graphics cards and running offline on an iPhone. Really rooting for these kinds of projects until someone makes the breakthrough.

[deleted]

Re: Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval

#58

Post by [deleted] »

kateklink wrote: post #2

We’ve finished training a new code model Refact LLM which took us about a month. The main use-case is for blazing-fast code completion with fill-in-the-middle, additionally, the model could reply to chat prompts. It has much better performance than all of the code models of similar size, and almost reaches the same HumanEval as Starcoder being 10x smaller in size. With the small size, it can work with most modern GPU…

[deleted]

Re: Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval

#59

Post by ttul »

axpy906 wrote: post #24
_xnmw wrote: post #9

For the sake of not giving Microsoft and a few other tech giants immense power over the world, I really do hope the cost and efficiency of LLMs improve dramatically, until we can get GPT-4-equivalent models trained on a few graphics cards and running offline on an iPhone. Really rooting for these kinds of projects until someone makes the breakthrough.

The key in that is models. Per the GPT4 leaked details, it’s not a a single model but 16 MOE mixture of experts. There’s probably quite a lot of complexity on the backend in sourcing the right model for the right query. In short, it’s probably better to focus on single models for specific tasks in the OS community as evidenced by Code Llama. Having a system like GPT4 is still difficult to replicate. Getting it to run…

There are many MoE architectures and I suppose we don’t know for sure which OpenAI is using. The “selection” of the right mix of models is something that a network learns and it’s not a complex process. Certainly no more complex than training an LLM.

Re: Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval

#60

Post by OlegKlimov1337 »

_lvbh wrote: post #52

Say I want to fine tune a Golang specific model. How much $ and effort would I have to put in? Would using this as a base help in any way compared to starting from llama?

Maybe it makes sense to start from llama-code, not llama :D I think golang specific model will not be that different from a multi-language model. But it definitely will work better after fine tuning on your code. Check out refact self hosting docker in a couple of days, finetune will be there soon. It will take you 1 GPU and almost no money )
Post reply on HN