Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval

refact.ai

Post reply on HN
Print view View original

61–70 of 111 posts

Re: Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval

#61

Post by ALittleLight »

kateklink wrote: post #2

We’ve finished training a new code model Refact LLM which took us about a month. The main use-case is for blazing-fast code completion with fill-in-the-middle, additionally, the model could reply to chat prompts. It has much better performance than all of the code models of similar size, and almost reaches the same HumanEval as Starcoder being 10x smaller in size. With the small size, it can work with most modern GPU…

How does it compare to Copilot? A metric I'd like to see is % of proposed completions accepted by a human user. If you had an extension that 50% of the time proposed a Copilot extension and 50% of the time proposed a Refact extension (blind to the user) then you could come up with a metric like this.

Re: Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval

#62

Post by qeternity »

yunwal wrote: post #42
mholubowski wrote: post #27

Hey, I have a genuine question: What is the point of a new model that isn’t better than the best possible model (example: OpenAI GPT-4)? What’s the point in having a smaller model? Who cares? —- This is a real, genuine question that I don’t have a clear answer to. Excuse my ignorance, plz enlighten your boi.

GPT4 is expensive to run, even more expensive to finetune, and for all practical purposes can’t be run offline (because the model is too big to run outside of a huge data center). Evaluation latency is also an issue for many usecases, and you have to share your query with openai, so you can’t run sensitive queries. The output is also controlled/censored by OpenAI. Here’s a few usecases that I wouldn’t want to use Ope…

> GPT4 is expensive to run, even more expensive to finetune

GPT4 can't even be finetuned at the moment (though I expect that to change).

Re: Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval

#63

Post by SparkyMcUnicorn »

breadsniffer01 wrote: post #51
SparkyMcUnicorn wrote: post #29

Earlier quoted context omitted.

The community has fine-tuned some really good llama models (much better than llama-chat), but I get what you're saying. I've been testing the best performing models on the huggingface leaderboard lately. Some of them are really impressive, and others are so bad that I second guess the prompt format or if the benchmarked model is actually the same one I'm testing.

Which models were really bad?

I was keeping track of the good ones, and don't have many notes on the bad ones.

I do remember testing "LoKuS" last week and it was quite terrible (sometimes gave completely off-topic answers). It scored as one of the highest 13B models on the leaderboard (~65 average), but appears to be removed now.

Re: Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval

#64

Post by brucethemoose2 »

One misleading thing is the notion that you need a 1-2B model to run on commodity hardware.

This is not really true. Llama 7B runs with Vulkan/llama.cpp on ~8GB smartphones and ~12GB laptops. That ease is going to get much better over time, as lower RAM hardware starts dropping out of the market and the Vulkan implementations get more widespread.

For users trying to run LLMs on 8GB or less machines, the AI Horde approach of distributed models seems much more practical anyway.

Re: Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval

#65

Post by taywrobel »

KRAKRISMOTT wrote: post #45
taywrobel wrote: post #19

Earlier quoted context omitted.

You may be interested in what we’re working on at Symbolica AI. We’re using formal logic in the form of abstract rewrite systems over a causal graph to perform geometric deep learning. In theory it should be able to learn the same topological structure of data that neural networks do, but using entirely discrete operations and without the random walk inherent to stochastic gradient descent. Current experiments are re…

> We’re using formal logic in the form of abstract rewrite systems over a causal graph to perform geometric deep learning. In theory it should be able to learn the same topological structure of data that neural networks do, but using entirely discrete operations and without the random walk inherent to stochastic gradient descent. Abstract rewrite like a computer algebra system's (e.g. Wolfram) term rewriting equation…

Heavily influenced by Wolfram's work on metamathematics and the physics project, in so far as using a rewrite system to uncover an emergent topology; we're just using it to uncover the topology of certain data (assuming that the manifold hypothesis is correct), rather than the topology of fundamental physics as he did.

Re: Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval

#66

Post by [deleted] »

mholubowski wrote: post #27

Hey, I have a genuine question: What is the point of a new model that isn’t better than the best possible model (example: OpenAI GPT-4)? What’s the point in having a smaller model? Who cares? —- This is a real, genuine question that I don’t have a clear answer to. Excuse my ignorance, plz enlighten your boi.

[deleted]

Re: Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval

#67

Post by taywrobel »

pawelduda wrote: post #41
taywrobel wrote: post #19

Earlier quoted context omitted.

You may be interested in what we’re working on at Symbolica AI. We’re using formal logic in the form of abstract rewrite systems over a causal graph to perform geometric deep learning. In theory it should be able to learn the same topological structure of data that neural networks do, but using entirely discrete operations and without the random walk inherent to stochastic gradient descent. Current experiments are re…

Sounds cool, but what are the drawbacks?

Biggest drawback is that since the structure is all discrete, it is inherently weak at modeling statistical distributions. For example, it'll likely never best a neural network at stock market prediction or medical data extrapolation.

However, for things that are discrete and/or causal in nature, we expect it to outperform deep learning by a wide margin. We're focused on language to start, but want to eventually target planning and controls problems as well, such as self-driving and robotics.

Another drawback is that the algorithm as it stands today is based on a subgraph isomorphism search, which is hard. Not hard as in tricky to get right like Paxos or other complex algorithms; like NP-Hard, so very difficult to scale. We have some fantastic Ph.Ds working with us who focus on optimization of subgraph isomorphism search, and category theorists working to formalize what constraints we can relax without effecting the learning mechanism of the rewrite system, so we're confident that it's achievable, but the time horizon is unknown currently.

Re: Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval

#68

Post by taywrobel »

paulsutter wrote: post #44
taywrobel wrote: post #19

Earlier quoted context omitted.

You may be interested in what we’re working on at Symbolica AI. We’re using formal logic in the form of abstract rewrite systems over a causal graph to perform geometric deep learning. In theory it should be able to learn the same topological structure of data that neural networks do, but using entirely discrete operations and without the random walk inherent to stochastic gradient descent. Current experiments are re…

Especially interested in learning directly on geometries, please keep us updated and share results

Would definitely recommend Bronstein et. al's work on geometric deep learning! https://geometricdeeplearning.com

That's effectively the right hand side of the bridge that we're building between formal logic and deep learning. So far their work has been viewed mainly as descriptive, helping to understand neural networks better, but as their abstract calls out: "it gives a constructive procedure to incorporate prior physical knowledge into neural architectures and provide principled way to build future architectures yet to be invented". That's us (we hope)!

Re: Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval

#69

Post by palmer_fox »

brucethemoose2 wrote: post #64

One misleading thing is the notion that you need a 1-2B model to run on commodity hardware. This is not really true. Llama 7B runs with Vulkan/llama.cpp on ~8GB smartphones and ~12GB laptops. That ease is going to get much better over time, as lower RAM hardware starts dropping out of the market and the Vulkan implementations get more widespread. For users trying to run LLMs on 8GB or less machines, the AI Horde appr…

Perhaps the wrong thread to ask this question... Is it not possible to load a model on something like an NVMe M.2 drive instead of RAM? It's slower of course, but only 5-10x if I understand correctly.
Post reply on HN