Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval

refact.ai

Post reply on HN
Print view View original

81–90 of 111 posts

Re: Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval

#81

Post by palmer_fox »

charcircuit wrote: post #78
hnhg wrote: post #74

Earlier quoted context omitted.

I am not an expert but it still has to learn human language/grammar/whathaveyou, and that is where scale seems to matter. Fine-tuning on a subset of knowledge after that is typically how the domain-specialisation is achieved, by my understanding.

Domain specialization is done by continuing the full training process. Fine tuning is more for changing the style of the output than adding new knowledge.

What if the initial training already contains all necessary data for a particular specialization? What would be the benefit of continuing the training process?

Re: Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval

#82

Post by kirill5pol »

palmer_fox wrote: post #69
brucethemoose2 wrote: post #64

One misleading thing is the notion that you need a 1-2B model to run on commodity hardware. This is not really true. Llama 7B runs with Vulkan/llama.cpp on ~8GB smartphones and ~12GB laptops. That ease is going to get much better over time, as lower RAM hardware starts dropping out of the market and the Vulkan implementations get more widespread. For users trying to run LLMs on 8GB or less machines, the AI Horde appr…

Perhaps the wrong thread to ask this question... Is it not possible to load a model on something like an NVMe M.2 drive instead of RAM? It's slower of course, but only 5-10x if I understand correctly.

Yes but they’re slow enough on normal hardware for that 5-10x to be painful…

Re: Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval

#83

Post by kateklink »

holoduke wrote: post #76

Whats the difference between 1% and 99% of HumanEval? What does it tell really?

for pass@1 HumanEval tells how well the model solves a task from a set, given only one chance to solve it. It's not the perfect metric, there're other like DS-1000, MBPP (we have included them on HuggingFace model card). HumanEval is good for benchmarking with other models as it gives a fast idea how powerful the model is.

Re: Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval

#84

Post by [deleted] »

axpy906 wrote: post #24
_xnmw wrote: post #9

For the sake of not giving Microsoft and a few other tech giants immense power over the world, I really do hope the cost and efficiency of LLMs improve dramatically, until we can get GPT-4-equivalent models trained on a few graphics cards and running offline on an iPhone. Really rooting for these kinds of projects until someone makes the breakthrough.

The key in that is models. Per the GPT4 leaked details, it’s not a a single model but 16 MOE mixture of experts. There’s probably quite a lot of complexity on the backend in sourcing the right model for the right query. In short, it’s probably better to focus on single models for specific tasks in the OS community as evidenced by Code Llama. Having a system like GPT4 is still difficult to replicate. Getting it to run…

[deleted]

Re: Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval

#85

Post by axpy906 »

ttul wrote: post #59
axpy906 wrote: post #24

Earlier quoted context omitted.

The key in that is models. Per the GPT4 leaked details, it’s not a a single model but 16 MOE mixture of experts. There’s probably quite a lot of complexity on the backend in sourcing the right model for the right query. In short, it’s probably better to focus on single models for specific tasks in the OS community as evidenced by Code Llama. Having a system like GPT4 is still difficult to replicate. Getting it to run…

There are many MoE architectures and I suppose we don’t know for sure which OpenAI is using. The “selection” of the right mix of models is something that a network learns and it’s not a complex process. Certainly no more complex than training an LLM.

When I wrote “backend” was a poor choice of a word. “Meta-model” is probably a better choice of wording.

I hope it did not detract too much from the point of focusing on subtasks and modalities for FOSS as GPT 4 was built on a $163 million budget.

Finally, good point. We’ve got no idea of what OpenAI’s MoE approach is and how it works. I went back to Metas 2022 NLLB-200 system paper and they didn’t even publish the exact details of the router (gate).

Re: Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval

#86

Post by swyx »

brucethemoose2 wrote: post #64

One misleading thing is the notion that you need a 1-2B model to run on commodity hardware. This is not really true. Llama 7B runs with Vulkan/llama.cpp on ~8GB smartphones and ~12GB laptops. That ease is going to get much better over time, as lower RAM hardware starts dropping out of the market and the Vulkan implementations get more widespread. For users trying to run LLMs on 8GB or less machines, the AI Horde appr…

> the AI Horde approach of distributed models seems much more practical anyway.

i wasnt aware this was a term of art. is there a definitive blogpost or product explaining this approach?

Re: Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval

#87

Post by krak12 »

taywrobel wrote: post #19
_xnmw wrote: post #9

For the sake of not giving Microsoft and a few other tech giants immense power over the world, I really do hope the cost and efficiency of LLMs improve dramatically, until we can get GPT-4-equivalent models trained on a few graphics cards and running offline on an iPhone. Really rooting for these kinds of projects until someone makes the breakthrough.

You may be interested in what we’re working on at Symbolica AI. We’re using formal logic in the form of abstract rewrite systems over a causal graph to perform geometric deep learning. In theory it should be able to learn the same topological structure of data that neural networks do, but using entirely discrete operations and without the random walk inherent to stochastic gradient descent. Current experiments are re…

[dead]

Re: Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval

#88

Post by swyx »

kateklink wrote: post #83
holoduke wrote: post #76

Whats the difference between 1% and 99% of HumanEval? What does it tell really?

for pass@1 HumanEval tells how well the model solves a task from a set, given only one chance to solve it. It's not the perfect metric, there're other like DS-1000, MBPP (we have included them on HuggingFace model card). HumanEval is good for benchmarking with other models as it gives a fast idea how powerful the model is.

> given only one chance to solve it

my understanding is that there are 2 usages of the pass@{number} syntax. the HumanEval/Codex paper interprets the {number} as number of attempts[0]. however language modelers seem to use it to denote the number of few shot example demonstrations given in the context. these are starkly different and i wish the syntax wasnt overloaded

---

[0] https://arxiv.org/pdf/2107.03374.pdf

> Kulal et al. (2019) evaluate functional correctness using the pass@k metric, where k code samples are generated per problem, a problem is considered solved if any sample passes the unit tests, and the total fraction of problems solved is reported.

Re: Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval

#89

Post by swyx »

palmer_fox wrote: post #72

All these LLMs are pretty general if I understand correctly. Are there any efforts to create specialized models (other than for coding)? Or, what would be even better, "extract" certain areas from existing LLMs as a way to specialize them? With the goal to drastically reduce model size to be able to run on less powerful devices. E.g. a model specializing in chemistry doesn't need to include data on world's history or…

so, so many. there are RAG specific models (contextual ai), finance specific models (bloomberg gpt, brightwave), contact center models (cresta), even telco models (anthropic).

Re: Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval

#90

Post by palmer_fox »

swyx wrote: post #89
palmer_fox wrote: post #72

All these LLMs are pretty general if I understand correctly. Are there any efforts to create specialized models (other than for coding)? Or, what would be even better, "extract" certain areas from existing LLMs as a way to specialize them? With the goal to drastically reduce model size to be able to run on less powerful devices. E.g. a model specializing in chemistry doesn't need to include data on world's history or…

so, so many. there are RAG specific models (contextual ai), finance specific models (bloomberg gpt, brightwave), contact center models (cresta), even telco models (anthropic).

Very interesting. Thanks for replying!
Post reply on HN