Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval

refact.ai

Post reply on HN
Print view View original

41–50 of 111 posts

Re: Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval

#41

Post by pawelduda »

taywrobel wrote: post #19
_xnmw wrote: post #9

For the sake of not giving Microsoft and a few other tech giants immense power over the world, I really do hope the cost and efficiency of LLMs improve dramatically, until we can get GPT-4-equivalent models trained on a few graphics cards and running offline on an iPhone. Really rooting for these kinds of projects until someone makes the breakthrough.

You may be interested in what we’re working on at Symbolica AI. We’re using formal logic in the form of abstract rewrite systems over a causal graph to perform geometric deep learning. In theory it should be able to learn the same topological structure of data that neural networks do, but using entirely discrete operations and without the random walk inherent to stochastic gradient descent. Current experiments are re…

Sounds cool, but what are the drawbacks?

Re: Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval

#42

Post by yunwal »

mholubowski wrote: post #27

Hey, I have a genuine question: What is the point of a new model that isn’t better than the best possible model (example: OpenAI GPT-4)? What’s the point in having a smaller model? Who cares? —- This is a real, genuine question that I don’t have a clear answer to. Excuse my ignorance, plz enlighten your boi.

GPT4 is expensive to run, even more expensive to finetune, and for all practical purposes can’t be run offline (because the model is too big to run outside of a huge data center). Evaluation latency is also an issue for many usecases, and you have to share your query with openai, so you can’t run sensitive queries. The output is also controlled/censored by OpenAI.

Here’s a few usecases that I wouldn’t want to use OpenAI/GPT for

- Advanced autocomplete for texting and private communications

- Querying sensitive document databases like emails

- Traveling in low connectivity areas

- Politically incorrect usecases (generating erotic content for example)

List kinda goes on and on

Re: Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval

#43

Post by notsylver »

mholubowski wrote: post #27

Hey, I have a genuine question: What is the point of a new model that isn’t better than the best possible model (example: OpenAI GPT-4)? What’s the point in having a smaller model? Who cares? —- This is a real, genuine question that I don’t have a clear answer to. Excuse my ignorance, plz enlighten your boi.

IMO, the main reasons are (but are definitely not limited to):

- You can fine tune these models for very specific tasks, which GPT-4 might not be as good at.

- Open source models are free. You can use them as much as you want without worrying about a $xx,xxx bill at the end of the month which makes tinkering with them easier.

- Smaller models like this can run on consumer hardware, even phones, and can run offline.

- Privacy and not having to abide by a third parties terms. You don't have to deal with "As a large language model...", especially with uncensored models.

- Tools like jsonformer https://github.com/1rgs/jsonformer are not possible with OpenAIs API.

- It's also just really cool, let's be honest.

Re: Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval

#44

Post by paulsutter »

taywrobel wrote: post #19
_xnmw wrote: post #9

For the sake of not giving Microsoft and a few other tech giants immense power over the world, I really do hope the cost and efficiency of LLMs improve dramatically, until we can get GPT-4-equivalent models trained on a few graphics cards and running offline on an iPhone. Really rooting for these kinds of projects until someone makes the breakthrough.

You may be interested in what we’re working on at Symbolica AI. We’re using formal logic in the form of abstract rewrite systems over a causal graph to perform geometric deep learning. In theory it should be able to learn the same topological structure of data that neural networks do, but using entirely discrete operations and without the random walk inherent to stochastic gradient descent. Current experiments are re…

Especially interested in learning directly on geometries, please keep us updated and share results

Re: Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval

#45

Post by KRAKRISMOTT »

taywrobel wrote: post #19
_xnmw wrote: post #9

For the sake of not giving Microsoft and a few other tech giants immense power over the world, I really do hope the cost and efficiency of LLMs improve dramatically, until we can get GPT-4-equivalent models trained on a few graphics cards and running offline on an iPhone. Really rooting for these kinds of projects until someone makes the breakthrough.

You may be interested in what we’re working on at Symbolica AI. We’re using formal logic in the form of abstract rewrite systems over a causal graph to perform geometric deep learning. In theory it should be able to learn the same topological structure of data that neural networks do, but using entirely discrete operations and without the random walk inherent to stochastic gradient descent. Current experiments are re…

> We’re using formal logic in the form of abstract rewrite systems over a causal graph to perform geometric deep learning. In theory it should be able to learn the same topological structure of data that neural networks do, but using entirely discrete operations and without the random walk inherent to stochastic gradient descent.

Abstract rewrite like a computer algebra system's (e.g. Wolfram) term rewriting equation simplication method?

Re: Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval

#46

Post by flangola7 »

_xnmw wrote: post #9

For the sake of not giving Microsoft and a few other tech giants immense power over the world, I really do hope the cost and efficiency of LLMs improve dramatically, until we can get GPT-4-equivalent models trained on a few graphics cards and running offline on an iPhone. Really rooting for these kinds of projects until someone makes the breakthrough.

I don't, how do you maintain control and prevent mass harm in that case? I don't see anyway out other than similar gatekeeping we apply to ownership and use of high explosives and radiological weapon tooling.

At all other times I support tech freedom. I use libre software, I use Tor, I donate to privacy and FOSS organizations constantly. I only write my software projects under an AGPL license. AI is qualitatively different. A world run amok with intelligent infinite Sybils is not good for anyone. I hope massive compute continues to be necessary, it may be the only hard chokepoint we have to keep a handle on the beast.

Re: Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval

#47

Post by tiborsaas »

mholubowski wrote: post #27

Hey, I have a genuine question: What is the point of a new model that isn’t better than the best possible model (example: OpenAI GPT-4)? What’s the point in having a smaller model? Who cares? —- This is a real, genuine question that I don’t have a clear answer to. Excuse my ignorance, plz enlighten your boi.

Your questions sounds like why do we need Alpine linux when we have Ubuntu? Why do we have SQLite when we have Postgres?

I think the point is to reach a baseline of something being super lightweight yet still useful that could be production for a number of use cases.

Re: Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval

#48

Post by riku_iki »

mityamitya wrote: post #39
riku_iki wrote: post #17

Earlier quoted context omitted.

> almost reaches the same HumanEval how can you tell that HumanEval is not leaked to your training data in some form?

Hi! We ran LSH filtering over datasets to remove all code that can be similar to HumanEval samples.

so, we have to trust your procedure..

Re: Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval

#49

Post by breadsniffer01 »

ldjkfkdsjnv wrote: post #15

I dont trust any benchmarks for any LLM thats not coming from FB, Google, OpenAI, Anthropic, or Microsoft. These models are so dynamic, the simple benchmark numbers never tell the whole story of the quality of the model. Take for instance, a recent posting by anyscale, claiming their fine tuning of Llama 2 was competitive with OpenAI's model. The reality being their fined tuned model is basically worthless, and was c…

They could have easily benchmarked with the Spider SQL test set but they didn’t.

I have a feeling that the more robust models might be the ones that don’t perform best on benchmarks right away.

Re: Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval

#50

Post by seydor »

mholubowski wrote: post #27

Hey, I have a genuine question: What is the point of a new model that isn’t better than the best possible model (example: OpenAI GPT-4)? What’s the point in having a smaller model? Who cares? —- This is a real, genuine question that I don’t have a clear answer to. Excuse my ignorance, plz enlighten your boi.

Imagine being on Mars and running on a small PV panel and needing to code a bugfix in your oxygen supply system through the wire with Microsoft Earth(tm) or something
Post reply on HN