Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval

refact.ai

Post reply on HN
Print view View original

91–100 of 111 posts

Re: Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval

#91

Post by ukuina »

swyx wrote: post #86
brucethemoose2 wrote: post #64

One misleading thing is the notion that you need a 1-2B model to run on commodity hardware. This is not really true. Llama 7B runs with Vulkan/llama.cpp on ~8GB smartphones and ~12GB laptops. That ease is going to get much better over time, as lower RAM hardware starts dropping out of the market and the Vulkan implementations get more widespread. For users trying to run LLMs on 8GB or less machines, the AI Horde appr…

> the AI Horde approach of distributed models seems much more practical anyway. i wasnt aware this was a term of art. is there a definitive blogpost or product explaining this approach?

This is a reference to Kobold Horde, a distributed volunteer network of GPUs that can be inferenced upon.

Re: Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval

#92

Post by lucubratory »

palmer_fox wrote: post #80
hnhg wrote: post #74

Earlier quoted context omitted.

I am not an expert but it still has to learn human language/grammar/whathaveyou, and that is where scale seems to matter. Fine-tuning on a subset of knowledge after that is typically how the domain-specialisation is achieved, by my understanding.

I was wondering about that too. Would it be possible in the future to have a more modular approach to LLMs? Have a module that is responsible for basic knowledge/language/grammar and then other more specialized modules that are added selectively. I don't know enough about fine-tuning, not sure if the process is capable of removing "unused" parts of the model (I guess not possible, similar to un-learning).

There are various methods for removing unused parts of the model, like distillation. The idea is generally that you always lose performance, but hopefully you lose more size/runcost than you do performance, proportionately.

Re: Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval

#93

Post by nacs »

brucethemoose2 wrote: post #64

One misleading thing is the notion that you need a 1-2B model to run on commodity hardware. This is not really true. Llama 7B runs with Vulkan/llama.cpp on ~8GB smartphones and ~12GB laptops. That ease is going to get much better over time, as lower RAM hardware starts dropping out of the market and the Vulkan implementations get more widespread. For users trying to run LLMs on 8GB or less machines, the AI Horde appr…

Yes, 7B is perfectly usable on low-end hardware if you're using it for instruction tuning/chat.

But for code completion in an IDE where it has to react as you type, every 100 millisecond delay in response time is noticable.

Even with a 24GB GPU, a 7B model doesn't feel snappy enough for code-completion in an IDE.

Re: Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval

#94

Post by viraptor »

palmer_fox wrote: post #81
charcircuit wrote: post #78

Earlier quoted context omitted.

Domain specialization is done by continuing the full training process. Fine tuning is more for changing the style of the output than adding new knowledge.

What if the initial training already contains all necessary data for a particular specialization? What would be the benefit of continuing the training process?

Imagine someone tells you about how someone committed a crime and asks you to summarise. Now imagine the same question is asked to a lawyer. Even if you both knew the same facts, the response would be very different in style, highlighted points, mentioned references, etc. The domain specific fine tuning does exactly that. Sure, sometimes you can get very close by changing the prompt to include "respond like a lawyer in situation X with following extra rules", but not always and the fine-tuning gives better results and shorter prompt.

Re: Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval

#95

Post by smcleod »

brucethemoose2 wrote: post #64

One misleading thing is the notion that you need a 1-2B model to run on commodity hardware. This is not really true. Llama 7B runs with Vulkan/llama.cpp on ~8GB smartphones and ~12GB laptops. That ease is going to get much better over time, as lower RAM hardware starts dropping out of the market and the Vulkan implementations get more widespread. For users trying to run LLMs on 8GB or less machines, the AI Horde appr…

Yeah but I remember thinking to myself every few years that surely next year will be the year that base model machines start at 32/64/… GB - but alas, it’s nearly the end of 2023 and your average computer still seems stuck on a measly 16GB! I don’t think average RAM size on consumer machines has increased at all in the last 8~ years or so.

Re: Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval

#96

Post by brucethemoose2 »

ukuina wrote: post #91
swyx wrote: post #86

Earlier quoted context omitted.

> the AI Horde approach of distributed models seems much more practical anyway. i wasnt aware this was a term of art. is there a definitive blogpost or product explaining this approach?

This is a reference to Kobold Horde, a distributed volunteer network of GPUs that can be inferenced upon.

^

I didn't mean to imply splitting llama up between machines (though that is a thing with llama.cpp), but a pool of clients and servers who make requests and process them:

https://lite.koboldai.net/

A few users with half decent PCs can serve a much larger group of people, and the "lesser" hosts can host smaller models to "earn" access to larger ones.

Re: Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval

#97

Post by Retric »

smcleod wrote: post #95
brucethemoose2 wrote: post #64

One misleading thing is the notion that you need a 1-2B model to run on commodity hardware. This is not really true. Llama 7B runs with Vulkan/llama.cpp on ~8GB smartphones and ~12GB laptops. That ease is going to get much better over time, as lower RAM hardware starts dropping out of the market and the Vulkan implementations get more widespread. For users trying to run LLMs on 8GB or less machines, the AI Horde appr…

Yeah but I remember thinking to myself every few years that surely next year will be the year that base model machines start at 32/64/… GB - but alas, it’s nearly the end of 2023 and your average computer still seems stuck on a measly 16GB! I don’t think average RAM size on consumer machines has increased at all in the last 8~ years or so.

It actually kind of makes sense.

RAM is only about 6x the speed of SSD’s for sequential access. Most people don’t actually need truly random access to all that much data rather than streaming video or loading video game assets to their GPU. So they shift spending to other components like video card, monitors, etc that actually provide significant value.

Which is how you get people with 16 GB of system RAM using graphics cards that also have 16GB of RAM.

Re: Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval

#98

Post by jmorgan »

brucethemoose2 wrote: post #64

One misleading thing is the notion that you need a 1-2B model to run on commodity hardware. This is not really true. Llama 7B runs with Vulkan/llama.cpp on ~8GB smartphones and ~12GB laptops. That ease is going to get much better over time, as lower RAM hardware starts dropping out of the market and the Vulkan implementations get more widespread. For users trying to run LLMs on 8GB or less machines, the AI Horde appr…

This is true! Although I'm also really excited at the potential speed (both for loading the model and token generation) of a 1B model for things like code completion.

Re: Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval

#99

Post by MichaelBurge »

qeternity wrote: post #62
yunwal wrote: post #42

Earlier quoted context omitted.

GPT4 is expensive to run, even more expensive to finetune, and for all practical purposes can’t be run offline (because the model is too big to run outside of a huge data center). Evaluation latency is also an issue for many usecases, and you have to share your query with openai, so you can’t run sensitive queries. The output is also controlled/censored by OpenAI. Here’s a few usecases that I wouldn’t want to use Ope…

> GPT4 is expensive to run, even more expensive to finetune GPT4 can't even be finetuned at the moment (though I expect that to change).

It can be finetuned. Bing is a finetuned GPT-4.

Re: Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval

#100

Post by brucethemoose2 »

nacs wrote: post #93
brucethemoose2 wrote: post #64

One misleading thing is the notion that you need a 1-2B model to run on commodity hardware. This is not really true. Llama 7B runs with Vulkan/llama.cpp on ~8GB smartphones and ~12GB laptops. That ease is going to get much better over time, as lower RAM hardware starts dropping out of the market and the Vulkan implementations get more widespread. For users trying to run LLMs on 8GB or less machines, the AI Horde appr…

Yes, 7B is perfectly usable on low-end hardware if you're using it for instruction tuning/chat. But for code completion in an IDE where it has to react as you type, every 100 millisecond delay in response time is noticable. Even with a 24GB GPU, a 7B model doesn't feel snappy enough for code-completion in an IDE.

This can be addressed with token streaming and input caching.

Would that be enough? shrug

Post reply on HN