101–110 of 111 posts
Earlier quoted context omitted.
> GPT4 is expensive to run, even more expensive to finetune GPT4 can't even be finetuned at the moment (though I expect that to change).
It can be finetuned. Bing is a finetuned GPT-4.
One misleading thing is the notion that you need a 1-2B model to run on commodity hardware. This is not really true. Llama 7B runs with Vulkan/llama.cpp on ~8GB smartphones and ~12GB laptops. That ease is going to get much better over time, as lower RAM hardware starts dropping out of the market and the Vulkan implementations get more widespread. For users trying to run LLMs on 8GB or less machines, the AI Horde appr…
Yes, 7B is perfectly usable on low-end hardware if you're using it for instruction tuning/chat. But for code completion in an IDE where it has to react as you type, every 100 millisecond delay in response time is noticable. Even with a 24GB GPU, a 7B model doesn't feel snappy enough for code-completion in an IDE.
Earlier quoted context omitted.
Perhaps the wrong thread to ask this question... Is it not possible to load a model on something like an NVMe M.2 drive instead of RAM? It's slower of course, but only 5-10x if I understand correctly.
Yes but they’re slow enough on normal hardware for that 5-10x to be painful…
The web interface for the LLM server is especially nice and clean compared to many of the others I've tried - and it "just works". Very interested to see how this evolves.
This post is misleading, in a way that is hard to do accidentally. - They compare the performance of this model to the worst 7B code llama model. The base code llama 7B python model scores 38.4% on humaneval, versus the non-python model, which only scores 33%. - They compare their instruct tuned model to non-instruct-tuned models. Instruction tuning can add 20% or more to humaneval performance. For example, WizardLM…
> They compare the performance of this model to the worst 7B code llama model. The base code llama 7B python model scores 38.4% on humaneval, versus the non-python model, which only scores 33%.
We are comparing multilingual models, and we are not focused on python-finetuned versions
> They compare their instruct tuned model to non-instruct-tuned models. Instruction tuning can add 20% or more to humaneval performance. For example, WizardLM 7B scores 55% on humaneval [1], and I've trained a 7B model that scores 62% [2]. > For another example of instruction tuning, Stablecode instruct tuned benchmarks at 26%, not the 20% they cite for the base model [3]
We have two separate comparisons (see https://huggingface.co/smallcloudai/Refact-1_6B-fim) for completion-based models and instruction-following-based models with different humaneval formats. But we are considering our model as a completion (FIM) one in the first place and we were using 85% non-instruction following data to make the final model. The chat functionality is really limited for such small models
> Starcoder, when prompted properly, scores 40% on humaneval
Yep, that is right. But worth mentioning, the starcoder model showed 40% while being extra finetuned exclusively on python
> They do not report their base model performance (as far as I can tell)
Our base model gets around 20-23% humaneval. But it is not the case since the model was trained using 50% non-code data (considering the model's size it was really hard to keep the model converging)
Earlier quoted context omitted.
Hi! We ran LSH filtering over datasets to remove all code that can be similar to HumanEval samples.
so, we have to trust your procedure..
There are a bunch of other benchmarks too, check out the page https://huggingface.co/smallcloudai/Refact-1_6B-fim
Also, feel free to run any new benchmarks
Earlier quoted context omitted.
Yes, 7B is perfectly usable on low-end hardware if you're using it for instruction tuning/chat. But for code completion in an IDE where it has to react as you type, every 100 millisecond delay in response time is noticable. Even with a 24GB GPU, a 7B model doesn't feel snappy enough for code-completion in an IDE.
GPU RAM quantity isn’t typically correlated to inference rate. Precision/quantization levels do affect model size, which will affect inference rate. However, I would expect a smaller model to be faster (less RAM).
The smaller the model, the less has to be read from ram for every single token.
Batching mixes up this calculus a bit.
Earlier quoted context omitted.
> GPT4 is expensive to run, even more expensive to finetune GPT4 can't even be finetuned at the moment (though I expect that to change).
It can be finetuned. Bing is a finetuned GPT-4.
Earlier quoted context omitted.
There are many MoE architectures and I suppose we don’t know for sure which OpenAI is using. The “selection” of the right mix of models is something that a network learns and it’s not a complex process. Certainly no more complex than training an LLM.
When I wrote “backend” was a poor choice of a word. “Meta-model” is probably a better choice of wording. I hope it did not detract too much from the point of focusing on subtasks and modalities for FOSS as GPT 4 was built on a $163 million budget. Finally, good point. We’ve got no idea of what OpenAI’s MoE approach is and how it works. I went back to Metas 2022 NLLB-200 system paper and they didn’t even publish the e…