Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fine-tune your own Llama 2 to replace GPT-3.5/4

news.ycombinator.com

Post reply on HN
Print view View original

51–60 of 194 posts

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#51

Post by brucethemoose2 »

3abiton wrote: post #44

Do you think this would end up facilitating the diffusion of finetuned LLMs ckpt models, just like stable diffusion? What's missing is web-UI?

There are already many hundreds of finetunes on huggingface, and many excellent UIs to run them in, like KoboldCPP and Text-gen-ui: https://huggingface.co/models?sort=modified&search=13B

There is even a crowdsourced version of the UI like artbot: https://lite.koboldai.net/#

And there are some excellent extant finetuning frameworks, like Aoxotol, that run on consumer GPUs: https://github.com/OpenAccess-AI-Collective/axolotl

IIRC Text-gen-ui had a QLORA finetuning UI too.

What I am saying is that its already like Stable Diffusion, but the community is just somewhat under the radar, and finetuning will never be quite as turnkey as dreambooth/sd 1.5 LORA due to the nature of the training data.

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#52

Post by AnonymousPlanet »

ronyfadel wrote: post #23

For translation jobs, I've experimented with Llama 2 70B (running on Replicate) v/s GPT-3.5; For about 1000 input tokens (and resulting 1000 output tokens), to my surprise, GPT-3.5 turbo was 100x cheaper than Llama 2. Llama 7B wasn't up to the task fyi, producing very poor translations. I believe that OpenAI priced GPT-3.5 aggressively cheap in order to make it a non-brainer to rely on them rather than relying on oth…

Cost isn't the only incentive not to use an LLM service that resides in a foreign country. Around here, there are industries for which it's pretty much a no-brainer to avoid anything that sends data across the atlantic.

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#53

Post by Arctic_fly »

idosh wrote: post #45

Can you elaborate on your plans for OpenPipe? Sounds like a very interesting project

Currently OpenPipe allows you to capture input/output from a powerful model and use it to fine-tune a much smaller one, then offers you the option to host through OpenPipe or download it and host it elsewhere. Models hosted on OpenPipe enjoy a few benefits, like data drift detection and automatic reformatting of output to match the original model you trained against (think extraction "function call" responses from a purely textual Llama 2 response) through the sdk.

Longer-term, we'd love to expand the selection of base models to include specialized LLMs that are particularly good at a certain task, e.g. language translation, and let you train off of those as well. Providing a ton of specialized starting models will decrease the amount of training data you need, and increase the number of tasks at which fine-tuned models can excel.

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#54

Post by brucethemoose2 »

ronyfadel wrote: post #23

For translation jobs, I've experimented with Llama 2 70B (running on Replicate) v/s GPT-3.5; For about 1000 input tokens (and resulting 1000 output tokens), to my surprise, GPT-3.5 turbo was 100x cheaper than Llama 2. Llama 7B wasn't up to the task fyi, producing very poor translations. I believe that OpenAI priced GPT-3.5 aggressively cheap in order to make it a non-brainer to rely on them rather than relying on oth…

TBH, Replicate is not a great way to run 7B beyond experimentation. You want a host with cheap consumer GPUs (like vast.ai) since the 4-bit requirements are so modest.

You either need a backend with good batching support (vLLM), or if you don't need much throughput, an extremely low end GPU or no GPU at all for exLlama/llama.cpp.

OpenAI benefits from quantization/batching, optimized kernels and very high utilization on their end, so the huge price gap vs a default HF Transformers instance is understandable. But even then, you are probably right about their aggressive pricing.

As for quality, you need a llama model finetunes on the target language (many already exist on Huggingface) and possibly custom grammar if your backend supports it.

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#55

Post by ramesh31 »

ronyfadel wrote: post #23

For translation jobs, I've experimented with Llama 2 70B (running on Replicate) v/s GPT-3.5; For about 1000 input tokens (and resulting 1000 output tokens), to my surprise, GPT-3.5 turbo was 100x cheaper than Llama 2. Llama 7B wasn't up to the task fyi, producing very poor translations. I believe that OpenAI priced GPT-3.5 aggressively cheap in order to make it a non-brainer to rely on them rather than relying on oth…

>For about 1000 input tokens (and resulting 1000 output tokens), to my surprise, GPT-3.5 turbo was 100x cheaper than Llama 2.

You'll never get actual economics out of switching to open models without running your own hardware. That's the whole point. There's orders of magnitude difference in price, where a single V100/3090 instance can run llama2-70b inference for ~0.50$/hr.

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#56

Post by ramesh31 »

loudmax wrote: post #41
MuffinFlavored wrote: post #35

Earlier quoted context omitted.

I thought Llama was opensource/free and you could run it yourself?

You can run the smaller Llama variants on consumer grade hardware, but people typically rent GPUs from the cloud to run the larger variants. It is possible to run even larger variants on a beefy workstation or gaming rig, but the performance on consumer hardware usually makes this impractical. So the comparison would be the cost of renting a cloud GPU to run Llama vs querying ChatGPT.

>So the comparison would be the cost of renting a cloud GPU to run Llama vs querying ChatGPT.

Yes, and it doesn't even come close. Llama2-70b can run inference at 300+tokens/s on a single V100 instance at ~$0.50/hr. Anyone who can should be switching away from OpenAI right now.

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#58

Post by kcorbitt »

rrherr wrote: post #49

"You do this by training an existing model on example input/output pairs that demonstrate the task you want your fine-tuned model to learn." Are fine-tuning datasets required to be input/output pairs? Or instead, can the fine-tuning be autoregressive (predict the next token throughout this corpus of unlabeled documents)?

There's no rule that your fine-tuning dataset needs to be split into input/output pairs -- you can of course fine-tune a model to just continue a sequence.

As a practical matter though, most of the fine-tuning frameworks, including Axolotl (which this guide uses) and HuggingFace's SFTTrainer (the actual fine-tuning trainer most frameworks use under the hood) assume your data comes in input/output pairs, and automatically inserts a separator token to let the model know that the input has finished and it should start generating the output. In general most tasks can be formulated this way, including autocomplete tasks, so I'd probably recommend going that way unless you have a very strong reason not to.

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#59

Post by gsuuon »

7speter wrote: post #22
moonchrome wrote: post #8

Earlier quoted context omitted.

We are talking about 7B models ? Those can run on consumer GPUs with lower latency than A100s AFAIK (because gaming GPUs are clocked different). Not to mention OpenAI has shit latency and terrible reliability - you should be using Azure models if you care about that - but pricing is also higher. I would say fixed costs and development time is on openai side but I've seen people post great practical comparisons for la…

When you say it can run on consumer gpus, do you mean pretty much just the 4090/3090 or can it run on lesser cards?

Quantized 7B's can comfortably run with 8GB vram

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#60

Post by kcorbitt »

divbzero wrote: post #57

Is Llama 2 currently the way to go for fine-tuning your own models? Are there other open-source LLMs worth considering?

Depends on your use case. If you're doing pure classification then there are smaller encoder-only models like DeBERTa that might get you better performance with a much smaller model size (so cheaper inference).

But if you need text generation and are ok with a 7B+ parameter model, Llama 2 or one of its derivatives is what I'd strongly recommend. The community around it is much larger than any of the alternatives so the tooling is better, and it's either state of the art or close to it on all evals when compared to other similarly-sized open models.

If you're comfortable sharing more details of the task you're trying to do I might be able to give more specific advice.

Post reply on HN