Fine-tune your own Llama 2 to replace GPT-3.5/4 • Hacker News Board

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#171

Post by mr_o47 » Wed, Sep 13, 2023, 4:58 PM UTC

ronyfadel wrote: post #23

For translation jobs, I've experimented with Llama 2 70B (running on Replicate) v/s GPT-3.5; For about 1000 input tokens (and resulting 1000 output tokens), to my surprise, GPT-3.5 turbo was 100x cheaper than Llama 2. Llama 7B wasn't up to the task fyi, producing very poor translations. I believe that OpenAI priced GPT-3.5 aggressively cheap in order to make it a non-brainer to rely on them rather than relying on oth…

I’m actually replicate user. I have experimented with LLAMA2 on the replicate and I have similar experience

But you are totally correct about the pricing part it can get expensive

I’m running this photo service https://msdosimagetools.ngrok.dev/

Its doing 200+ photos every day and I’m using open source models behind the scene on replicate. My costs increasing day by day

Plus this is hosted locally

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#172

Post by Dowwie » Wed, Sep 13, 2023, 5:14 PM UTC

bfirsh wrote: post #102

kcorbitt wrote: post #28

Earlier quoted context omitted.

Yes, if you're just using Llama 2 off the shelf (without fine-tuning) I don't think there are a lot of workloads where it makes sense as a replacement for GPT-3.5. The one exception being for organizations where data security is non-negotiable and they really need to host on-prem. The calculus changes drastically though when you bring fine-tuning in, which lets a much smaller model outperform a larger one on many cla…

Founder of Replicate here. It's early indeed. OpenAI aren't doing anything magic. We're optimizing Llama inference at the moment and it looks like we'll be able to roughly match GPT 3.5's price for Llama 2 70B. Running a fine-tuned GPT-3.5 is surprisingly expensive. That's where using Llama makes a ton of sense. Once we’ve optimized inference, it’ll be much cheaper to run a fine-tuned Llama.

How heavy of a lift is it to optimize inference?

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#173

Post by Dowwie » Wed, Sep 13, 2023, 6:15 PM UTC

If I paid $20 to fine-tune a model to do X, and you paid $20 to fine-tune a model to do Y, is there a way to merge models, aggregating X and Y training, without training from scratch again?

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#174

Post by [deleted] » Wed, Sep 13, 2023, 7:28 PM UTC

[deleted]

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#175

Post by brucethemoose2 » Wed, Sep 13, 2023, 9:15 PM UTC

carom wrote: post #137

What are your thoughts on fine tuning vs low rank adaptations?

Llama LORAs are very customizable, and range from "almost full finetuning" to "barely taking more VRAM than GPTQ inference"

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#176

Post by mohitgangrade » Wed, Sep 13, 2023, 11:02 PM UTC

loganathanspr wrote: post #167

I am a little bit confused whether I need fine-tuning or RAG for my use case? My use case is this: I have some private data (say 1000 word documents), I want a QA capability on those 1000 documents. What is the best approach? Any help is appreciated.

Look at it like this:

- Fine tuning: Difficult, time-consuming, slow, takes time to add new information, costs a lot more.

- RAG: Can be free if you use free options like Chrome, Weaviate, or Postgres with Vector Plugin. Really fast. Once you set it up, you just need to upload a document, and it's available for GPT to answer with.

I'm using RAG for a client right now, and it was a breeze. Really easy, especially if you use something like Langchain. Compared to fine-tuning, it's a lot easier, cheaper, and faster...

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#177

Post by Mystery-Machine » Thu, Sep 14, 2023, 12:42 AM UTC

octacat wrote: post #75

ronyfadel wrote: post #23

For translation jobs, I've experimented with Llama 2 70B (running on Replicate) v/s GPT-3.5; For about 1000 input tokens (and resulting 1000 output tokens), to my surprise, GPT-3.5 turbo was 100x cheaper than Llama 2. Llama 7B wasn't up to the task fyi, producing very poor translations. I believe that OpenAI priced GPT-3.5 aggressively cheap in order to make it a non-brainer to rely on them rather than relying on oth…

Google Maps was also cheap. Initially. So it is aggressively cheap now, but would aggressively change later.

What?

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#178

Post by robertnishihara » Thu, Sep 14, 2023, 3:37 AM UTC

ronyfadel wrote: post #23

For translation jobs, I've experimented with Llama 2 70B (running on Replicate) v/s GPT-3.5; For about 1000 input tokens (and resulting 1000 output tokens), to my surprise, GPT-3.5 turbo was 100x cheaper than Llama 2. Llama 7B wasn't up to the task fyi, producing very poor translations. I believe that OpenAI priced GPT-3.5 aggressively cheap in order to make it a non-brainer to rely on them rather than relying on oth…

It shouldn't be 100x. We've built an LLM API at Anyscale, and the price comparison works out as follows (per million tokens)

- Llama-2-70B: $1 (on Anyscale Endpoints [1]) - GPT-3.5-turbo: $1.50 - $2 (OpenAI [2])

[1] https://app.endpoints.anyscale.com/ [2] https://openai.com/pricing

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#179

Post by mikewang » Thu, Sep 14, 2023, 6:07 AM UTC

kelseyfrog wrote: post #108

Arctic_fly wrote: post #30

Earlier quoted context omitted.

> Llama 7B wasn't up to the task fyi, producing very poor translations. From what I've read and personally experimented with, none of the Llama 2 models are well-suited to translation in particular (they were mainly trained on English data). Still, there are a number of tasks that they're really good at if fine-tuned correctly, such as classification and data extraction. > I believe that OpenAI priced GPT-3.5 aggress…

Is that because translation is typically an encoder-decoder task and llama is decoder only or is there something else about it that makes the last difficult for llama?

As I learned that 85% of its trainig data is English. Othere languanges composed of 15%.

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#180

Post by two_in_one » Thu, Sep 14, 2023, 6:34 AM UTC

famouswaffles wrote: post #93

halflings wrote: post #27

Earlier quoted context omitted.

I don't think translation is a great use case for ChatGPT and LLAMA. These models are overwhelmingly trained on English, and LLAMA2 which should have more data from other languages is still focused on languages w/ Latin/Cyrillic characters (so won't work well for Arabic, Hebrew, or CJK languages). You're better off using models specialized in translation; General purpose LLMs are more useful when fine-tuning on speci…

>You're better off using models specialized in translation For a couple dozen languages, GPT-4 is by far the best translator you can get your hand on so basically no.

What would be the best local standalone solution for translation model? Personal use, mostly self-education. 2 popular languages both ways (like en-spa, fr-ger). Free, pretrained off the github would be the best. I can try and train say 100M params LLM on 4090 RTX. But I'm not sure satisfactory result are achievable.