Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fine-tune your own Llama 2 to replace GPT-3.5/4

news.ycombinator.com

Post reply on HN
Print view View original

41–50 of 194 posts

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#41

Post by loudmax »

MuffinFlavored wrote: post #35
ronyfadel wrote: post #23

For translation jobs, I've experimented with Llama 2 70B (running on Replicate) v/s GPT-3.5; For about 1000 input tokens (and resulting 1000 output tokens), to my surprise, GPT-3.5 turbo was 100x cheaper than Llama 2. Llama 7B wasn't up to the task fyi, producing very poor translations. I believe that OpenAI priced GPT-3.5 aggressively cheap in order to make it a non-brainer to rely on them rather than relying on oth…

I thought Llama was opensource/free and you could run it yourself?

You can run the smaller Llama variants on consumer grade hardware, but people typically rent GPUs from the cloud to run the larger variants. It is possible to run even larger variants on a beefy workstation or gaming rig, but the performance on consumer hardware usually makes this impractical.

So the comparison would be the cost of renting a cloud GPU to run Llama vs querying ChatGPT.

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#42

Post by kcorbitt »

Maschinesky wrote: post #32

What makes sense to fine-tune and what not? You said 50-1000 examples. Do I fine-tune when having specific q/a sets like from real customers and I want to add the right answer to the model? Do I fine-tune facts or should I use some lookup? Does adding some code and API docs for a current version of something I want more support make sense? Like chatgpt knows quarkus 2 but not quarkus 3

> What makes sense to fine-tune and what not?

In general, fine-tuning helps a model figure out how to do the exact task that is being done in the examples it's given. So fine-tuning it on 1000 examples of an API being used in the wild is likely to teach it to use that API really effectively, but fine-tuning it on just the API docs probably won't.

That said, there are a lot of interesting ideas floating around on how to most effectively teach a model purely from instructions like API docs. Powerful models like GPT-4 can figure it out from in-context learning (ie. if you paste in a page of API docs and ask GPT-4 to write something with the API it can usually do a decent job). I suspect the community will figure out techniques either through new training objectives or synthetic training data to do it for smaller fine-tuned models as well.

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#43

Post by Arctic_fly »

Maschinesky wrote: post #32

What makes sense to fine-tune and what not? You said 50-1000 examples. Do I fine-tune when having specific q/a sets like from real customers and I want to add the right answer to the model? Do I fine-tune facts or should I use some lookup? Does adding some code and API docs for a current version of something I want more support make sense? Like chatgpt knows quarkus 2 but not quarkus 3

Generally speaking, fine-tuning a small model makes sense when the task that you want it to carry out is well-defined and doesn't vary too much from one prompt to another. Fine-tuning facts into a model doesn't seem to scale super well, but general textual style, output format, and evaluation criteria for example can all be instilled through the fine-tuning process. I would use lookup if you need your answers to include a wide array of information that the model you're basing off of wasn't initially trained on.

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#47

Post by Arctic_fly »

binarymax wrote: post #36

This looks awesome! Tangential question - do you find GPT function calling to work consistently and without error, or do you get errors when using it? By errors I mostly mean incorrect function signatures/types or missing values...but if you see other unpredictable behavior that would help too.

I haven't had much trouble with GPT 3.5 or 4 function calls returning in an undesirable format recently. I did get a few bad syntax responses when OpenAI first rolled it out, but not for the past few months.

Llama 2 can also pick the function call format up, given sufficient training data that contains function call responses, though you'll then have to parse the returned object out of the text-based response.

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#48

Post by NavinF »

tikkun wrote: post #3

Looks really well executed, nice! I'd shared this idea with a few people. GPT and other LLMs don't allow you to use their output to train competing models, but the implication is that it's fine to use their output to train your own internal alternative models. So you can't sell access to the output as an API, but you can use it to replace your GPT API calls. My other thoughts to extend this are that you could make it…

>GPT and other LLMs don't allow you to use their output to train competing models

ToS is unenforceable and irrelevant to anyone that's in this space

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#49

Post by rrherr »

"You do this by training an existing model on example input/output pairs that demonstrate the task you want your fine-tuned model to learn."

Are fine-tuning datasets required to be input/output pairs? Or instead, can the fine-tuning be autoregressive (predict the next token throughout this corpus of unlabeled documents)?

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#50

Post by daniels11 »

halflings wrote: post #27
ronyfadel wrote: post #23

For translation jobs, I've experimented with Llama 2 70B (running on Replicate) v/s GPT-3.5; For about 1000 input tokens (and resulting 1000 output tokens), to my surprise, GPT-3.5 turbo was 100x cheaper than Llama 2. Llama 7B wasn't up to the task fyi, producing very poor translations. I believe that OpenAI priced GPT-3.5 aggressively cheap in order to make it a non-brainer to rely on them rather than relying on oth…

I don't think translation is a great use case for ChatGPT and LLAMA. These models are overwhelmingly trained on English, and LLAMA2 which should have more data from other languages is still focused on languages w/ Latin/Cyrillic characters (so won't work well for Arabic, Hebrew, or CJK languages). You're better off using models specialized in translation; General purpose LLMs are more useful when fine-tuning on speci…

what models would you use for translation? I am working on a language learning tutor (trytutor.app, very early) and GPT-3.5 turbo has been working fine, for the most part.

For foreign language corrections ("correct this German sentence and give a reason for the correction"), GPT-3.5 doesn't quite have the horsepower so I use GPT-4

Post reply on HN