51–60 of 244 posts
Are there major advantages of GPT-3.5 Turbo tuning over PEFT/LoRA with Llama2?
Llama2 is still slow even with all the LLM inference tricks in the book and you need to pay for expensive GPUs to get it to a production-worthy latency, along with a scaling infra if there is a spike in usage.
Generating from a finetuned GPT 3.5 Turbo is 8x the cost of generating from the base model, so you really have to be in the “reduce prompt size by 90%” bucket they mention to get cost effectiveness out of it.
I'm out of this particular loop. What does 'reduce prompt size' mean in this case? What is the advantage?
Earlier quoted context omitted.
Fine tuning is not a replacement for retrieval augmentation. If you wanted to do that you'd be better off fine-tuning BERT to retrieve data more effectively (identify key points, smarter summarization, cross-encoder relevancy, etc)
Could you elaborate? It may not be a replacement, but I can think of a few use cases where RAG could be avoided entirely by having the model be fine tuned on the right dataset.
A model using RAG can tell you why it answered a certain way, and cite chapter and verse from the underlying data. That's impossible just from model weights.
Generating from a finetuned GPT 3.5 Turbo is 8x the cost of generating from the base model, so you really have to be in the “reduce prompt size by 90%” bucket they mention to get cost effectiveness out of it.
Generating from a finetuned GPT 3.5 Turbo is 8x the cost of generating from the base model, so you really have to be in the “reduce prompt size by 90%” bucket they mention to get cost effectiveness out of it.
I'm out of this particular loop. What does 'reduce prompt size' mean in this case? What is the advantage?
Generating from a finetuned GPT 3.5 Turbo is 8x the cost of generating from the base model, so you really have to be in the “reduce prompt size by 90%” bucket they mention to get cost effectiveness out of it.
I'm out of this particular loop. What does 'reduce prompt size' mean in this case? What is the advantage?
A short prompt - "Say hi" - costs less money than a long prompt: "Five cute names for a pet weasel".
You're charged for both the input and the output tokens: https://openai.com/pricing
So for a fine-tuned model that costs 8x more to be a purely financial win, it would need to let you reduce the size of your input and output prompts by 8x or more.
(I couldn't resist trying that pet weasel quote against Llama 2 13B running on my laptop with https://llm.datasette.io/ and I had to tiny-jailbreak-it because it didn't think weasels were appropriate pets: https://gist.github.com/simonw/8b935c2034fb69a9ecb52205ae342... )
Are there major advantages of GPT-3.5 Turbo tuning over PEFT/LoRA with Llama2?
Can anyone who's worked with fine-tuning models this large weigh in on how much fine-tuning data is typically required to be effective? Can 100k tokens (as mentioned in the docs) really influence the behavior of the base model that much, or was that just a toy example?
Earlier quoted context omitted.
Fine tuning is not a replacement for retrieval augmentation. If you wanted to do that you'd be better off fine-tuning BERT to retrieve data more effectively (identify key points, smarter summarization, cross-encoder relevancy, etc)
Could you elaborate? It may not be a replacement, but I can think of a few use cases where RAG could be avoided entirely by having the model be fine tuned on the right dataset.