Thanks for sharing this! I think you're working on something amazing. I will include your links in my newsletter, I think it will help a lot of folks: https://www.theprompt.io/
Fine-tune your own Llama 2 to replace GPT-3.5/4
191–194 of 194 posts
Re: Fine-tune your own Llama 2 to replace GPT-3.5/4
#192Re: Fine-tune your own Llama 2 to replace GPT-3.5/4
#193Earlier quoted context omitted.
You can run the smaller Llama variants on consumer grade hardware, but people typically rent GPUs from the cloud to run the larger variants. It is possible to run even larger variants on a beefy workstation or gaming rig, but the performance on consumer hardware usually makes this impractical. So the comparison would be the cost of renting a cloud GPU to run Llama vs querying ChatGPT.
>So the comparison would be the cost of renting a cloud GPU to run Llama vs querying ChatGPT. Yes, and it doesn't even come close. Llama2-70b can run inference at 300+tokens/s on a single V100 instance at ~$0.50/hr. Anyone who can should be switching away from OpenAI right now.
How do you fit Llama2-70b into V100? V100 is 16GB. Llama2-70b 4bit would require up to 40GB. Also, what do you use for inference to get 300+tokens/s?
Re: Fine-tune your own Llama 2 to replace GPT-3.5/4
#194Is Llama 2 currently the way to go for fine-tuning your own models? Are there other open-source LLMs worth considering?
The Huggingface Leaderboard is mostly dominated by Llama 2 variants: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb... It depends a lot on what you're trying to do. If have a focused use case of the type of fine-tuning you want, you can probably get away with one of the smaller models. Another thing to look out for is Retrieval Augmented Generation (RAG). I don't see it in wide use yet, but it may turn…
RAG is THE rage right now. Everybody is talking about it in enterprise world because they want to make all their legacy documents searchable.