Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fine-Tuning Llama-2: A Comprehensive Case Study for Tailoring Custom Models

anyscale.com

Post reply on HN
Print view View original

51–60 of 61 posts

Re: Fine-Tuning Llama-2: A Comprehensive Case Study for Tailoring Custom Models

#51

Post by zora_goron »

purplecats wrote: post #49
jawerty wrote: post #3

Just to add to this, I run through a lot of these topics around fine-tuning Llama 2 on your own dataset (for me it's my own code :P) in a coding live stream a couple weeks ago. All on Colab single GPU Fine-tuning Llama stream: https://www.youtube.com/watch?v=TYgtG2Th6fI&t=2282s I have a couple more one where I do a QLoRa fine tuning session and explain the concepts as a personally self taught engineer (software engin…

really need a simple "put your source stuff in this directory, then press this button, then chat with your contents" type app/module/library. too much implementation detail required make it inaccessible for any non-significant use case. i imagine privateGpt will get there slowly

I wrote a simple implementation to do this in ChatGPT via local plugin [0]. Obviously it doesn’t hit the “fully private” requirement but I imagine it would be relatively straightforward to integrate into a local LLM. The question is whether a local LLM would be as good at grabbing enough context and nuance from the project to answer meaningfully as GPT-4 is able to do with plugins.

[0] https://github.com/samrawal/chatgpt-localfiles

Re: Fine-Tuning Llama-2: A Comprehensive Case Study for Tailoring Custom Models

#52

Post by jawerty »

purplecats wrote: post #49
jawerty wrote: post #3

Just to add to this, I run through a lot of these topics around fine-tuning Llama 2 on your own dataset (for me it's my own code :P) in a coding live stream a couple weeks ago. All on Colab single GPU Fine-tuning Llama stream: https://www.youtube.com/watch?v=TYgtG2Th6fI&t=2282s I have a couple more one where I do a QLoRa fine tuning session and explain the concepts as a personally self taught engineer (software engin…

really need a simple "put your source stuff in this directory, then press this button, then chat with your contents" type app/module/library. too much implementation detail required make it inaccessible for any non-significant use case. i imagine privateGpt will get there slowly

One of my streams I essentially build this from scratch https://www.youtube.com/watch?v=kBB1A2ot-Bw&t=236s. A retriever reader model, let me know if you want the code I think I like the colab in the comments but let me know if you need more.

Re: Fine-Tuning Llama-2: A Comprehensive Case Study for Tailoring Custom Models

#53

Post by aldarisbm »

praveenhm wrote: post #50

Is this possible to fine tune llama-2 locally on M1 Ultra 64GB, I would like to know or any pointer would be good. Most of them are on Cloud or using Nvidia Cuda on linux.

I don't think so. I have M1 Max 64GB and it works okay for some inference. I'm buying a few credits from RunPod. It will be a few 10's of dollars to get it trained.

Re: Fine-Tuning Llama-2: A Comprehensive Case Study for Tailoring Custom Models

#54

Post by whoevercares »

purplecats wrote: post #49
jawerty wrote: post #3

Just to add to this, I run through a lot of these topics around fine-tuning Llama 2 on your own dataset (for me it's my own code :P) in a coding live stream a couple weeks ago. All on Colab single GPU Fine-tuning Llama stream: https://www.youtube.com/watch?v=TYgtG2Th6fI&t=2282s I have a couple more one where I do a QLoRa fine tuning session and explain the concepts as a personally self taught engineer (software engin…

really need a simple "put your source stuff in this directory, then press this button, then chat with your contents" type app/module/library. too much implementation detail required make it inaccessible for any non-significant use case. i imagine privateGpt will get there slowly

At this stage of the AI, The implementation details matters a lot for the chat to be actually meaningful… RAG is over-hyped

Re: Fine-Tuning Llama-2: A Comprehensive Case Study for Tailoring Custom Models

#55

Post by jauntbox »

behnamoh wrote: post #17

> Additionally, while this wasn’t an issue for GPT, the Llama chat models would often output hundreds of miscellaneous tokens that were unnecessary for the task, further slowing down their inference time (e.g. “Sure! Happy to help…”). That's the problem I've been facing with Llama 2 as well. It's almost impossible to have it just output the desired text. It will always add something before and after its response. Doe…

It depends on what your goal is, but I've had success reproducing specific output formatting by fine-tuning the base LLaMA2 models instead of the RLHF'd models. My use cases were simpler - information extraction/synthesis from text rather than creative writing. The base models might not be good fits for your task.

Re: Fine-Tuning Llama-2: A Comprehensive Case Study for Tailoring Custom Models

#56

Post by zhz_ray »

Disclaimer: I work for Anyscale

This blog seems to got good attention :) So we definitely plan to add it to Ray Summit https://raysummit.anyscale.com/agenda

Please comment on this thread if you have ideas of what kind of content you want to see more at Ray Summit

Re: Fine-Tuning Llama-2: A Comprehensive Case Study for Tailoring Custom Models

#57

Post by sandGorgon »

jawerty wrote: post #46
sandGorgon wrote: post #39

Earlier quoted context omitted.

this is brilliant. could you do a series about how to prepare custom data sets for finetuning. thats the part that a lot of other tutorials skip on. Especially for different goals - like safety, accuracy, etc.

Of course I have a few where I web scrape and build a dataset for myself with prefix tokens. I can break that down more on a specific stream about it.

well not so much as the raw data acquisition (scraping and stuff), but really data prep for finetuning. I'm hearing that each model needs it in a different format - chat finetuning data is different from instruct, etc etc

Re: Fine-Tuning Llama-2: A Comprehensive Case Study for Tailoring Custom Models

#58

Post by zby »

sandGorgon wrote: post #41
zby wrote: post #36

Earlier quoted context omitted.

There is an article at the original site about that: https://www.anyscale.com/blog/fine-tuning-is-for-form-not-fa... Everybody new to this field thinks that he needs finetuning to teach the LLM of new facts. I made the same mistake initially, later I published a slightly ranty post on that: https://zzbbyy.substack.com/p/why-you-need-rag-not-finetunin...

Quick question - Gorilla paper talks about finetuning for RAG. Do you see this in practice ? can you do finetuning that specifically affects RAG ?

Sorry - myself I don't have much experience yet, I am at the research phase, but from what I read it makes sense to finetune the model to better understand the format used for calling the external tools including a search engine.

Re: Fine-Tuning Llama-2: A Comprehensive Case Study for Tailoring Custom Models

#59

Post by screye »

behnamoh wrote: post #17

> Additionally, while this wasn’t an issue for GPT, the Llama chat models would often output hundreds of miscellaneous tokens that were unnecessary for the task, further slowing down their inference time (e.g. “Sure! Happy to help…”). That's the problem I've been facing with Llama 2 as well. It's almost impossible to have it just output the desired text. It will always add something before and after its response. Doe…

Prompt the model to always output answers / code within ```content``` strings or json. If it's json, then you can identify where it starts and ends. Strip everything outside the json.

Re: Fine-Tuning Llama-2: A Comprehensive Case Study for Tailoring Custom Models

#60

Post by solaxun »

kouroshh wrote: post #21
behnamoh wrote: post #17

> Additionally, while this wasn’t an issue for GPT, the Llama chat models would often output hundreds of miscellaneous tokens that were unnecessary for the task, further slowing down their inference time (e.g. “Sure! Happy to help…”). That's the problem I've been facing with Llama 2 as well. It's almost impossible to have it just output the desired text. It will always add something before and after its response. Doe…

Llama-2-chat models have been overly fine-tuned to be like this. You can give a few-shot prompting a try, but they still don't gurantee a desired output. The best way to guarantee is to fine-tune on small (~1k) data points and go from there.

fine tune the chat or base model?
Post reply on HN