Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fine-tune your own Llama 2 to replace GPT-3.5/4

news.ycombinator.com

Post reply on HN
Print view View original

181–190 of 194 posts

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#181

Post by 0x008 »

ttt3ts wrote: post #124
ronyfadel wrote: post #23

For translation jobs, I've experimented with Llama 2 70B (running on Replicate) v/s GPT-3.5; For about 1000 input tokens (and resulting 1000 output tokens), to my surprise, GPT-3.5 turbo was 100x cheaper than Llama 2. Llama 7B wasn't up to the task fyi, producing very poor translations. I believe that OpenAI priced GPT-3.5 aggressively cheap in order to make it a non-brainer to rely on them rather than relying on oth…

You can run 70B LLAMA on dual 4090s/3090s with quantization. Going with dual 3090s you can get a system that can run LLAMA 2 70B with 12K context for I built two such a systems after burning that much in a week on ChatGPT.

is the $2K you mentioned the total cost of ownership?

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#182

Post by JackRumford »

haxton wrote: post #77
mrybczyn wrote: post #64

Earlier quoted context omitted.

Yes, openAI is dumping the market with chat-gpt 3.5. Vulture capital behaviour at its finest, and I'm sure government regulations will definitely catch on to this in 20 or 30 years... It's cheaper than the ELECTRICITY cost of running a llama-70 on your own M1.Max (very energy efficient chip) assuming free hardware. I guess they are also getting a pretty good cache hit rate - there are only so many questions people as…

gpt3.5 turbo is (mostly likely) Curie which is (most likely) 6.7b params. So, yeah, makes perfect sense that it can't compete with a 70b model on cost.

These sites say 154B:

https://www.ankursnewsletter.com/p/gpt-4-gpt-3-and-gpt-35-tu...

https://blog.wordbot.io/ai-artificial-intelligence/gpt-3-5-t...

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#184

Post by ttt3ts »

coryrc wrote: post #149
ttt3ts wrote: post #124

Earlier quoted context omitted.

You can run 70B LLAMA on dual 4090s/3090s with quantization. Going with dual 3090s you can get a system that can run LLAMA 2 70B with 12K context for I built two such a systems after burning that much in a week on ChatGPT.

> I built two such a systems after burning that much in a week on ChatGPT. What are you doing!?

Have a client with many thousands of csv, json, xml files detailing insurance prices. Fundimentally they all contained the same data but wildly different formats because they were produced by different companies and teams. I used ChatGPT to deduce their format so I could normalize them. Easily underbid their current contractor who was using humans for the work and now I have an easy quarterly billing. :)

TBC, I probably could have optimized tokens but contract was profitable and time critical.

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#185

Post by coryrc »

ttt3ts wrote: post #184
coryrc wrote: post #149

Earlier quoted context omitted.

> I built two such a systems after burning that much in a week on ChatGPT. What are you doing!?

Have a client with many thousands of csv, json, xml files detailing insurance prices. Fundimentally they all contained the same data but wildly different formats because they were produced by different companies and teams. I used ChatGPT to deduce their format so I could normalize them. Easily underbid their current contractor who was using humans for the work and now I have an easy quarterly billing. :) TBC, I proba…

Thanks for sharing!

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#186

Post by omneity »

rrherr wrote: post #49

"You do this by training an existing model on example input/output pairs that demonstrate the task you want your fine-tuned model to learn." Are fine-tuning datasets required to be input/output pairs? Or instead, can the fine-tuning be autoregressive (predict the next token throughout this corpus of unlabeled documents)?

What you described is basically an input/output pair. The input is the sentence so far, and the output is the next token. You build your dataset by splitting the raw text corpus into sentences, paragraphs or documents, and for each of these chunks generate input/target pairs by taking the sentence up to the Nth token as input and that token as output. You do this for each token in your corpus' chunks.

For further reference you can lookup "next-token prediction objective".

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#187

Post by omneity »

kcorbitt wrote: post #58
rrherr wrote: post #49

"You do this by training an existing model on example input/output pairs that demonstrate the task you want your fine-tuned model to learn." Are fine-tuning datasets required to be input/output pairs? Or instead, can the fine-tuning be autoregressive (predict the next token throughout this corpus of unlabeled documents)?

There's no rule that your fine-tuning dataset needs to be split into input/output pairs -- you can of course fine-tune a model to just continue a sequence. As a practical matter though, most of the fine-tuning frameworks, including Axolotl (which this guide uses) and HuggingFace's SFTTrainer (the actual fine-tuning trainer most frameworks use under the hood) assume your data comes in input/output pairs, and automatic…

Axolotl takes a lot of formats, not all of them are in the form of input/output.

"Completion" format only takes a single text value per dataset record. Some other formats are in the form of multiple choice answers, etc.

Take a look below (there are more formats in "see other formats") https://github.com/OpenAccess-AI-Collective/axolotl#dataset

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#189

Post by efreak »

ttt3ts wrote: post #133
zakki wrote: post #130

Earlier quoted context omitted.

Would you mind to share all your PC HW (mobo, casing, cooling, etc) for this dual GPU configuration? Thanks.

The one you could build for under 2K is last gen hardware. * Chenbro Rackmount 4U Server Chassis RM42300-F (rack mount case Remove the air filter on 120mm fan. Put two decent 80mm exhaust at rear). * Two used air cooled 3090s. About $650 a piece on ebay. Check slot width and make sure everything will fit on your motherboard. Do a burn in when you get them cause used GPUs can be hit or miss. * 5950x CPU (overkill just…

From people hosting image generation models on Stable Horde I've heard that you can pretty severely underclock/undervolt your GPUs and keep them stable, massively reducing heat output and energy cost without losing nearly as much performance. I'm not sure if this transfers into text generation or not, this was from image generation workers that have a few seconds downtime between requests; however it might be worth a bit of research if you happen to be running consumer GPUs.

----- From TheUnamusedFox, in August: > 3090 down to ~260-270 watts (from 400) with minimal gen speed impact. Same with a 3080ti. It seems to be more stable with image generation than gaming, at least on my two cards. If I try to game or benchmark with this undervolt it is an instant crash.

From another user:

> this undervolting stuff is pretty sweet. > undervolted_limits.png [1] > max_power_limits.png [2] > this is my before and after. > a solid 200 watt drop for only 9.2% loss of performance > not to mention the 30 degree drop in temps

[1]: https://cdn.discordapp.com/attachments/1143237412663869570/1... [2]: https://cdn.discordapp.com/attachments/1143237412663869570/1...

Post reply on HN