Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fine-tune your own Llama 2 to replace GPT-3.5/4

news.ycombinator.com

Post reply on HN
Print view View original

121–130 of 194 posts

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#121

Post by jiggawatts »

haxton wrote: post #77
mrybczyn wrote: post #64

Earlier quoted context omitted.

Yes, openAI is dumping the market with chat-gpt 3.5. Vulture capital behaviour at its finest, and I'm sure government regulations will definitely catch on to this in 20 or 30 years... It's cheaper than the ELECTRICITY cost of running a llama-70 on your own M1.Max (very energy efficient chip) assuming free hardware. I guess they are also getting a pretty good cache hit rate - there are only so many questions people as…

gpt3.5 turbo is (mostly likely) Curie which is (most likely) 6.7b params. So, yeah, makes perfect sense that it can't compete with a 70b model on cost.

I thought it was fairly well established that GPT 3.5 has something like 130B parameters and that GPT 4 is on the order of 600-1,000

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#122

Post by sandkoan »

OhNoNotAgain_99 wrote: post #116

just curious would it be possible to add a small network perhaps a books of study material like programming books. freeze the weights of the existing large network, and combined with the new network try to predict the book. The existing networks know language but not the content, the combined network will be trained on the content, and eventually toegther they score better, These "small" added networks might just be…

This is part of what we're doing at Automorphic. Building shareable, stackable adapters that you can compose like lego bricks.

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#123

Post by axpy906 »

MuffinFlavored wrote: post #35
ronyfadel wrote: post #23

For translation jobs, I've experimented with Llama 2 70B (running on Replicate) v/s GPT-3.5; For about 1000 input tokens (and resulting 1000 output tokens), to my surprise, GPT-3.5 turbo was 100x cheaper than Llama 2. Llama 7B wasn't up to the task fyi, producing very poor translations. I believe that OpenAI priced GPT-3.5 aggressively cheap in order to make it a non-brainer to rely on them rather than relying on oth…

I thought Llama was opensource/free and you could run it yourself?

Unfortunately, Lama2 is not a fully open source license.

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#124

Post by ttt3ts »

ronyfadel wrote: post #23

For translation jobs, I've experimented with Llama 2 70B (running on Replicate) v/s GPT-3.5; For about 1000 input tokens (and resulting 1000 output tokens), to my surprise, GPT-3.5 turbo was 100x cheaper than Llama 2. Llama 7B wasn't up to the task fyi, producing very poor translations. I believe that OpenAI priced GPT-3.5 aggressively cheap in order to make it a non-brainer to rely on them rather than relying on oth…

You can run 70B LLAMA on dual 4090s/3090s with quantization. Going with dual 3090s you can get a system that can run LLAMA 2 70B with 12K context for I built two such a systems after burning that much in a week on ChatGPT.

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#125

Post by ramesh31 »

pocketarc wrote: post #85
ramesh31 wrote: post #68

Earlier quoted context omitted.

Offloading 40% of layers to CPU, about 50t/s with 16 threads.

That is more than an order of magnitude better than my experience; I get around 2 t/s with similar hardware. I had also seen others reporting similar figures to mine so I assumed it was normal. Is there a secret to what you're doing?

>Is there a secret to what you're doing?

Core speed and memory bandwidth matter a lot. This is on a Ryzen 7950 with DDR5.

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#126

Post by jlm521 »

tikkun wrote: post #3

Looks really well executed, nice! I'd shared this idea with a few people. GPT and other LLMs don't allow you to use their output to train competing models, but the implication is that it's fine to use their output to train your own internal alternative models. So you can't sell access to the output as an API, but you can use it to replace your GPT API calls. My other thoughts to extend this are that you could make it…

I would be like to help in reviewing your handbook.

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#127

Post by nborwankar »

ronyfadel wrote: post #23

For translation jobs, I've experimented with Llama 2 70B (running on Replicate) v/s GPT-3.5; For about 1000 input tokens (and resulting 1000 output tokens), to my surprise, GPT-3.5 turbo was 100x cheaper than Llama 2. Llama 7B wasn't up to the task fyi, producing very poor translations. I believe that OpenAI priced GPT-3.5 aggressively cheap in order to make it a non-brainer to rely on them rather than relying on oth…

Llama and GPT are auto-regressive decoder only architectures which for pure translation jobs are not the optimal architectures. Training seq2seq models or encoder/decoder models on datasets of sentence pairs designed for translation will likely allow you to use much smaller models. You will not be wasting parameters on general “language understanding” capability that Llama and GPT have if pure translation is all you need. T5 or Flan-T5 might be good starting points.

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#128

Post by FeepingCreature »

kelseyfrog wrote: post #108
Arctic_fly wrote: post #30

Earlier quoted context omitted.

> Llama 7B wasn't up to the task fyi, producing very poor translations. From what I've read and personally experimented with, none of the Llama 2 models are well-suited to translation in particular (they were mainly trained on English data). Still, there are a number of tasks that they're really good at if fine-tuned correctly, such as classification and data extraction. > I believe that OpenAI priced GPT-3.5 aggress…

Is that because translation is typically an encoder-decoder task and llama is decoder only or is there something else about it that makes the last difficult for llama?

If you don't make it learn other-language texts, it won't be able to speak that language.

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#130

Post by zakki »

ttt3ts wrote: post #124
ronyfadel wrote: post #23

For translation jobs, I've experimented with Llama 2 70B (running on Replicate) v/s GPT-3.5; For about 1000 input tokens (and resulting 1000 output tokens), to my surprise, GPT-3.5 turbo was 100x cheaper than Llama 2. Llama 7B wasn't up to the task fyi, producing very poor translations. I believe that OpenAI priced GPT-3.5 aggressively cheap in order to make it a non-brainer to rely on them rather than relying on oth…

You can run 70B LLAMA on dual 4090s/3090s with quantization. Going with dual 3090s you can get a system that can run LLAMA 2 70B with 12K context for I built two such a systems after burning that much in a week on ChatGPT.

Would you mind to share all your PC HW (mobo, casing, cooling, etc) for this dual GPU configuration? Thanks.
Post reply on HN