Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fine-tune your own Llama 2 to replace GPT-3.5/4

news.ycombinator.com

Post reply on HN
Print view View original

101–110 of 194 posts

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#101

Post by moonchrome »

minimaxir wrote: post #11
moonchrome wrote: post #8

Earlier quoted context omitted.

We are talking about 7B models ? Those can run on consumer GPUs with lower latency than A100s AFAIK (because gaming GPUs are clocked different). Not to mention OpenAI has shit latency and terrible reliability - you should be using Azure models if you care about that - but pricing is also higher. I would say fixed costs and development time is on openai side but I've seen people post great practical comparisons for la…

"Running" and "acceptable inference speed and quality" are two different constraints, particularly at scale/production.

I don't understand what you're trying to say ?

From what I've read 4090 should blow A100 away if you can fit within 22GB VRAM, which a 7B model should comfortably.

And the latency (along with variability and availability) on OpenAI API is terrible because of the load they are getting.

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#102

Post by bfirsh »

kcorbitt wrote: post #28
ronyfadel wrote: post #23

For translation jobs, I've experimented with Llama 2 70B (running on Replicate) v/s GPT-3.5; For about 1000 input tokens (and resulting 1000 output tokens), to my surprise, GPT-3.5 turbo was 100x cheaper than Llama 2. Llama 7B wasn't up to the task fyi, producing very poor translations. I believe that OpenAI priced GPT-3.5 aggressively cheap in order to make it a non-brainer to rely on them rather than relying on oth…

Yes, if you're just using Llama 2 off the shelf (without fine-tuning) I don't think there are a lot of workloads where it makes sense as a replacement for GPT-3.5. The one exception being for organizations where data security is non-negotiable and they really need to host on-prem. The calculus changes drastically though when you bring fine-tuning in, which lets a much smaller model outperform a larger one on many cla…

Founder of Replicate here. It's early indeed.

OpenAI aren't doing anything magic. We're optimizing Llama inference at the moment and it looks like we'll be able to roughly match GPT 3.5's price for Llama 2 70B.

Running a fine-tuned GPT-3.5 is surprisingly expensive. That's where using Llama makes a ton of sense. Once we’ve optimized inference, it’ll be much cheaper to run a fine-tuned Llama.

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#103

Post by kcorbitt »

rookie123 wrote: post #94

To all those who are on this panel, which is the most comprehensive way a newbie can learn fine-tuning these models with or without the GPUs? Are there any well directed courses available?

I wrote the notebooks in the post with the intention of them being a gentle introduction to fine-tuning. Would love any feedback on open questions you have as you go through them!

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#105

Post by thewataccount »

hereonout2 wrote: post #95
thewataccount wrote: post #62

Earlier quoted context omitted.

You (currently) need a GPU to run any of the useful models. I haven't really seen a business use-case that runs it on the user's computer, but given the hardware requirements it wouldn't be very feasible to expect. So you'll have to figure out how to run/scale the model inference. Cloud GPU instances are generally very expensive, and once you start needing to horizontally scale it'll get messy fast. At least at the m…

> once you start needing to horizontally scale it'll get messy fast. It gets expensive fast, but not messy, these things scale horizontally really well. All the state is encapsulated in the request, no replication, synchronisation, user data to worry about. I'd rather have the job of horizontally scaling llama2 than a relational database.

For sure, and yeah it wouldn't be terrible you're right. You'd just need the api servers + a load balancer.

My thing is that dynamically doing that is still a lot compared to just calling a single endpoint and all of that is handled for you.

But for sure this is a very decent horizontal use-case.

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#106

Post by [deleted] »

7speter wrote: post #22
moonchrome wrote: post #8

Earlier quoted context omitted.

We are talking about 7B models ? Those can run on consumer GPUs with lower latency than A100s AFAIK (because gaming GPUs are clocked different). Not to mention OpenAI has shit latency and terrible reliability - you should be using Azure models if you care about that - but pricing is also higher. I would say fixed costs and development time is on openai side but I've seen people post great practical comparisons for la…

When you say it can run on consumer gpus, do you mean pretty much just the 4090/3090 or can it run on lesser cards?

[deleted]

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#107

Post by derekpankaew »

Can you clarify the 50x cheaper number? Is this for self-hosting, or if you're hosting on OpenPipe?

The pricing on OpenPipe says it's 0.0012 to 0.0016 per 1K tokens for Llama 7b. GPT-3.5 pricing is 0.0015 to 0.002, so not that different.

I'm assuming the 50x cost reductions are primarily from self-hosting?

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#108

Post by kelseyfrog »

Arctic_fly wrote: post #30
ronyfadel wrote: post #23

For translation jobs, I've experimented with Llama 2 70B (running on Replicate) v/s GPT-3.5; For about 1000 input tokens (and resulting 1000 output tokens), to my surprise, GPT-3.5 turbo was 100x cheaper than Llama 2. Llama 7B wasn't up to the task fyi, producing very poor translations. I believe that OpenAI priced GPT-3.5 aggressively cheap in order to make it a non-brainer to rely on them rather than relying on oth…

> Llama 7B wasn't up to the task fyi, producing very poor translations. From what I've read and personally experimented with, none of the Llama 2 models are well-suited to translation in particular (they were mainly trained on English data). Still, there are a number of tasks that they're really good at if fine-tuned correctly, such as classification and data extraction. > I believe that OpenAI priced GPT-3.5 aggress…

Is that because translation is typically an encoder-decoder task and llama is decoder only or is there something else about it that makes the last difficult for llama?

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#109

Post by why_only_15 »

haxton wrote: post #77
mrybczyn wrote: post #64

Earlier quoted context omitted.

Yes, openAI is dumping the market with chat-gpt 3.5. Vulture capital behaviour at its finest, and I'm sure government regulations will definitely catch on to this in 20 or 30 years... It's cheaper than the ELECTRICITY cost of running a llama-70 on your own M1.Max (very energy efficient chip) assuming free hardware. I guess they are also getting a pretty good cache hit rate - there are only so many questions people as…

gpt3.5 turbo is (mostly likely) Curie which is (most likely) 6.7b params. So, yeah, makes perfect sense that it can't compete with a 70b model on cost.

gpt3.5 turbo is a new model, not Curie. As others have stated, it probably uses Mixture of Experts which lowers inference cost.

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#110

Post by why_only_15 »

PUSH_AX wrote: post #69
mrybczyn wrote: post #64

Earlier quoted context omitted.

Yes, openAI is dumping the market with chat-gpt 3.5. Vulture capital behaviour at its finest, and I'm sure government regulations will definitely catch on to this in 20 or 30 years... It's cheaper than the ELECTRICITY cost of running a llama-70 on your own M1.Max (very energy efficient chip) assuming free hardware. I guess they are also getting a pretty good cache hit rate - there are only so many questions people as…

You think they are caching? Even though one of the parameters is temperature? Can of worms, and should be reflected in the pricing if true, don't get me started if they are charging per token for cached responses. I just don't see it.

You can keep around the KV cache from previous generations which lowers the cost of prompts significantly.
Post reply on HN