Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fine-tune your own Llama 2 to replace GPT-3.5/4

news.ycombinator.com

Post reply on HN
Print view View original

91–100 of 194 posts

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#91

Post by avereveard »

ronyfadel wrote: post #23

For translation jobs, I've experimented with Llama 2 70B (running on Replicate) v/s GPT-3.5; For about 1000 input tokens (and resulting 1000 output tokens), to my surprise, GPT-3.5 turbo was 100x cheaper than Llama 2. Llama 7B wasn't up to the task fyi, producing very poor translations. I believe that OpenAI priced GPT-3.5 aggressively cheap in order to make it a non-brainer to rely on them rather than relying on oth…

Together AI has new aggressive pricing where 70b are on par with gpt35 and everything smaller is fairly cheaper. The catch is the only 32k context length model as of today is their llama 7b which is fairly limited.

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#92

Post by idosh »

Arctic_fly wrote: post #53
idosh wrote: post #45

Can you elaborate on your plans for OpenPipe? Sounds like a very interesting project

Currently OpenPipe allows you to capture input/output from a powerful model and use it to fine-tune a much smaller one, then offers you the option to host through OpenPipe or download it and host it elsewhere. Models hosted on OpenPipe enjoy a few benefits, like data drift detection and automatic reformatting of output to match the original model you trained against (think extraction "function call" responses from a…

Thanks! I need to dive into the project and learn more. Sounds exciting

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#93

Post by famouswaffles »

halflings wrote: post #27
ronyfadel wrote: post #23

For translation jobs, I've experimented with Llama 2 70B (running on Replicate) v/s GPT-3.5; For about 1000 input tokens (and resulting 1000 output tokens), to my surprise, GPT-3.5 turbo was 100x cheaper than Llama 2. Llama 7B wasn't up to the task fyi, producing very poor translations. I believe that OpenAI priced GPT-3.5 aggressively cheap in order to make it a non-brainer to rely on them rather than relying on oth…

I don't think translation is a great use case for ChatGPT and LLAMA. These models are overwhelmingly trained on English, and LLAMA2 which should have more data from other languages is still focused on languages w/ Latin/Cyrillic characters (so won't work well for Arabic, Hebrew, or CJK languages). You're better off using models specialized in translation; General purpose LLMs are more useful when fine-tuning on speci…

>You're better off using models specialized in translation

For a couple dozen languages, GPT-4 is by far the best translator you can get your hand on so basically no.

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#95

Post by hereonout2 »

thewataccount wrote: post #62
MuffinFlavored wrote: post #35

Earlier quoted context omitted.

I thought Llama was opensource/free and you could run it yourself?

You (currently) need a GPU to run any of the useful models. I haven't really seen a business use-case that runs it on the user's computer, but given the hardware requirements it wouldn't be very feasible to expect. So you'll have to figure out how to run/scale the model inference. Cloud GPU instances are generally very expensive, and once you start needing to horizontally scale it'll get messy fast. At least at the m…

> once you start needing to horizontally scale it'll get messy fast.

It gets expensive fast, but not messy, these things scale horizontally really well. All the state is encapsulated in the request, no replication, synchronisation, user data to worry about. I'd rather have the job of horizontally scaling llama2 than a relational database.

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#96

Post by daniels11 »

famouswaffles wrote: post #93
halflings wrote: post #27

Earlier quoted context omitted.

I don't think translation is a great use case for ChatGPT and LLAMA. These models are overwhelmingly trained on English, and LLAMA2 which should have more data from other languages is still focused on languages w/ Latin/Cyrillic characters (so won't work well for Arabic, Hebrew, or CJK languages). You're better off using models specialized in translation; General purpose LLMs are more useful when fine-tuning on speci…

>You're better off using models specialized in translation For a couple dozen languages, GPT-4 is by far the best translator you can get your hand on so basically no.

I will say that GPT-4 is just incredibly expensive. For my app I only use it for advanced translations/corrections, and usually a combination of GPT-3.5+Wiktionary is able to get the more simple stuff done

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#97

Post by csjh »

haxton wrote: post #77
mrybczyn wrote: post #64

Earlier quoted context omitted.

Yes, openAI is dumping the market with chat-gpt 3.5. Vulture capital behaviour at its finest, and I'm sure government regulations will definitely catch on to this in 20 or 30 years... It's cheaper than the ELECTRICITY cost of running a llama-70 on your own M1.Max (very energy efficient chip) assuming free hardware. I guess they are also getting a pretty good cache hit rate - there are only so many questions people as…

gpt3.5 turbo is (mostly likely) Curie which is (most likely) 6.7b params. So, yeah, makes perfect sense that it can't compete with a 70b model on cost.

Is there a source on that? I've never seen anyone think it's below even 70B

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#98

Post by csjh »

sacred_numbers wrote: post #76
mrybczyn wrote: post #64

Earlier quoted context omitted.

Yes, openAI is dumping the market with chat-gpt 3.5. Vulture capital behaviour at its finest, and I'm sure government regulations will definitely catch on to this in 20 or 30 years... It's cheaper than the ELECTRICITY cost of running a llama-70 on your own M1.Max (very energy efficient chip) assuming free hardware. I guess they are also getting a pretty good cache hit rate - there are only so many questions people as…

Based on my research, GPT-3.5 is likely significantly smaller than 70B parameters, so it would make sense that it's cheaper to run. My guess is that OpenAI significantly overtrained GPT-3.5 to get as small a model as possible to optimize for inference. Also, Nvidia chips are way more efficient at inference than M1 Max. OpenAI also has the advantage of batching API calls which leads to better hardware utilization. I d…

What makes you think 3.5 is significantly smaller than 70B?

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#99

Post by achileas »

halflings wrote: post #27
ronyfadel wrote: post #23

For translation jobs, I've experimented with Llama 2 70B (running on Replicate) v/s GPT-3.5; For about 1000 input tokens (and resulting 1000 output tokens), to my surprise, GPT-3.5 turbo was 100x cheaper than Llama 2. Llama 7B wasn't up to the task fyi, producing very poor translations. I believe that OpenAI priced GPT-3.5 aggressively cheap in order to make it a non-brainer to rely on them rather than relying on oth…

I don't think translation is a great use case for ChatGPT and LLAMA. These models are overwhelmingly trained on English, and LLAMA2 which should have more data from other languages is still focused on languages w/ Latin/Cyrillic characters (so won't work well for Arabic, Hebrew, or CJK languages). You're better off using models specialized in translation; General purpose LLMs are more useful when fine-tuning on speci…

There are plenty of examples in the literature of using LLMs for translation beating the metrics of non-LLM models, even for languages for which there isn't a lot of data. Transliterating non-Latin characters helps a lot with accuracy as well.
Post reply on HN