Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fine-tune your own Llama 2 to replace GPT-3.5/4

news.ycombinator.com

Post reply on HN
Print view View original

111–120 of 194 posts

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#111

Post by mjirv »

thewataccount wrote: post #82
ramesh31 wrote: post #56

Earlier quoted context omitted.

>So the comparison would be the cost of renting a cloud GPU to run Llama vs querying ChatGPT. Yes, and it doesn't even come close. Llama2-70b can run inference at 300+tokens/s on a single V100 instance at ~$0.50/hr. Anyone who can should be switching away from OpenAI right now.

What's the best way to use LLama2-70b without existing infrastructure for orchestrating it?

I stumbled upon OpenRouter[0] a few days ago. Easiest I’ve seen by far (if you want SaaS, not hosting it yourself).

[0] https://openrouter.ai

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#112

Post by facu17y »

"to replace GPT-3.5/4"

Very inflated statement when it comes to GPT4 since it is a MoE model with 8 separate models each an expert in one area, and you can't replace all 8 models with one model trained for $19.

I call BS on this claim. Maybe it matches GPT4 in the narrow domain you fine-tune it for, and if that can be done for $19 then for $19*8 you can take OpenAI out of business. That doesn't add up.

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#113

Post by all2 »

daniels11 wrote: post #96
famouswaffles wrote: post #93

Earlier quoted context omitted.

>You're better off using models specialized in translation For a couple dozen languages, GPT-4 is by far the best translator you can get your hand on so basically no.

I will say that GPT-4 is just incredibly expensive. For my app I only use it for advanced translations/corrections, and usually a combination of GPT-3.5+Wiktionary is able to get the more simple stuff done

> GPT-3.5+Wiktionary

Can you share more about your app and what you're doing?

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#115

Post by kcorbitt »

derekpankaew wrote: post #107

Can you clarify the 50x cheaper number? Is this for self-hosting, or if you're hosting on OpenPipe? The pricing on OpenPipe says it's 0.0012 to 0.0016 per 1K tokens for Llama 7b. GPT-3.5 pricing is 0.0015 to 0.002, so not that different. I'm assuming the 50x cost reductions are primarily from self-hosting?

Yep, the 50x cost reduction is if you self-host a fine-tuned model using the setup demonstrated in in the linked notebooks.

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#116

Post by OhNoNotAgain_99 »

just curious would it be possible to add a small network perhaps a books of study material like programming books. freeze the weights of the existing large network, and combined with the new network try to predict the book. The existing networks know language but not the content, the combined network will be trained on the content, and eventually toegther they score better, These "small" added networks might just be specific towards a certain topic (ea learn python or so). Then these small networks can be become modular. esesentially creating some kind of lora networks for LLM's.

Maybe start this way from the ground up, so you can get modular units, for health, finance, programming, education, writting assitance, phyloophy, ethics etc etc. If the modules can be changed, then one might be able to reduce their seize. Ea pick 2 or 3 chain them and one has a LLM for a specific area of interest. (reducing running cost)

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#117

Post by llwj »

binarymax wrote: post #36

This looks awesome! Tangential question - do you find GPT function calling to work consistently and without error, or do you get errors when using it? By errors I mostly mean incorrect function signatures/types or missing values...but if you see other unpredictable behavior that would help too.

I see wrong responses about 1% of the time, but I love it, considering parsing raw text output without function calling had a much higher error rate.

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#118

Post by rrherr »

kcorbitt wrote: post #58
rrherr wrote: post #49

"You do this by training an existing model on example input/output pairs that demonstrate the task you want your fine-tuned model to learn." Are fine-tuning datasets required to be input/output pairs? Or instead, can the fine-tuning be autoregressive (predict the next token throughout this corpus of unlabeled documents)?

There's no rule that your fine-tuning dataset needs to be split into input/output pairs -- you can of course fine-tune a model to just continue a sequence. As a practical matter though, most of the fine-tuning frameworks, including Axolotl (which this guide uses) and HuggingFace's SFTTrainer (the actual fine-tuning trainer most frameworks use under the hood) assume your data comes in input/output pairs, and automatic…

“most tasks can be formulated this way, including autocomplete tasks”

For autocomplete tasks, with a corpus of unlabeled documents, would you insert a separator token at an arbitrary space in each document, in order to form input/output pairs?

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#119

Post by daniels11 »

all2 wrote: post #113
daniels11 wrote: post #96

Earlier quoted context omitted.

I will say that GPT-4 is just incredibly expensive. For my app I only use it for advanced translations/corrections, and usually a combination of GPT-3.5+Wiktionary is able to get the more simple stuff done

> GPT-3.5+Wiktionary Can you share more about your app and what you're doing?

Sure! I'm building a personalized AI language learning tutor using Open AI's API and ElevenLabs (for Text to Speech).

Right now it's basically a chat bot that you can use to practice conversing with. It provides corrections for the things you type. Eventually I'd like to try adding Whisper as well to allow users to speak out loud.

When you hover over a word, you get a translation. Initially I thought using Open AI for every word translation would be too much, but I've been able to get it down to ~36-40 tokens/request. (3-4 cents/1000 requests). I also began parsing and uploading some of this [Wiktionary data](https://kaikki.org/dictionary/rawdata.html) and am working on a feature that integrates the GPT-3.5 translation with this Wiktionary data.

A lot of these features are still in the works but you can feel free to try it if you like (https://trytutor.app).

Re: Fine-tune your own Llama 2 to replace GPT-3.5/4

#120

Post by daemonologist »

divbzero wrote: post #57

Is Llama 2 currently the way to go for fine-tuning your own models? Are there other open-source LLMs worth considering?

We've found Flan-T5 to be useful for text-to-text (mostly document QA). Haven't done a lot of testing on fine-tuning yet though.
Post reply on HN