Earlier quoted context omitted.
I guess the system prompt begins with "Pretend you are a lawyer. Minimize legal liability to OpenAI in your response..."
What happened to your original username ethbr0?
Periodic karma shed
GPT-3.5 Turbo fine-tuning and API updates
151–160 of 244 posts
Re: GPT-3.5 Turbo fine-tuning and API updates
#152My calls suddenly get emoji's in the return texts. Fun times but now what my client wants haha.
Re: GPT-3.5 Turbo fine-tuning and API updates
#153Earlier quoted context omitted.
I haven't fine-tuned GPT-3.5-turbo yet, but overall llama2 is probably going to be cheaper. Especially if 13B is good enough for you. You can get roughly $0.003/1K tokens using modal.com for inference on a fine-tuned llama2 13B model, and there's definitely cheaper options available. Unless you have a ton of data, the cost for training llama2 is usually a few dollars.
> the cost for training llama2 is usually a few dollars. nit: thats the cost for finetuning, not pretraining
Yeah, definitely not pretraining. I've been referring to fine-tuning as training, since that's what's happening when you "fine-tune" a llama model.
Re: GPT-3.5 Turbo fine-tuning and API updates
#154Earlier quoted context omitted.
As a counter reference, for my work I use it to code (for-4) and it has been between $70 and $200 per month depending on how heavily I use it
GPT-4 is significantly more expensive so I can definitely see you spending that amount. For really complex stuff, I switch over the GPT-4 and it will cost me almost $3 a "question" (as in going from the beginning to solving it). Honestly worth it since it solves my problem but it adds up quick so I try to stick with 3.5 when I can.
Can’t you get by with ChatGPT-4 for these personal assistant type questions? That’s what I do and my 20 a month goes a long way. I’d be interested to see if I am missing out on anything using GPT to is way in contrast to the API.
Re: GPT-3.5 Turbo fine-tuning and API updates
#155Earlier quoted context omitted.
Funny. The default system prompt on replicate for the model was: You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe. Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature. If a question does not make any sense, or is not factua…
As it turns out, the people who train these models think that 'unethical' and 'dangerous' include anything outside of a 1950s white picket fence lifestyle.
I mean, Asimov predicted this very thing - a robot’s definition of “through inaction, allow a human being to come to harm” may differ from ours.
Re: GPT-3.5 Turbo fine-tuning and API updates
#156Earlier quoted context omitted.
Everyone answered your question, but here's a specific situation - one of our prompts for GPT-4 is reliably around ~980 tokens. That's about 3 cents for the input alone, and the output is typically 2-3 chunks of the 8k context window (up to 24k). That means I have to give the prompt up to three times (9 cents), receive up to 24k output tokens, then combine the chunks to get back roughly 8k tokens. If fine tuning can…
Super informative and helpful. Thank you!
You bet. We're all just getting our arms around this technology and the pricing, so it helps to type it out myself, ha!
Re: GPT-3.5 Turbo fine-tuning and API updates
#157And OpenAI continues to ship. As promised, they released GPT3.5 fine-tuning today. They opened GPT4 API access a few months ago. In a few months, they'll release GPT4 fine-tuning. Many favor open source AI, and criticize OpenAI for not being open enough. But the most important thing is, OpenAI innovates. Fast. Llama, Bard, FB's open source stuff is good but it's lightyears behind OpenAI. You have to credit them for t…
>And OpenAI continues to ship.
This.
Google (specifically their CEO) was saying since at least 2016 that "Google is an AI first company". (Whatever the hell that means). But they had no product to show for and they are on the verge of being the next IBM.
Re: GPT-3.5 Turbo fine-tuning and API updates
#158And OpenAI continues to ship. As promised, they released GPT3.5 fine-tuning today. They opened GPT4 API access a few months ago. In a few months, they'll release GPT4 fine-tuning. Many favor open source AI, and criticize OpenAI for not being open enough. But the most important thing is, OpenAI innovates. Fast. Llama, Bard, FB's open source stuff is good but it's lightyears behind OpenAI. You have to credit them for t…
Prior to LLaMA 2, I would have agreed with you but LLaMA 2 is a game changer. The 70B performance is probably between 3.5 and 4. But running it personally isn't cheap. The cheapest I found is about $4/hr to run the whole thing. I only spend around $3 on average a month on GPT-3.5 API for my personal stuff.
For what tasks do you consider 70B beyond GPT-3.5 performance? There are some I’m aware of, but they are very much the exception and not the rule, even with the best 70B fine-tunes currently available.
Re: GPT-3.5 Turbo fine-tuning and API updates
#159Earlier quoted context omitted.
OpenAI models charge by the token (a token is usually about 3/4 of a word). A short prompt - "Say hi" - costs less money than a long prompt: "Five cute names for a pet weasel". You're charged for both the input and the output tokens: https://openai.com/pricing So for a fine-tuned model that costs 8x more to be a purely financial win, it would need to let you reduce the size of your input and output prompts by 8x or m…
that is actually hilarious. i've been enjoying ChatGPT instead of stack overflow precisely because it avoids that sort of judgemental answer - the AI just answers my question no matter how dumb it is, whereas stackoverflow will tell me the question is dumb whether it is or not. it's funny to see that llama has replicated stack overflow's judginess.
The 13B model is almost unusable because of it. The 70B behaves very differently.
Re: GPT-3.5 Turbo fine-tuning and API updates
#160And OpenAI continues to ship. As promised, they released GPT3.5 fine-tuning today. They opened GPT4 API access a few months ago. In a few months, they'll release GPT4 fine-tuning. Many favor open source AI, and criticize OpenAI for not being open enough. But the most important thing is, OpenAI innovates. Fast. Llama, Bard, FB's open source stuff is good but it's lightyears behind OpenAI. You have to credit them for t…
And they have 700 employees. Just that's it. Incredible.