LLaMA2 Chat 70B outperformed ChatGPT

#1

Post by georgehill » Thu, Jul 27, 2023, 3:44 PM UTC

LLaMA2 Chat 70B outperformed ChatGPT
tatsu-lab.github.io

Re: LLaMA2 Chat 70B outperformed ChatGPT

#2

Post by generalizations » Thu, Jul 27, 2023, 4:05 PM UTC

* ChatGPT 3.5. But it's also within spitting distance of GPT4, which is very exciting.

Re: LLaMA2 Chat 70B outperformed ChatGPT

#3

Post by nmfisher » Thu, Jul 27, 2023, 4:06 PM UTC

I haven't had a chance to use the GPT-4 API yet - is it that much better than the GPT-4 available via ChatGPT? Or am I misunderstanding?

Re: LLaMA2 Chat 70B outperformed ChatGPT

#4

Post by knodi123 » Thu, Jul 27, 2023, 4:06 PM UTC

Yeah, my experience has been that every one of these freely downloadable models can be measured as "percent of chatgpt quality". and getting up to 85% is shockingly good.

*edit: oops, my brain inserted "by" in the middle of "outperformed chatgpt". I'll leave my wrong comment up as a testament to shame.

Re: LLaMA2 Chat 70B outperformed ChatGPT

#5

Post by cj » Thu, Jul 27, 2023, 4:07 PM UTC

It looks like ChatGPT length is 827 while LLaMA2 length is more than double at 1790.

Disclaimer from the site:

> Caution: GPT-4 may favor models with longer outputs and/or those that were fine-tuned on GPT-4 outputs.

> While AlpacaEval provides a useful comparison of model capabilities in following instructions, it is not a comprehensive or gold-standard evaluation of model abilities. For one, as detailed in the AlpacaFarm paper, the auto annotator winrates are correlated with length.

Re: LLaMA2 Chat 70B outperformed ChatGPT

#6

Post by freedomben » Thu, Jul 27, 2023, 4:08 PM UTC

Llama2 beat ChatGPT 3.5 with a 92.66% win rate to 89.37%, but lost to GPT-4 which got 95.28%. Still pretty amazing though!

Re: LLaMA2 Chat 70B outperformed ChatGPT

#7

Post by freedomben » Thu, Jul 27, 2023, 4:10 PM UTC

nmfisher wrote: post #3

I haven't had a chance to use the GPT-4 API yet - is it that much better than the GPT-4 available via ChatGPT? Or am I misunderstanding?

ChatGPT uses the GPT-4 API, so it's the same. With the API directly though you can change the system prompt, which can enable better results if you know what you're doing.

Re: LLaMA2 Chat 70B outperformed ChatGPT

#8

Post by accrual » Thu, Jul 27, 2023, 4:12 PM UTC

Does this mean it may be possible to self-host a ChatGPT clone assuming you have a 70B model? I've used a 13B model with LLaMA1 and it's surprisingly good, but still nowhere near ChatGPT for coding questions.

Re: LLaMA2 Chat 70B outperformed ChatGPT

#9

Post by make3 » Thu, Jul 27, 2023, 4:13 PM UTC

generalizations wrote: post #2

* ChatGPT 3.5. But it's also within spitting distance of GPT4, which is very exciting.

performance is logarithmic as a function of money invested in compute, so maybe it's close but it's also far away

Re: LLaMA2 Chat 70B outperformed ChatGPT

#10

Post by bazmattaz » Thu, Jul 27, 2023, 4:14 PM UTC

nmfisher wrote: post #3

I haven't had a chance to use the GPT-4 API yet - is it that much better than the GPT-4 available via ChatGPT? Or am I misunderstanding?

ChatGPT uses the GPT-4, but there are conspiracy theories circling that ChatGPT is neutered and thus not as good as GPT-4 through the API. The theory being that OpenAI are thottling the free version of GPT-4 (ChatGPT)