Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLaMA2 Chat 70B outperformed ChatGPT

tatsu-lab.github.io

Post reply on HN
Print view View original

11–20 of 135 posts

Re: LLaMA2 Chat 70B outperformed ChatGPT

#11

Post by fullshark »

knodi123 wrote: post #4

Yeah, my experience has been that every one of these freely downloadable models can be measured as "percent of chatgpt quality". and getting up to 85% is shockingly good. *edit: oops, my brain inserted "by" in the middle of "outperformed chatgpt". I'll leave my wrong comment up as a testament to shame.

The reality is probably some queries ChatGPT outperforms and vice versa. Regardless the premise that ChatGPT's secret sauce could be hidden forever is very dead.

Re: LLaMA2 Chat 70B outperformed ChatGPT

#13

Post by alecco »

Better evaluation paints a bit different picture:

https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb...

*FreeWilly2 is a Llama2 70B model finetuned on an Orca style Dataset

EDIT: actually, impressive:

                   FreeWilly2  GPT-3.5  GPT-4
    ARC               71.1      85.2     96.3
    HellaSwag         86.4      85.5     95.3
    MMLU              68.8      70.0     86.4
    TruthfulQA        59.4      47.0     59.0
So reasoning (ARC) is lagging behind, but the other evaluations are at GPT-3.5 level and closing the gap with 4.

Source for GPT-3.5 and GPT-4.0 values (but mind it might not be the same # of shots)

https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb...

Re: LLaMA2 Chat 70B outperformed ChatGPT

#16

Post by bestcoder69 »

bazmattaz wrote: post #10
nmfisher wrote: post #3

I haven't had a chance to use the GPT-4 API yet - is it that much better than the GPT-4 available via ChatGPT? Or am I misunderstanding?

ChatGPT uses the GPT-4, but there are conspiracy theories circling that ChatGPT is neutered and thus not as good as GPT-4 through the API. The theory being that OpenAI are thottling the free version of GPT-4 (ChatGPT)

Free ChatGPT runs 3.5. You have to upgrade to plus to use GPT-4. The APIs seem close to ChatGPT, but it’s a little opaque what they’re actually doing. If you inspect network requests the models are named something like “chat-render-3.5” instead of the API model names.

I’d imagine OpenAI might run experiments on ChatGPT that they wouldn’t on the API, to avoid breaking 3rd party applications unannounced.

Re: LLaMA2 Chat 70B outperformed ChatGPT

#17

Post by Fergusonb »

nmfisher wrote: post #3

I haven't had a chance to use the GPT-4 API yet - is it that much better than the GPT-4 available via ChatGPT? Or am I misunderstanding?

Anecdotal evidence here - I find that the API is less likely to ask questions about what you are doing and get straight to the answer.

For example, if I were to ask how to do something with burp it will just answer instead of going into the "as an AI" monologue.

Re: LLaMA2 Chat 70B outperformed ChatGPT

#18

Post by lolinder »

luckystarr wrote: post #14

The value of GPT-4 also lies in its stored knowledge. A 70B model can't store that much.

The advantage of LLaMA 2 is that a company can fine tune it on the knowledge that they actually care about and then run it on their own hardware without paying API fees or relying on an unstable dependency that's constantly being tweaked.

Re: LLaMA2 Chat 70B outperformed ChatGPT

#19

Post by valine »

generalizations wrote: post #2

* ChatGPT 3.5. But it's also within spitting distance of GPT4, which is very exciting.

GPT4 is more difficult to measure I think. The value I get from GPT4 is in the details it gets right on very obscure, complex questions. I'm not sure benchmarks are capturing how far GPT4 is ahead of other models. For simple stuff it's not that much better than 3.5.
Post reply on HN