81–90 of 135 posts
There is a cool website where you can blind judge the outputs from LLaMa 2 vs ChatGPT-3.5: https://llmboxing.com/ Surprisingly, LLaMa 2 won 5-0 for me.
Earlier quoted context omitted.
Funnily enough I think you both might be right here, there isn't such a thing as an average person, but ChatGPT may be the synthesis of the average opinion.
what is an average opinion? it is the sum of opinions which disagree with the result
(But maybe it's a good filter: if someone is talking about "ChatGPT's" performance, they probably don't have anything useful to say.)
Earlier quoted context omitted.
You will want to look at HumanEval ( https://github.com/abacaj/code-eval ) and Eval+ ( https://github.com/my-other-github-account/llm-humaneval-ben... ) results for coding. While Llama2 is an improvement over LLaMA v1, it's still nowhere near even the best open models (currently, sans test contamination, WizardCoder-15B, a StarCoder fine tune is at top). It's really not a competition atm though, ChatGPT-4 wipes the f…
this all numbers can be missleading, and simply indicate that gpt have these tasks in training data, and another model doesn't.
This was the HumanEval contamination one dev measured: ``` replit_glaive: 56.71% replit: 7.32% wizard: 4.88% ```
From the WizardCoder paper https://arxiv.org/pdf/2306.08568.pdf you can see that it hits SOTA (for open models) in not just HumanEval and HumanEval+, but also MBPP and DS-1000 as well, so it's not a one off.
For those interested in reading more about various considerations for coding models, I highly recommend reading the MSR phi-1 paper: https://arxiv.org/pdf/2306.11644.pdf
Looking forward to if they ever publish code/model/dataset since it has extremely strong performance trained on a very small number of tokens very manageable 1.3B and 350M parameter models.
There is a cool website where you can blind judge the outputs from LLaMa 2 vs ChatGPT-3.5: https://llmboxing.com/ Surprisingly, LLaMa 2 won 5-0 for me.
They both gave great answers overall though.
Earlier quoted context omitted.
Anecdotal evidence here - I find that the API is less likely to ask questions about what you are doing and get straight to the answer. For example, if I were to ask how to do something with burp it will just answer instead of going into the "as an AI" monologue.
Do something with burp?
Earlier quoted context omitted.
Anecdotal evidence here - I find that the API is less likely to ask questions about what you are doing and get straight to the answer. For example, if I were to ask how to do something with burp it will just answer instead of going into the "as an AI" monologue.
Do something with burp?
The value of GPT-4 also lies in its stored knowledge. A 70B model can't store that much.
The advantage of LLaMA 2 is that a company can fine tune it on the knowledge that they actually care about and then run it on their own hardware without paying API fees or relying on an unstable dependency that's constantly being tweaked.
There is a cool website where you can blind judge the outputs from LLaMa 2 vs ChatGPT-3.5: https://llmboxing.com/ Surprisingly, LLaMa 2 won 5-0 for me.
Better evaluation paints a bit different picture: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb... *FreeWilly2 is a Llama2 70B model finetuned on an Orca style Dataset EDIT: actually, impressive: FreeWilly2 GPT-3.5 GPT-4 ARC 71.1 85.2 96.3 HellaSwag 86.4 85.5 95.3 MMLU 68.8 70.0 86.4 TruthfulQA 59.4 47.0 59.0 So reasoning (ARC) is lagging behind, but the other evaluations are at GPT-3.5 level and closi…
LLaMA2 is far and away from GPT 3.5. Just look at HumanEval and other code generation metrics. All these GPT-4 based "chat evals" are extremely misleading and people should take it with a bag of salt.
The ranking I linked and quoted in my comment's is much better. See the About tab. It has 4 evaluations and it doesn't use GPT-4 to evaluate.
Also the top one is a tuned Llama 2. Also clarified in my original comment.