There are more ways of evaluating LLMs than there are LLMs. All of the "X is better than Y" statements are pointless unless there is a very clear consensus.
LLaMA2 Chat 70B outperformed ChatGPT
71–80 of 135 posts
Re: LLaMA2 Chat 70B outperformed ChatGPT
#72The fact Guanaco 33B is at 65% with Vicuna 13B at 70.43% immediately makes these results non-sensical from my own experience with them.
Heck from my experience Guanaco 33B is better then Vicuna 33B!
Not to mention GPT4 at 95% and ChatGPT at 89% - I use chatgpt(3.5turbo)/gpt4 daily for work, and I rarely ever bother with 3.5turbo because of how unreliable it's answers are compared to gpt4.
So whatever this is effectively measuring is useless for comparing these models, especially across work types.
Re: LLaMA2 Chat 70B outperformed ChatGPT
#73Earlier quoted context omitted.
> to get an idea of pretty much what the average person would think There is no such thing as an average person. https://www.thestar.com/news/insight/when-u-s-air-force-disc...
Funnily enough I think you both might be right here, there isn't such a thing as an average person, but ChatGPT may be the synthesis of the average opinion.
what is an average opinion? it is the sum of opinions which disagree with the result
Re: LLaMA2 Chat 70B outperformed ChatGPT
#74Earlier quoted context omitted.
80% agreement is high, but the margins between models at the top are so low that even that remaining 20% could be enough to alter the final rankings, depending on which direction it errs.
Is perfect agreement possible? And what is the definition of agreement? Humans don't agree about much.. are we saying agreement means it matches the truth after intensive investigation by humans?
Ugh, when I was doing my PhD work we were studying creativity in an experiment, and we needed an assessment for how creative different solutions were, and trying to get inter-rater reliability on this quite simple thing was just agonizing.
I wound up abandoning the experiment because getting enough reliability would have required screwing down the standards so tightly that it would have ruined the underlying point of the thing.
Re: LLaMA2 Chat 70B outperformed ChatGPT
#75Earlier quoted context omitted.
There is one: "The agreement between GPT-4 and humans reaches 85%, which is even higher than the agreement among humans (81%). This means GPT-4’s judgments closely align with the majority of humans. We also show that GPT-4’s judgments may help humans make better judgments. During our data collection, when a human’s choice deviated from GPT-4, we presented GPT-4’s judgments to humans and ask if they are reasonable. De…
It's funny how ChatGPT really does give you the most balanced, middle of the road answers. It feels like a distillation of all human knowledge and sentiments. I use it constantly to get advice on plans, architectures, thoughts, etc.. to get an idea of pretty much what the average person would think. It often points out things I've overlooked which I'll improve my design with and go back and forth with ChatGPT until w…
Would put slight caution around asking it anything more than "what should I read next about this?"
For whatever reason*, it is particularly bad at discussing philosophy I find. When I was grading philosophy 101, I would have probably given it a passing grade against the overall curve, but that's about it. Philosophy is a discipline of careful, sometimes jargoney, and always very couched assertions that can be easily misunderstood and appropriated. This is probably its greatest weakness, and in many ways this weakness is the progenitor of philosophy itself in the Western world, with Plato at the start (i.e. with the figure of the sophist, the paradox of a false wisdom).
- Maybe one reason: there is a huge amount of, lets say, "armchair philosophy" on the internet, compared to other disciplines. Many blogposts and tiny manifestos of people really excited by some out of context quote from Spinoza or whatever. And you start to really feel this part of the dataset when you ask about philosophy. Many strange takes and misunderstandings.
Re: LLaMA2 Chat 70B outperformed ChatGPT
#76Better evaluation paints a bit different picture: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb... *FreeWilly2 is a Llama2 70B model finetuned on an Orca style Dataset EDIT: actually, impressive: FreeWilly2 GPT-3.5 GPT-4 ARC 71.1 85.2 96.3 HellaSwag 86.4 85.5 95.3 MMLU 68.8 70.0 86.4 TruthfulQA 59.4 47.0 59.0 So reasoning (ARC) is lagging behind, but the other evaluations are at GPT-3.5 level and closi…
That seems more in-line with my experience. I have been using GPT-3.5 and GPT-4 for data cleaning pipelines, and have tried to swap out LLaMA2 70B in a few of the "easier" tasks, and it hasn't performed well enough yet for any of my tasks done by GPT-3.5.
Was this with or without fine-tuning?
Re: LLaMA2 Chat 70B outperformed ChatGPT
#77There is a cool website where you can blind judge the outputs from LLaMa 2 vs ChatGPT-3.5: https://llmboxing.com/ Surprisingly, LLaMa 2 won 5-0 for me.
all the shorter answers were from GPT-3 - if you like long answers you pick llama 2...
Re: LLaMA2 Chat 70B outperformed ChatGPT
#78Those MacBook Pros with 96 GB of unified GPU/CPU memory are looking pretty good right now. It would be awesome to have all this running on a laptop in a completely offline mode.
I think it'd be more fun to spend an extra $700 and get an M2 Ultra Mac Studio with way more GPU cores and 128GB of RAM, and set up a private server. But if you really want a portable offline thing, sure.
I get more excited at the prospect of popping into some random cafe, SSHing from my iPad into some vast.ai server and setting loose this giant AI brain on whatever stuff. Feels badass.
Re: LLaMA2 Chat 70B outperformed ChatGPT
#79There is a cool website where you can blind judge the outputs from LLaMa 2 vs ChatGPT-3.5: https://llmboxing.com/ Surprisingly, LLaMa 2 won 5-0 for me.
Thanks for the link!
At least in my examples, the llama output was more verbose/comprehensive. Sometimes ChatGPT didn't expand enough, sometimes Llama missed the mark entirely (eg explaining the Eiffel's architecture.)
Re: LLaMA2 Chat 70B outperformed ChatGPT
#80Earlier quoted context omitted.
Hi! Could you please share a few words on what type of data you are cleaning using GPT? It is an intriguing idea and I would love to learn more to see if I could use a similar approach.
Youtube transcripts. It only works with single-person channels at the moment, as I haven't worked on disambiguating multiple speakers. They are very messy if they are just an auto transcription from Google. Practically unusable in most cases. So first step in the pipeline is cleaning up the transcripts for incorrectly transcribed words or sentences. Using the context of the rest of the transcript, it is able to fix t…
That's very informative - thanks so much for the explanation!
My only experience with transcripts is in the context of transcribing short interviews. I used Whisper and it was pretty good. I mostly work with quantitative data, though.
In terms of the disambiguation of speakers, I haven't done it, but I remember blind signal separation discussed in a signal processing seminar I attended. There is also this paper, in case you haven't seen it already: https://enk100.github.io/speaker_separation/
Thanks again!