31–40 of 135 posts
Does this mean it may be possible to self-host a ChatGPT clone assuming you have a 70B model? I've used a 13B model with LLaMA1 and it's surprisingly good, but still nowhere near ChatGPT for coding questions.
But, well, I am currently not hyped enough about it to actually try.
Does this mean it may be possible to self-host a ChatGPT clone assuming you have a 70B model? I've used a 13B model with LLaMA1 and it's surprisingly good, but still nowhere near ChatGPT for coding questions.
Lets just hope that there won’t be any embarrassing vulnerabilities coming out of this when someone could prompt the model to reveal its own environment variables or API keys or the internal prompt that it is using.
But it seems the $0 free AI models are eating OpenAI’s lunch and Meta so far is winning the race to zero.
GPT-4 level models that regular people can run with a reasonable hardware budget are going to require innovations in optimization and model efficiency beyond just quantizing weights. Rumor has it that GPT-4 is a "committee" of ~220G models, which would require ~128GiB VRAM at 4-bit quantization to run each model.
Better evaluation paints a bit different picture: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb... *FreeWilly2 is a Llama2 70B model finetuned on an Orca style Dataset EDIT: actually, impressive: FreeWilly2 GPT-3.5 GPT-4 ARC 71.1 85.2 96.3 HellaSwag 86.4 85.5 95.3 MMLU 68.8 70.0 86.4 TruthfulQA 59.4 47.0 59.0 So reasoning (ARC) is lagging behind, but the other evaluations are at GPT-3.5 level and closi…
It beats out ChatGPT in every category except SAT-Math. We definitely need harder benchmarks.
So far, there's BIG-Bench Hard https://github.com/suzgunmirac/BIG-Bench-Hard and just published, Advanced Reasoning Benchmark https://arb.duckai.org/
*When asked by GPT4 to compare the outputs. I'm a staunch believer that it would be foolish to rely on GPT4 for quality comparisons, and it has been mind boggling to see so many people do it and treat it as perfect proof of anything. It would be slightly more understandable if there was a study to see how human and gpt4 preferences compare, but I'm unaware of any such thing.
Paul Gauthier made this benchmark [1] to measure correct git diffs. If you ask GPT-4 for help with your code, it can output a change in a git diff more reliably than 3.5.
My hope is that we can do that with Llama 2.
*When asked by GPT4 to compare the outputs. I'm a staunch believer that it would be foolish to rely on GPT4 for quality comparisons, and it has been mind boggling to see so many people do it and treat it as perfect proof of anything. It would be slightly more understandable if there was a study to see how human and gpt4 preferences compare, but I'm unaware of any such thing.
There is one: "The agreement between GPT-4 and humans reaches 85%, which is even higher than the agreement among humans (81%). This means GPT-4’s judgments closely align with the majority of humans. We also show that GPT-4’s judgments may help humans make better judgments. During our data collection, when a human’s choice deviated from GPT-4, we presented GPT-4’s judgments to humans and ask if they are reasonable. De…
Does this mean it may be possible to self-host a ChatGPT clone assuming you have a 70B model? I've used a 13B model with LLaMA1 and it's surprisingly good, but still nowhere near ChatGPT for coding questions.
While Llama2 is an improvement over LLaMA v1, it's still nowhere near even the best open models (currently, sans test contamination, WizardCoder-15B, a StarCoder fine tune is at top). It's really not a competition atm though, ChatGPT-4 wipes the floor for coding atm.
Earlier quoted context omitted.
There is one: "The agreement between GPT-4 and humans reaches 85%, which is even higher than the agreement among humans (81%). This means GPT-4’s judgments closely align with the majority of humans. We also show that GPT-4’s judgments may help humans make better judgments. During our data collection, when a human’s choice deviated from GPT-4, we presented GPT-4’s judgments to humans and ask if they are reasonable. De…
80% agreement is high, but the margins between models at the top are so low that even that remaining 20% could be enough to alter the final rankings, depending on which direction it errs.
Better evaluation paints a bit different picture: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb... *FreeWilly2 is a Llama2 70B model finetuned on an Orca style Dataset EDIT: actually, impressive: FreeWilly2 GPT-3.5 GPT-4 ARC 71.1 85.2 96.3 HellaSwag 86.4 85.5 95.3 MMLU 68.8 70.0 86.4 TruthfulQA 59.4 47.0 59.0 So reasoning (ARC) is lagging behind, but the other evaluations are at GPT-3.5 level and closi…