Does anyone know of any good test suites we can use to benchmark these local models? It would be really interesting to compare all the ones capable of running on consumer hardware so that users can easily choose the best ones to use. Currently, I'm a bit unsure how this compares to the Alpaca model released a few weeks ago.
Gpt4all: A chatbot trained on ~800k GPT-3.5-Turbo Generations based on LLaMa
191–200 of 314 posts
Re: Gpt4all: A chatbot trained on ~800k GPT-3.5-Turbo Generations based on LLaMa
#192This is a game changer! Just to clarify, how does this compare to OpenAI's GPT4 in terms of logical reasoning and whatnot?
[deleted]
Re: Gpt4all: A chatbot trained on ~800k GPT-3.5-Turbo Generations based on LLaMa
#193Earlier quoted context omitted.
That's not dystopian, that's already happened and is happening
It is dystopian when the way humans are using AI is causing inequality to deepen.
I'm sorry, this medical ai model only has a small training set and runs on limited resources. It may only provide a treatment with a 50% chance of survival. If you'd like, you can apply for a loan for our MedAI 3000 that creates custom drugs to target your child's cancer.
We're headed to Elysium and that's dystopian.
Re: Gpt4all: A chatbot trained on ~800k GPT-3.5-Turbo Generations based on LLaMa
#194Having spent quite a bit of time playing around with llama.cpp, alpaca.cpp, loras, and the many other llama-based weights lately, here is my impression: The biggest deal with this isn't the published lora adapter (which seems limited to llama 7b), but the cleaned training data, which is likely better than the previous data sets used to train the alpaca-inspired loras that have been publicly released so far. [0] If yo…
Yes, i haven't seen any fine-tuned LLaMA-65B model so far unfortunately. I guess the cost is a bit high. Perhaps with LoRa someone will do it.
Keep an eye on these guys: https://github.com/ZrrSkywalker/LLaMA-Adapter/issues/2
Re: Gpt4all: A chatbot trained on ~800k GPT-3.5-Turbo Generations based on LLaMa
#195This is a game changer! Just to clarify, how does this compare to OpenAI's GPT4 in terms of logical reasoning and whatnot?
Not even close.
Re: Gpt4all: A chatbot trained on ~800k GPT-3.5-Turbo Generations based on LLaMa
#196Does anyone know of any good test suites we can use to benchmark these local models? It would be really interesting to compare all the ones capable of running on consumer hardware so that users can easily choose the best ones to use. Currently, I'm a bit unsure how this compares to the Alpaca model released a few weeks ago.
The measure of a "good" model is still very subjective. OpenAI has used stuff like standardized test scores to compare the latest iterations of GPT, but that is only one of the many possible objective measures and might not be relevant in a lot of cases. Maybe we'll come to a consensus around such a methodology soon, or maybe it'll be something every user has to judge on their own depending on their goals.
It's even harder to measure when what you value are subjective things like creativity.
Re: Gpt4all: A chatbot trained on ~800k GPT-3.5-Turbo Generations based on LLaMa
#197Does anyone know of any good test suites we can use to benchmark these local models? It would be really interesting to compare all the ones capable of running on consumer hardware so that users can easily choose the best ones to use. Currently, I'm a bit unsure how this compares to the Alpaca model released a few weeks ago.
Test suites are not reflection complete! https://sdrinf.com/reflection-completeness -essentially, the moment a set of testing data gets significant traction, it becomes a target to optimize for. Instead, I strongly recommend to put together a list of "control questions" of your own, that covers the general, and specific use cases you're interested in. Specifically, I'd recommend adding questions on topics you have hi…
You can try holding a tournament with it vs other models if you can think of a game for them to play.
Re: Gpt4all: A chatbot trained on ~800k GPT-3.5-Turbo Generations based on LLaMa
#198Earlier quoted context omitted.
As I understand it, it's not a copyright issue, it's a terms-of-service issue. You're only authorized to use GPT in accordance with their terms, which prohibit using the service to develop a competitor. Doing so anyway is unauthorized use of the service, which is a Computer Fraud and Abuse Act violation, independent of what is and isn't copyrightable.
This is an Americanism. In many parts is Europe copyright claims over copyright from the US are honoured, but yet, violation of TOS by itself does not put you in risk of criminal lawsuit.
There are also plenty of places around the world where US copyright is not honored.
This includes, arguably, the US itself, though copyright violation is a lot more blatant in some parts of the world, which don't even offer the fig leaf of trying to enforce US copyright in their legal systems.
Re: Gpt4all: A chatbot trained on ~800k GPT-3.5-Turbo Generations based on LLaMa
#199Earlier quoted context omitted.
There is a real concern that this particular technology, as powerful as it is, will be kept behind the doors of the biggest corps and information gleamed from prompts will be further used to our collective detriment. I am personally still coming to terms with it, but it is not just wanting stuff. It is making sure society does not get too separated in terms of power ( because it is already pretty stratified ).
I agree. I think that's why it's important that the lesson isn't "if you give an inch, the community will take a mile." Otherwise, they will keep it behind closed doors.
Even if they wanted to keep it behind close doors, it could easily get leaked.
Re: Gpt4all: A chatbot trained on ~800k GPT-3.5-Turbo Generations based on LLaMa
#200Earlier quoted context omitted.
Why bother with the weights when you can just take the literal output of another model and consume as part of your training corpus? What kind of copyright claims can be made against the content generated by a model wielding the desired weights?
According to the US Copyright office's recent ruling, nothing generated by these models can be copyrighted (!), so in theory you should be fine!
They're the first word on this, not the last, so it doesn't mean much. It also clearly leads to absurd results for text models, which can memorize their training data, or echo back your inputs. Or you could pass their output through a human to copyright it.