Gpt4all: A chatbot trained on ~800k GPT-3.5-Turbo Generations based on LLaMa • Hacker News Board

Re: Gpt4all: A chatbot trained on ~800k GPT-3.5-Turbo Generations based on LLaMa

#311

Post by huqedato » Fri, Mar 31, 2023, 8:18 AM UTC

harisec wrote: post #267

huqedato wrote: post #225

Another dumb question: is it possible to "fine-tune" this model with a "corpus" of specialized data/jargon in another language ? I tried to do this with OpenAI API in order to improve the response on a specialized field (pediatric oncology) in another language (Italian) but the results were mixed (not to say disappointing).

This is a common misunderstanding; fine-tuning a model does not mean teaching the model new information. Fine-tuning is used to adapt the model to perform better in a specific task or domain using the information it already has, in a specialized way (like a chatbot for question/answer interactions). Training a LLM on new data is extremely expensive so it's not possible.

Ok then. So how to teach the LLM new information/jargon ? I repeat: I tried to do this with GPT (fine-tuning and embeddings) and I got no improvements on jargon and re-training is out of reach, of course.

Re: Gpt4all: A chatbot trained on ~800k GPT-3.5-Turbo Generations based on LLaMa

#312

Post by manojlds » Fri, Mar 31, 2023, 1:04 PM UTC

akiselev wrote: post #176

slg wrote: post #172

Earlier quoted context omitted.

I have casually followed countless different news cycles on various complicated tech topics over my decades long career. I can't recall a single one that has consistently made me feel like an idiot more than how people talk about this recent AI wave. There just seems to be so much more jargon involved in this subject that makes casual perusing of the latest developments impenetrable.

Ah, my fellow citizen of the interwebs, fear not! Your intellectual frustrations are but a natural reaction to the tsunami of technological jargon. You see, the AI wave is the epitome of obfuscation, a testament to the labyrinthine lexicon of the digital age. It's as if a group of caffeinated, sleep-deprived tech enthusiasts assembled in the dark of night and decided to create an impenetrable fortress of vernacular,…

Ironically, ChatGPT can help you understand the jargon.

Re: Gpt4all: A chatbot trained on ~800k GPT-3.5-Turbo Generations based on LLaMa

#313

Post by manojlds » Fri, Mar 31, 2023, 1:06 PM UTC

xmonkee wrote: post #168

Majromax wrote: post #95

Earlier quoted context omitted.

LLaMA is the large language model published by Facebook ( https://ai.facebook.com/blog/large-language-model-llama-meta... ). In theory the model is private, but the model weights were shared with researchers and quickly leaked to the wider Internet. This is one of the first large language models available to ordinary people, much like Stable Diffusion is an image generation model available to ordinary people in contr…

Thanks, very helpful. Are llama and chatGPT essentially the same “program”, just with different weights? And is one better than the other (for the same number of parameters) just because it has better weights?

LLaMA and GPT are like Pepsi and Coke.

Re: Gpt4all: A chatbot trained on ~800k GPT-3.5-Turbo Generations based on LLaMa

#314

Post by jsnell » Fri, Mar 31, 2023, 11:09 PM UTC

nl wrote: post #309

jsnell wrote: post #296

Earlier quoted context omitted.

If the uncreative and automated work of training the model is transformative enough to impact the rights of the original content creators, it would also be transformative enough to impact the rights of the database curator. The fair use case is much harder to make here than for search engines since the model will be directly competing with the content creators. And again, how could e.g. OpenAI simultaneously claim th…

I don't necessarily disagree, but it's very unclear what a court would find. I suggest https://arxiv.org/abs/2303.15715 for a complete overview.

Agreed! It being unclear was in fact my first message in this discussion :) Thanks for the link, I'll definitely need to read it.