Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gpt4all: A chatbot trained on ~800k GPT-3.5-Turbo Generations based on LLaMa

github.com

Post reply on HN
Print view View original

211–220 of 314 posts

Re: Gpt4all: A chatbot trained on ~800k GPT-3.5-Turbo Generations based on LLaMa

#211

Post by Grothendank »

slg wrote: post #172
tmountain wrote: post #61

Earlier quoted context omitted.

This sentence defies lay people: The biggest deal with this isn't the published lora adapter (which seems limited to llama 7b), but the cleaned training data, which is likely better than the previous data sets used to train the alpaca-inspired loras that have been publicly released so far.

I have casually followed countless different news cycles on various complicated tech topics over my decades long career. I can't recall a single one that has consistently made me feel like an idiot more than how people talk about this recent AI wave. There just seems to be so much more jargon involved in this subject that makes casual perusing of the latest developments impenetrable.

There's a neat trick when you encounter jargon.

1. Identify the jargon terms you don't understand

2. Lookup papers that introduce the jargon terms

3. Skim-read the paper to get the gist of the jargon

If you don't want to do this, then you don't have to feel uneducated. You can simply choose to feel like your time is more important than skimming a dozen AI papers a week.

But for example, here's what I did to understand the parent comment:

1. I had no idea what lora is or how it relates to alpaca.

2. I looked up https://github.com/tloen/alpaca-lora

3. I read the abstract of the Lora paper: https://arxiv.org/pdf/2106.09685.pdf https://github.com/tloen/alpaca-lora

4. Now I know that Lora is just a way of using low rank matrices to reduce finetuning difficulty by a factor of like 10,000 or something ridiculous

5. Since I don't actually care about /how/ Lora does this, that's all I need to know.

6. TLDR; Lora is a way to fine-tune models like Llama while only touching a small fraction of the weights.

You can do this with any jargon term at all. Sure, I introduced more jargon in step 4 - low rank matrices. But if you need to, you can use the same trick again to learn about those. Eventually you'll ground yourself on basic college level linear algebra, which if you don't know, again you should learn.

The sooner you evolve this "dejargonizing" instinct rather than blocking yourself when you see new jargon, the less overwhelmed and uneducated you will feel.

Re: Gpt4all: A chatbot trained on ~800k GPT-3.5-Turbo Generations based on LLaMa

#212

Post by londons_explore »

2bitencryption wrote: post #66
alchemist1e9 wrote: post #60

Earlier quoted context omitted.

I’ll ask a dumb question. On another of the numerous LLM related posts I was asking if any of the self host-able open model can do code summaries at close to the quality of GPT 3.5 turbo. I was basically told nowhere close yet. Can this potentially do that? Ideally I’d like to have it generate descriptions of large amounts of code but would rather not burn tokens and lose privacy via OpenAI api. But I’d gladly keep a…

Here's my experience, having used llama+lora 7b, 13b, and 30b, on both cpu and gpu: On gpu, processing the input prompt, even for huge prompts, is almost instant. Meaning, even if your prompt is huge, it will start generating new tokens after your prompt very quickly. On a rented A6000 gpu, using llama+lora 30b, you can use huge prompts and it will start giving a new output right away. On cpu (i.e. the project llama.…

That is likely a bug. There is nothing in the maths that should make prompt tokens slower on CPU.

Re: Gpt4all: A chatbot trained on ~800k GPT-3.5-Turbo Generations based on LLaMa

#213

Post by joshxyz »

slg wrote: post #172
tmountain wrote: post #61

Earlier quoted context omitted.

This sentence defies lay people: The biggest deal with this isn't the published lora adapter (which seems limited to llama 7b), but the cleaned training data, which is likely better than the previous data sets used to train the alpaca-inspired loras that have been publicly released so far.

I have casually followed countless different news cycles on various complicated tech topics over my decades long career. I can't recall a single one that has consistently made me feel like an idiot more than how people talk about this recent AI wave. There just seems to be so much more jargon involved in this subject that makes casual perusing of the latest developments impenetrable.

its ok you are not alone, most of us feel the same way on theiri buzwords

Re: Gpt4all: A chatbot trained on ~800k GPT-3.5-Turbo Generations based on LLaMa

#214

Post by IIAOPSW »

computerex wrote: post #208
IIAOPSW wrote: post #201

Earlier quoted context omitted.

Your wish for a lawsuit misguidedly presupposes they would lose on the grounds of fairness. Being a hypocrite isn't a crime and neither is profiting off of it. Point to any data they trained on that was neither already public domain or they paid royalties for.

I am not presuming that they'd lose. The point is that a lawsuit may bring this issue to light and establish a legal precedence. If OpenAI is allowed to use potentially dubious data to train their models to profit, then others should be allowed to do the same with their models. If OpenAI wins, and successfully manages to stop alpaca/gpt4all type datasets, then they should be counter-sued for doing the same with upstr…

You can only "establish legal precedence" when the law is ambiguous and thus in need of interpretation to begin with. Calling OpenAI's data / access thereof "dubious" is fundamentally an opinion you have about their methods, not a tangible wrong doing which you can sue over. Unless you believe they did not actually pay royalties on certain data they used (and can prove it). OpenAI has a slam dunk case no matter how hypocritical they are forced to appear in the process.

Re: Gpt4all: A chatbot trained on ~800k GPT-3.5-Turbo Generations based on LLaMa

#215

Post by Grothendank »

akiselev wrote: post #176
slg wrote: post #172

Earlier quoted context omitted.

I have casually followed countless different news cycles on various complicated tech topics over my decades long career. I can't recall a single one that has consistently made me feel like an idiot more than how people talk about this recent AI wave. There just seems to be so much more jargon involved in this subject that makes casual perusing of the latest developments impenetrable.

Ah, my fellow citizen of the interwebs, fear not! Your intellectual frustrations are but a natural reaction to the tsunami of technological jargon. You see, the AI wave is the epitome of obfuscation, a testament to the labyrinthine lexicon of the digital age. It's as if a group of caffeinated, sleep-deprived tech enthusiasts assembled in the dark of night and decided to create an impenetrable fortress of vernacular,…

Should jackasses on HN use plain language instead of jargon? Surely.

But AI workers mainly develop and use jargon because it is an easy and natural way to consolidate concepts.

Sure, there is a kind of conspiracy caused by publish or perish. Researchers may use jargon to make their work harder to reject on review; laborious speech and jargon can make statements sound more profound. However, no technical field is immune to this. We'll need to systematically change science before we can eliminate that problem.

Until we manage that, if you care about the concepts enough to want to understand them before there are good plain speech descriptions, just pop the jargon into google scholar and skim read a few papers, and you're good to go. If you don't care about the concepts that much, then don't worry about the jargon. The important concepts will get their own non-technical explanations in time.

As it stands, AI jargon is not that bad. It tends to be pretty fair and easy to understand, compared to jargon in, say, biochemistry or higher math.

Re: Gpt4all: A chatbot trained on ~800k GPT-3.5-Turbo Generations based on LLaMa

#216

Post by computerex »

IIAOPSW wrote: post #214
computerex wrote: post #208

Earlier quoted context omitted.

I am not presuming that they'd lose. The point is that a lawsuit may bring this issue to light and establish a legal precedence. If OpenAI is allowed to use potentially dubious data to train their models to profit, then others should be allowed to do the same with their models. If OpenAI wins, and successfully manages to stop alpaca/gpt4all type datasets, then they should be counter-sued for doing the same with upstr…

You can only "establish legal precedence" when the law is ambiguous and thus in need of interpretation to begin with. Calling OpenAI's data / access thereof "dubious" is fundamentally an opinion you have about their methods, not a tangible wrong doing which you can sue over. Unless you believe they did not actually pay royalties on certain data they used (and can prove it). OpenAI has a slam dunk case no matter how h…

Situation is more nuanced than you are making it seem. It's difficult to prove whether OpenAI used copyright data, because of the nature of these models. But that applies to people "abusing" OpenAI's terms as well to produce datasets for fine tuning models.

Re: Gpt4all: A chatbot trained on ~800k GPT-3.5-Turbo Generations based on LLaMa

#217

Post by nullsense »

tetete wrote: post #124
barking_biscuit wrote: post #85

Earlier quoted context omitted.

>Buying a "high end GPU" isn't buying a 4090 or even two, it's 250k on a DGX unit and putting it in a datacentre. You will probably be able to find a service that would sign a confidentiality agreement and provide you with this service for less than 250k. I haven't started to play with LLMs locally in anger yet, but I was under the impression that you could use a 4090 in combination with FlexGen to achieve this rathe…

With 8 bit training you can do ~13B pram LLM on 3090/4090. https://huggingface.co/blog/trl-peft But it is pretty cheap to rent something at vast.ai or whatever to get 40GB for a final run.

Awesome. Between crypto hype in 2017 and AI hype in 2023 I've acquired a collection of 2x 1080ti, an RTX 3060 and an RTX 4090. All together it's a total of 58GB of VRAM. Is there a way I can pool it all across a distributed cluster of 2 machines for doing anything? I'm assuming it would bottleneck on both network speeds and the slowest GPUs if it's possible at all...

Re: Gpt4all: A chatbot trained on ~800k GPT-3.5-Turbo Generations based on LLaMa

#218

Post by mszcz »

Grothendank wrote: post #211
slg wrote: post #172

Earlier quoted context omitted.

I have casually followed countless different news cycles on various complicated tech topics over my decades long career. I can't recall a single one that has consistently made me feel like an idiot more than how people talk about this recent AI wave. There just seems to be so much more jargon involved in this subject that makes casual perusing of the latest developments impenetrable.

There's a neat trick when you encounter jargon. 1. Identify the jargon terms you don't understand 2. Lookup papers that introduce the jargon terms 3. Skim-read the paper to get the gist of the jargon If you don't want to do this, then you don't have to feel uneducated. You can simply choose to feel like your time is more important than skimming a dozen AI papers a week. But for example, here's what I did to understan…

> 3. Skim-read the paper to get the gist of the jargon

Or, you know, you could ask ChatGPT to explain it to you... Granted the term was coined 2021>=. Even if it wasn't but the paper is less than 32k tokens... 0.6c for the answer doesn't seem all that steep.

edit: grammar

Re: Gpt4all: A chatbot trained on ~800k GPT-3.5-Turbo Generations based on LLaMa

#219

Post by jwestbury »

IIAOPSW wrote: post #201
computerex wrote: post #180

Earlier quoted context omitted.

I think their terms are the most hipocritical thing I have ever read and I get a visceral reaction at the entitlement they show with their terms stating that no one can use thier model output to train competing models WHEN THEY TRAINED THOSE MODELS ON THE COLLECTIVE IP OF HUMAN KIND. Screw Open AI for these terms. I hope there is a lawsuit so this gets ironed out in the court of law. It is so high and mighty for them…

Your wish for a lawsuit misguidedly presupposes they would lose on the grounds of fairness. Being a hypocrite isn't a crime and neither is profiting off of it. Point to any data they trained on that was neither already public domain or they paid royalties for.

"Public domain" does not mean "available for public consumption." If I publish a blog post, I hold the copyright on that blog post, and reproducing it is a violation of copyright, despite the fact that it was posted publicly.

Public domain means that something can be reproduced without violating copyright, because the work is effectively owned by the public.

Re: Gpt4all: A chatbot trained on ~800k GPT-3.5-Turbo Generations based on LLaMa

#220

Post by Grothendank »

joshxyz wrote: post #213
slg wrote: post #172

Earlier quoted context omitted.

I have casually followed countless different news cycles on various complicated tech topics over my decades long career. I can't recall a single one that has consistently made me feel like an idiot more than how people talk about this recent AI wave. There just seems to be so much more jargon involved in this subject that makes casual perusing of the latest developments impenetrable.

its ok you are not alone, most of us feel the same way on theiri buzwords

There's a difference between buzzwords and jargon. Buzzwords can start out as jargon, but have their technical meaning stripped by users who are just trying to sound persuasive. Examples include words like synergy, vertical, dynamic, cyber strategy, and NFT.

That's not what's happening in the parent comment. They're talking about projects like

https://github.com/ZrrSkywalker/LLaMA-Adapter

https://github.com/microsoft/LoRA

https://github.com/tloen/alpaca-lora

and specifically the paper: https://arxiv.org/pdf/2106.09685.pdf

Lora is just a way to re-train a network for less effort. Before we had to fiddle with all the weights, but with Lora we're only touching 1 in every 10,000 weights.

The parent comment says GPT4all doesn't give us a way to train the full size Llama model using the new lora technique. We'll have to build that ourselves. But it does give us a very huge and very clean dataset to work with, which will aid us in the quest to create an open source chatGPT killer.

Post reply on HN