131–140 of 143 posts
Earlier quoted context omitted.
You're not at all wrong :) I think a lot of people confuse the pre-training and fine-tuning runs because these are all novel concepts.
Llama: https://arxiv.org/pdf/2302.13971.pdf You can clearly see in table on second page that higher quality data is trained on more than 1 epoch. Most Open LLM's do this.
Probably unrelated, but I tried to get ChatGPT to write me some code to programmatically control the details of a column filter in an Excel spreadsheet in PowerShell. Nothing it tried worked, it got close, but it didn't work. Finally I found some C# code that fixed the problem, and I pasted that code into ChatGPT, asked it to read it, and then fix the problem in PowerShell. It said it understood the solution, updated…
Interesting anecdote. I think there's a common theme with current LLMs, that people focus unreasonably much on "knowledge retrieval" from the models (1) and under-hype and under-appreciate the "language model" part. These things are really easy to anthropomize, partly because they are good at "talking" and "articulating". So good that we tend to just accept that magical, enormous feat of statistical engineering as a…
Earlier quoted context omitted.
But if you have say 50bn weights, and you run backprop, you are going to update most of the weights (except the dropout ones, but which ones drop out changes on every token I think). This means you need 50bn deltas. It might compress, but if you do then you need extra compute to do that.
You would do dropout on every epoch of training, not on every token .
Earlier quoted context omitted.
I have been training a natural intelligence model for 3 years now and she still doesn’t get nuance. Things are either good or bad in her book: nothing in between. My plan is to let her train with binary good/bad labels till the age of 5 and then start smoothing the labels after that. Wonder if that works for your AI.
in my mind I've built an 'emotional engine' to add nuance to models understanding, take something like Plutchik's wheel of emotions and create a high quality multi-modal dataset based on that structure, given our current technology takes inspiration from the brain, it would seem like having discrete models specialising in particular aspects of 'intelligence' that are then organised into a mixture of experts is an int…
Earlier quoted context omitted.
Llama: https://arxiv.org/pdf/2302.13971.pdf You can clearly see in table on second page that higher quality data is trained on more than 1 epoch. Most Open LLM's do this.
Llama2 doesn’t and outclasses llama. I believe GPT was trained in same manner
If you are talking about GPT-4, unless you are insider (doubt it) you'd have no way of proving it either way because that info is not public.
> Llama2 doesn’t and outclasses llama.
My point still stands, llama2 is just one llm, and we still don't know the distribution of their training set.
Earlier quoted context omitted.
in my mind I've built an 'emotional engine' to add nuance to models understanding, take something like Plutchik's wheel of emotions and create a high quality multi-modal dataset based on that structure, given our current technology takes inspiration from the brain, it would seem like having discrete models specialising in particular aspects of 'intelligence' that are then organised into a mixture of experts is an int…
That seems similar to https://en.m.wikipedia.org/wiki/Internal_Family_Systems_Mode...
Earlier quoted context omitted.
in my mind I've built an 'emotional engine' to add nuance to models understanding, take something like Plutchik's wheel of emotions and create a high quality multi-modal dataset based on that structure, given our current technology takes inspiration from the brain, it would seem like having discrete models specialising in particular aspects of 'intelligence' that are then organised into a mixture of experts is an int…
I have code stubbed out for this in mitta.us. It has 9 states, based on the Plutchik wheel, with emojis for the states. States drive temp and a few other things and drop the state into prompts.
Earlier quoted context omitted.
Many neurons are polysynthactic, that makes interventions like the proposed difficult.
Is that necessarily the case for massive LLMs, or could there be a more refined grouping?
From a physics perspective it’s entropy: There are just more local minima that have neurons code multiple things.
I suspect that dropout and similar tricks also play a role in this: Removing connections during training means that pathways need to be redundant somewhat.
Earlier quoted context omitted.
We wouldn't know how to construct those matrices because we don't know where in the layers what knowledge is represented. One thing that helps a little bit is freezing the lower layers, so at least the model won't forget its most fundamental knowledge. Note that the only reason that things are catastrophically forgotten, is that the original examples are not shown again. If the model learns in a single shot, there mi…
where in the layers what knowledge is represented. " This seems like a ripe angle for evolvement of our understanding of AIs use in LLMs... can we throw AIs at AIs (is AI synonymous to LLM?) Can we throw LLMs at LLMs? and have them recursively learn from themselves.. or is it a Rat King. AI recognize AI in the GangPlane
In any case, looking at and understanding how a neural network encodes information is like gene editing. Perhaps you could isolate a gene in the human genome that achieves something interesting like giving a child blue eyes. But even if you would do that, there's a chance you break something else if you modify that gene and give the child health risk. Since all neurons in a deep neural network are interconnected, there is a butterfly effect in it that makes them inherently somewhat of a black box.
I see similar loss curves when training ViTs (from scratch), which has always bothered me but I had bigger concerns so never delved too deep into it. The only difference is that I see the training loss go _up_ during each epoch. The cliffs between epochs are large enough that training loss goes down overall and validation loss keeps going down the whole time as well. The model gets close-ish to SoTA so I guess it's "…