Another roadblock to AGI starting to fall. WOW.
[dead]
Scaling Transformer to 1M tokens and beyond with RMT
11–20 of 147 posts
Re: Scaling Transformer to 1M tokens and beyond with RMT
#12An AI can theoretically ingest 1M token of data, analyze, summarize, indexing, and storing the condensed information in a hard drive. Then later on retrieve it based on the metadata for specific situations that needs it. At millions of token, it is basically a superhuman that can learn new things on the fly at the speed unimaginable to any organic based life. We are so close to something amazing, and scary.
Do we yet have a single example of a transformer based AI, i.e. LLM, learning something new that we didn’t teach it in the training data?
Maybe we do.
I’m not sure how to define it, but new should be a discovery or insight or even relationship that is not explicitly taught in the training data.
If we don’t then it suggests without humans in the loop that the super intelligence is not so close?
Re: Scaling Transformer to 1M tokens and beyond with RMT
#13Wow! I don't know how accuracy translates, I do see charts that look strong but unless I'm missing something, this is incredible. Would be curious and also terrified to see an endpoint so I can play around with it. I thought we were stopping this kind of research?
> I thought we were stopping this kind of research?
I’m going to, hopefully correctly, assume that was a funny joke.
Re: Scaling Transformer to 1M tokens and beyond with RMT
#14Wow! I don't know how accuracy translates, I do see charts that look strong but unless I'm missing something, this is incredible. Would be curious and also terrified to see an endpoint so I can play around with it. I thought we were stopping this kind of research?
> I thought we were stopping this kind of research? I’m going to, hopefully correctly, assume that was a funny joke.
I’m just saying, it was more a hopeful nod in the direction of wishful thinking. Not a joke, just a bit of having my head in the sand.
Re: Scaling Transformer to 1M tokens and beyond with RMT
#15Wow! I don't know how accuracy translates, I do see charts that look strong but unless I'm missing something, this is incredible. Would be curious and also terrified to see an endpoint so I can play around with it. I thought we were stopping this kind of research?
[deleted]
Re: Scaling Transformer to 1M tokens and beyond with RMT
#16An AI can theoretically ingest 1M token of data, analyze, summarize, indexing, and storing the condensed information in a hard drive. Then later on retrieve it based on the metadata for specific situations that needs it. At millions of token, it is basically a superhuman that can learn new things on the fly at the speed unimaginable to any organic based life. We are so close to something amazing, and scary.
Do we yet have a single example of a transformer based AI, i.e. LLM, learning something new that we didn’t teach it in the training data? Maybe we do. I’m not sure how to define it, but new should be a discovery or insight or even relationship that is not explicitly taught in the training data. If we don’t then it suggests without humans in the loop that the super intelligence is not so close?
GPT can learn things "in context" E.g. you can teach it something by chatting with it, but it will eventually forget it after its context length is exceeded. It cannot continuously learn and remember like biological organisms since its weights are frozen.
Re: Scaling Transformer to 1M tokens and beyond with RMT
#17Earlier quoted context omitted.
> I thought we were stopping this kind of research? I’m going to, hopefully correctly, assume that was a funny joke.
I’m just saying, it was more a hopeful nod in the direction of wishful thinking. Not a joke, just a bit of having my head in the sand.
> I’m just saying, it was more a hopeful nod in the direction of wishful thinking.
I still don’t understand why this is even wishful thinking, I don’t understand what the point of delaying progress is.
Re: Scaling Transformer to 1M tokens and beyond with RMT
#18Earlier quoted context omitted.
Do we yet have a single example of a transformer based AI, i.e. LLM, learning something new that we didn’t teach it in the training data? Maybe we do. I’m not sure how to define it, but new should be a discovery or insight or even relationship that is not explicitly taught in the training data. If we don’t then it suggests without humans in the loop that the super intelligence is not so close?
GPT can learn things "in context" E.g. you can teach it something by chatting with it, but it will eventually forget it after its context length is exceeded. It cannot continuously learn and remember like biological organisms since its weights are frozen.
So we still have catastrophic forgetting, just on a slightly longer scale.
Re: Scaling Transformer to 1M tokens and beyond with RMT
#19Earlier quoted context omitted.
Do we yet have a single example of a transformer based AI, i.e. LLM, learning something new that we didn’t teach it in the training data? Maybe we do. I’m not sure how to define it, but new should be a discovery or insight or even relationship that is not explicitly taught in the training data. If we don’t then it suggests without humans in the loop that the super intelligence is not so close?
GPT can learn things "in context" E.g. you can teach it something by chatting with it, but it will eventually forget it after its context length is exceeded. It cannot continuously learn and remember like biological organisms since its weights are frozen.
Except it cannot conceptualize reasonably. It is utterly incapable of symbolic thinking in a general sense.
Re: Scaling Transformer to 1M tokens and beyond with RMT
#20Earlier quoted context omitted.
GPT can learn things "in context" E.g. you can teach it something by chatting with it, but it will eventually forget it after its context length is exceeded. It cannot continuously learn and remember like biological organisms since its weights are frozen.
Except it cannot conceptualize reasonably. It is utterly incapable of symbolic thinking in a general sense.
It’s incapable of thinking. But what it’s surprisingly good at is symbolic reasoning similar to human common sense.
Most people forget that there is no memory or thought independent of whatever the output it generates.
But when you ask it to follow a chain of thought and generate it in the output, the eventual conclusions it lands on is scarily human.
People asking ChatGPT for word counts and sighing at how wrong it is. Yeah of course, it has no counter variables to hold numbers and increment them. It has no memory.
But when you ask it to generate a numbered list of all the words in a passage and then output the word count, it gets it right every time. Because you basically gave it a memory by encoding the counter in the generated output.