Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scaling Transformer to 1M tokens and beyond with RMT

arxiv.org

Post reply on HN
Print view View original

121–130 of 147 posts

Re: Scaling Transformer to 1M tokens and beyond with RMT

#121

Post by geraneum »

nullsense wrote: post #83
kmac_ wrote: post #70

Earlier quoted context omitted.

I verified that it has all required facts (line of succession, current circumstances). I managed to get the right answer when got everything in context, but it failed again when all three abdicate (same context). Prince Harry was indicated once. I tested GPT a lot in other domains, what I found that as long the information explicitly exists (connection between facts) then the responses are fine. I assume that if GPT…

>Nothing like that happened yet. Feels like we're only one paper away now that the context window has absolutely ballooned.

This reminded me of "Two Minute Papers" YouTube channel where in most of the videos he always, "Two papers down the line and...". I think ML/AI is the main topic of his videos. Interesting stuff.

Re: Scaling Transformer to 1M tokens and beyond with RMT

#122

Post by bratao »

cs702 wrote: post #112

Here's a list of tools for scaling up transformer context that have github repos: * FlashAttention: In my experience, the current best solution for n² attention, but it's very hard to scale up beyond the low tens of thousands of tokens. Memory use is O(n) but compute is O(n²). Code: https://github.com/HazyResearch/flash-attention * Heinsen Routing: In my experience, the current best solution for n×m attention, i.e.,…

Very nice list. I didn´t knew about Heinsen Routing, looks very interesting.

From my tests, SSMs are a very promising line of research and on my (small) tests on S4, it really has better characteristics than transformers, as it learned faster, a larger context and with smaller dataset.

Re: Scaling Transformer to 1M tokens and beyond with RMT

#123

Post by [deleted] »

grugagag wrote: post #109
nzach wrote: post #108

Earlier quoted context omitted.

This is already happening right now. Companies will find a way to put all their knowledge inside a LLM. The problem is that I don't think they are ready for what this will cause. Imagine that Amazon managed to do that and now every employee has access to it. What prevents anyone asking "inconvenient" questions? Think something like: - Does the Echo dot records audio even when the trigger word isn't used? Show me the…

Consider someone stealing this LLM company knowledge trove. They now have the ability to outcompete and sabotage.

[deleted]

Re: Scaling Transformer to 1M tokens and beyond with RMT

#124

Post by ca_tech »

cced wrote: post #104
nl wrote: post #26

For those who don't completely get the impact of this: We already know Large Language Models (LLMs) can learn at runtime (ie, separately to the training process.) This is called "In Context Learning". See [1], [2] for more details. (BTW, when anyone says "LLMs are stochastic parrots" you know they are ignorant of this) In context learning is wonderful because it means you can "train" a LLM at run time by filling the…

Thank you very much for this comment. I just recently started dabbling with "AI/ML" by having Youtube videos transcribed and summarized. I was pretty amazed to see that I could simply 'plug' an .mp3 of audio and have openai/whisper transcribe the audio - locally - albeit at a non-trivial transcription time. I was looking at getting those transcriptions summarized using a similarily wonderful tool but was kind of stum…

You can work around this by chunking the data using a sliding window. Take the first X amount of sentences that fit into the token length and summarize. Now slide your window of sentences so that you still overlap just a bit with your previous selection. Summarize that information. Continue until you have a bunch of summaries for your text. You can then pass that back into the model for a more concise summary of the summaries.

Re: Scaling Transformer to 1M tokens and beyond with RMT

#125

Post by cs702 »

bratao wrote: post #122
cs702 wrote: post #112

Here's a list of tools for scaling up transformer context that have github repos: * FlashAttention: In my experience, the current best solution for n² attention, but it's very hard to scale up beyond the low tens of thousands of tokens. Memory use is O(n) but compute is O(n²). Code: https://github.com/HazyResearch/flash-attention * Heinsen Routing: In my experience, the current best solution for n×m attention, i.e.,…

Very nice list. I didn´t knew about Heinsen Routing, looks very interesting. From my tests, SSMs are a very promising line of research and on my (small) tests on S4, it really has better characteristics than transformers, as it learned faster, a larger context and with smaller dataset.

Agree on SSMs: they look promising. They're on my list of "things to explore more thoroughly." I've done very little with them so far. I'm still making my way through the related papers, trying to get a superficial but accurate/intuitive understanding of these models.

Re: Scaling Transformer to 1M tokens and beyond with RMT

#126

Post by cced »

ca_tech wrote: post #124
cced wrote: post #104

Earlier quoted context omitted.

Thank you very much for this comment. I just recently started dabbling with "AI/ML" by having Youtube videos transcribed and summarized. I was pretty amazed to see that I could simply 'plug' an .mp3 of audio and have openai/whisper transcribe the audio - locally - albeit at a non-trivial transcription time. I was looking at getting those transcriptions summarized using a similarily wonderful tool but was kind of stum…

You can work around this by chunking the data using a sliding window. Take the first X amount of sentences that fit into the token length and summarize. Now slide your window of sentences so that you still overlap just a bit with your previous selection. Summarize that information. Continue until you have a bunch of summaries for your text. You can then pass that back into the model for a more concise summary of the…

I’ll be exploring a combination of pre-processing such as stemming in order to also reduce the token length while preserving as much important information.

Thanks for the feedback.

Edit: Won’t the sliding window solution above introduce a sort of bias?

For instance, with a sliding window of 3 units, unit 1 is captured once whereas units 2 and 3 are captured twice and three times respectively.

Re: Scaling Transformer to 1M tokens and beyond with RMT

#127

Post by kgwgk »

baobabKoodaa wrote: post #49
nl wrote: post #41

Earlier quoted context omitted.

> The “learning” in “In context learning” is not the same concept as learning during the network “training”. The former is loosely named and does not have any impact on what’s actually “learned” by the model. Actually (and very surprisingly!) it is related. See the "Similarity of Attention Output Updates" and "Similarity of Attention Map" metrics in https://arxiv.org/abs/2212.10559 (linked above) where they show that…

Whoa. How is that possible?

Depending on what one understands by “new passes through the model” it may not be.

Re: Scaling Transformer to 1M tokens and beyond with RMT

#128

Post by Workaccount2 »

nzach wrote: post #108
DebtDeflation wrote: post #92

Earlier quoted context omitted.

>if you wanted to say load all of a companies documents and ask questions This is literally the use case that every large enterprise wants. Why? The potential cost savings are enormous. A significant percentage of white collar jobs primarily involve performing repetitive tasks and answering frequently asked questions. If you could automate even 20% of this........

This is already happening right now. Companies will find a way to put all their knowledge inside a LLM. The problem is that I don't think they are ready for what this will cause. Imagine that Amazon managed to do that and now every employee has access to it. What prevents anyone asking "inconvenient" questions? Think something like: - Does the Echo dot records audio even when the trigger word isn't used? Show me the…

You're assuming the plebs will have full access.

There will likely be compartmentalized models, and then the master model that the execs get. No reason you can't have 10, 20 500 different models running.

Re: Scaling Transformer to 1M tokens and beyond with RMT

#129

Post by Robotbeat »

geraneum wrote: post #37
nl wrote: post #26

For those who don't completely get the impact of this: We already know Large Language Models (LLMs) can learn at runtime (ie, separately to the training process.) This is called "In Context Learning". See [1], [2] for more details. (BTW, when anyone says "LLMs are stochastic parrots" you know they are ignorant of this) In context learning is wonderful because it means you can "train" a LLM at run time by filling the…

> In context learning is wonderful because it means you can "train" a LLM at run time The “learning” in “In context learning” is not the same concept as learning during the network “training”. The former is loosely named and does not have any impact on what’s actually “learned” by the model. The LLM can, by nature, contradict its training data and whatever you/it prompt(s), and incorporates randomness when completing…

It’s not literally the same as the base model training, but you ARE literally training feed forward networks when you do this in the prompt. That’s how transformers work.
Post reply on HN