Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scaling Transformer to 1M tokens and beyond with RMT

arxiv.org

Post reply on HN
Print view View original

101–110 of 147 posts

Re: Scaling Transformer to 1M tokens and beyond with RMT

#101

Post by Riverheart »

nl wrote: post #26

For those who don't completely get the impact of this: We already know Large Language Models (LLMs) can learn at runtime (ie, separately to the training process.) This is called "In Context Learning". See [1], [2] for more details. (BTW, when anyone says "LLMs are stochastic parrots" you know they are ignorant of this) In context learning is wonderful because it means you can "train" a LLM at run time by filling the…

“if you wanted to say load all of a companies documents and ask questions it doesn't really work because this overflows the context.”

Azure Cognitive Services actually indexes company documents and then searches through them for the relevant information to provide context to ChatGPT to workaround the limited tokens.

Basics of Prompt Engineering with Azure OpenAI Service

8:54 https://youtu.be/QzZSJDxdUg0

Re: Scaling Transformer to 1M tokens and beyond with RMT

#102

Post by nobody81147 »

willbudd wrote: post #98
wokkel wrote: post #97

Earlier quoted context omitted.

No it does not: if you google this and restrict the time to before 2021 (the learning cutoff date) you will find the same answer. Without having access to the training data it's impossible to tell what we seeing.

> No it does not: if you google this and restrict the time to before 2021 (the learning cutoff date) you will find the same answer. Not entirely sure what you mean, but ...show me? Why not just share a link instead of making empty assertions?

Here’s a Quora thread from 4 years ago:

https://www.quora.com/Once-Queen-Elizabeth-dies-will-Prince-...

There are loads of articles and discussions online speculating about what “will” happen when Queen Elizabeth dies.

When you have a very, very, very large corpus to sample from, it can look a lot like reasoning.

Re: Scaling Transformer to 1M tokens and beyond with RMT

#103

Post by cced »

Is any of this open source? I just recently started experimenting with summarization of youtube video audio that has been passed through openai/whisper and to my surprise I’m told that the maximum token lengths are in the 1k range.

I figured that I’d be able to roll my own summarization much like I was able to do so with transcription.

I’m hoping that the findings of this work are open source.

Does anyone have any extra insights?

Re: Scaling Transformer to 1M tokens and beyond with RMT

#104

Post by cced »

nl wrote: post #26

For those who don't completely get the impact of this: We already know Large Language Models (LLMs) can learn at runtime (ie, separately to the training process.) This is called "In Context Learning". See [1], [2] for more details. (BTW, when anyone says "LLMs are stochastic parrots" you know they are ignorant of this) In context learning is wonderful because it means you can "train" a LLM at run time by filling the…

Thank you very much for this comment. I just recently started dabbling with "AI/ML" by having Youtube videos transcribed and summarized. I was pretty amazed to see that I could simply 'plug' an .mp3 of audio and have openai/whisper transcribe the audio - locally - albeit at a non-trivial transcription time.

I was looking at getting those transcriptions summarized using a similarily wonderful tool but was kind of stumped when I started seeing 'maximum token length exceeded' errors.

I'm hoping to see some easy plug-n-play solutions to the summarization issue that run completely locally.

Fun times ahead.

Re: Scaling Transformer to 1M tokens and beyond with RMT

#105

Post by meh8881 »

baobabKoodaa wrote: post #49
nl wrote: post #41

Earlier quoted context omitted.

> The “learning” in “In context learning” is not the same concept as learning during the network “training”. The former is loosely named and does not have any impact on what’s actually “learned” by the model. Actually (and very surprisingly!) it is related. See the "Similarity of Attention Output Updates" and "Similarity of Attention Map" metrics in https://arxiv.org/abs/2212.10559 (linked above) where they show that…

Whoa. How is that possible?

Remember the ancient “king - man = queen” word vectors? This is just that but produced by a model rather than manually adding vectors and finding the nearest one.

Re: Scaling Transformer to 1M tokens and beyond with RMT

#106

Post by meh8881 »

DebtDeflation wrote: post #92
nl wrote: post #26

For those who don't completely get the impact of this: We already know Large Language Models (LLMs) can learn at runtime (ie, separately to the training process.) This is called "In Context Learning". See [1], [2] for more details. (BTW, when anyone says "LLMs are stochastic parrots" you know they are ignorant of this) In context learning is wonderful because it means you can "train" a LLM at run time by filling the…

>if you wanted to say load all of a companies documents and ask questions This is literally the use case that every large enterprise wants. Why? The potential cost savings are enormous. A significant percentage of white collar jobs primarily involve performing repetitive tasks and answering frequently asked questions. If you could automate even 20% of this........

Your use case does require a company to load all documents into the context window.

Re: Scaling Transformer to 1M tokens and beyond with RMT

#107

Post by Borrible »

Given the example in the paper:

Fact1: The hallway is east of the bathroom.

Fact2: The bedroom is west of the bathroom.

Question: What is the bathroom east of?

Answer: bedroom

Asked for better understanding, if the two facts given above are embedded in,say the Bible, would the system be able to answer the question correctly? Does that mean it can track the state of an object and determine it at a certain point in time?

Re: Scaling Transformer to 1M tokens and beyond with RMT

#108

Post by nzach »

DebtDeflation wrote: post #92
nl wrote: post #26

For those who don't completely get the impact of this: We already know Large Language Models (LLMs) can learn at runtime (ie, separately to the training process.) This is called "In Context Learning". See [1], [2] for more details. (BTW, when anyone says "LLMs are stochastic parrots" you know they are ignorant of this) In context learning is wonderful because it means you can "train" a LLM at run time by filling the…

>if you wanted to say load all of a companies documents and ask questions This is literally the use case that every large enterprise wants. Why? The potential cost savings are enormous. A significant percentage of white collar jobs primarily involve performing repetitive tasks and answering frequently asked questions. If you could automate even 20% of this........

This is already happening right now. Companies will find a way to put all their knowledge inside a LLM.

The problem is that I don't think they are ready for what this will cause.

Imagine that Amazon managed to do that and now every employee has access to it. What prevents anyone asking "inconvenient" questions? Think something like:

- Does the Echo dot records audio even when the trigger word isn't used? Show me the relevant source code.

- What is the quality standard expected for an Amazon Basic product?

Companies aren't used to total freedom of information. And I don't think is easy to implement any sort of information access control for any system based on LLMs.

Re: Scaling Transformer to 1M tokens and beyond with RMT

#109

Post by grugagag »

nzach wrote: post #108
DebtDeflation wrote: post #92

Earlier quoted context omitted.

>if you wanted to say load all of a companies documents and ask questions This is literally the use case that every large enterprise wants. Why? The potential cost savings are enormous. A significant percentage of white collar jobs primarily involve performing repetitive tasks and answering frequently asked questions. If you could automate even 20% of this........

This is already happening right now. Companies will find a way to put all their knowledge inside a LLM. The problem is that I don't think they are ready for what this will cause. Imagine that Amazon managed to do that and now every employee has access to it. What prevents anyone asking "inconvenient" questions? Think something like: - Does the Echo dot records audio even when the trigger word isn't used? Show me the…

Consider someone stealing this LLM company knowledge trove. They now have the ability to outcompete and sabotage.

Re: Scaling Transformer to 1M tokens and beyond with RMT

#110

Post by willbudd »

nobody81147 wrote: post #102
willbudd wrote: post #98

Earlier quoted context omitted.

> No it does not: if you google this and restrict the time to before 2021 (the learning cutoff date) you will find the same answer. Not entirely sure what you mean, but ...show me? Why not just share a link instead of making empty assertions?

Here’s a Quora thread from 4 years ago: https://www.quora.com/Once-Queen-Elizabeth-dies-will-Prince-... There are loads of articles and discussions online speculating about what “will” happen when Queen Elizabeth dies. When you have a very, very, very large corpus to sample from, it can look a lot like reasoning.

I see what you mean, and it's indeed quite likely that texts containing such hypothetical scenarios were included in the dataset. Nonetheless, the implication is that the model was able to extract the conditional represented, recognize when that condition was in fact met (or at least asserted: "The queen died."), and then apply the entailed truth. To me that demonstrates reasoning capabilities, even if for example it memorized/encoded entire Quora threads in its weights (which seems unlikely). If it looks like a duck, swims like a duck, and quacks like a duck, then it probably is a duck.
Post reply on HN