61–70 of 75 posts
On a first quick pass, this looks so good that I'm wondering if it's too good to be true ! But the work looks to be of decent quality and the technique is remarkably straightforward: The idea is to apply attention over the first token and a sliding context window, ignoring everything in-between, in each layer. By implication, each layer must be gradually shifting relevant information forward in the sequence, enabling…
Earlier quoted context omitted.
One such project is RWKV[1]. On the open source leaderboard it lived in the middle of the board for a while, so it really is a legit approach, it's just not hot. [1]: https://huggingface.co/blog/rwkv
side note - do you think the open source leaderboard is a fair representation of the diversity of OSS models?
What's really needed is a leaderboard based on some private test sets, where you can submit a model to be judged and some entity will run it on their own machines without disclosing the tests to you. Even that could be vulnerable to cheating if the submission process is automated and you can submit many times, as you could use the score as feedback to disclose information about the contents of the private test set. So it would take some care to run such a service, even beyond normal security concerns like securing the sandbox that submitted code runs in, etc.
Earlier quoted context omitted.
side note - do you think the open source leaderboard is a fair representation of the diversity of OSS models?
It's the best we have for large scale comparison I think. But the major problem is that the tests are public and you can easily cheat by including them in your training set and increase your score an arbitrary amount. You might even be able to do it in a way that would be difficult to detect. And it can happen by accident as well. There are already many cases of intentional and disclosed contamination in the leaderbo…
How do any of these sliding window techniques handle instructions that are non expected and only show up at the end? For example imagine feeding a book to the model and the last sentence being the instruction “return the count of the letter m in the previous input”. A human would handle this by first letting out an exasperated sigh but then restarting the reading while counting. An LLM has no ability to loop back and…
With the needed preprocessing, a LLM that can "go and do some research to adequately respond" could be extremely powerful.
We've spent the last ~10 millennia improving knowledge management technology to scale beyond the capacity/time of individual brains. Let the language model use actual research on this and pre-digest, not just Bing search. No need for it's short term memory to remember what say piece of code did something, just tag it when reading and rely on scalable shared indexing of tags.
Though the more I think about it, the more it sounds like normal LLM pretraining with the knowledge index being the giant chunk of LLM weights.
So I can let llama2 summarize books now or are there any non-obvious caveats to this approach?
https://huggingface.co/Tostino/Inkbot-13B-8k-0.2
Just chunk your document up, and pass in the prior summary along with this chunk of text, you can mention that it is chunk X of Y if you want (which can help with how it starts the summary often).
Earlier quoted context omitted.
It's the best we have for large scale comparison I think. But the major problem is that the tests are public and you can easily cheat by including them in your training set and increase your score an arbitrary amount. You might even be able to do it in a way that would be difficult to detect. And it can happen by accident as well. There are already many cases of intentional and disclosed contamination in the leaderbo…
Who would you trust (and verify) to run such a leaderboard?
How do any of these sliding window techniques handle instructions that are non expected and only show up at the end? For example imagine feeding a book to the model and the last sentence being the instruction “return the count of the letter m in the previous input”. A human would handle this by first letting out an exasperated sigh but then restarting the reading while counting. An LLM has no ability to loop back and…
Is it so hard to ask the user to put instructions at the beginning? Claude 100K asks users to put instructions at the end. Or you just use a quick model to check if there area instructions at the end and bring it to the beginning.
That allows you to do things like chunk the document while leaving the rest of their instruction alone, or do a sliding window of just the document while your instruction stays static.
The authors just uploaded a FAQ section, which may clarify some of the confusions: https://github.com/mit-han-lab/streaming-llm/blob/main/READM...
Nice update. I think the key question they added that clarifies a lot is #3 (quoted below) Can I input an extensive text, like a book, into StreamingLLM for summarization? While you can input a lengthy text, the model will only recognize the latest tokens. Thus, if a book is an input, StreamingLLM might only summarize the concluding paragraphs, which might not be very insightful. As emphasized earlier, we neither exp…