31–40 of 146 posts
Earlier quoted context omitted.
> RNNs famously have infinite context in theory and very short context in reality. any sources to read more about this please? its the first ive heard of it
I am not sure this article will answer your question, but Karpathy has an article about RNNs. https://karpathy.github.io/2015/05/21/rnn-effectiveness
Earlier quoted context omitted.
This is a remarkably good take that just didn't dawn on me until I read your comment. Even if ChatGPT had a lesser quality than the current iteration, the fact that they had a way for anyone to easily interact with it really was a homerun, snd can be for any software, really.
My family told me that ChatGPT came up from the pulpit AT CHURCH Me? I made a few comments like a scared luddite when ChatGPT solved two of my outstanding engineering problems instantly. I got better. But this is exactly right. The world in general now knows about AI and ML. It’s a pivot point. When something scares a seasoned engineer for a minute, and anyone can now make use of this… write it down in your diary as…
THe RWKV model seems really cool. If you could get transformer-like performance with an RNN, the “hard coded” context length problem might go away. (That said, RNNs famously have infinite context in theory and very short context in reality.) Is there a primer for what RWKV does differently? According to the Github page it seems the key is multiple channels of state with different decaying rates, giving I assume, a co…
THe RWKV model seems really cool. If you could get transformer-like performance with an RNN, the “hard coded” context length problem might go away. (That said, RNNs famously have infinite context in theory and very short context in reality.) Is there a primer for what RWKV does differently? According to the Github page it seems the key is multiple channels of state with different decaying rates, giving I assume, a co…
> RNNs famously have infinite context in theory and very short context in reality. any sources to read more about this please? its the first ive heard of it
Earlier quoted context omitted.
I am not sure this article will answer your question, but Karpathy has an article about RNNs. https://karpathy.github.io/2015/05/21/rnn-effectiveness
it doesnt touch on the "infinite context in theory and very short context in reality" piece which is what i was asking about
The overall trajectory looks ok, so I’ll be more rigorously investigating whether it’s possible to squeeze more precise context out of it. For example, since the first frame is sharp, you could discard the other future frames and use that first frame as the last history entry (rolling completion window). If “the first frame is always sharp” is true, then it seems reasonable that you can generate N sharp frames with that technique, which might work better than predicting N all at once.
Turns out it does not matter if you have transformer/MLP/lstm or whatever, as long as there are enough parameters and training epochs over large dataset things "just work"
In general RNNs cannot handle long term dependencies (ie, long pieces of text) because the gradient vanishes. It's unclear how this solves this problem although they do reference the "attention free transformer" paper: https://arxiv.org/abs/2105.14103
Earlier quoted context omitted.
My family told me that ChatGPT came up from the pulpit AT CHURCH Me? I made a few comments like a scared luddite when ChatGPT solved two of my outstanding engineering problems instantly. I got better. But this is exactly right. The world in general now knows about AI and ML. It’s a pivot point. When something scares a seasoned engineer for a minute, and anyone can now make use of this… write it down in your diary as…
I wonder what kind of engineering problems you’ve talked with ChatGPT. Because its limited understanding of spatial problems, I found it hard to use it for mechanical engineering. How did you do it?
Is it something where we'll have 100s of competing AIs, or is it gated to only a few large companies? Not up to date on current training/querying costs.
Can these models feasibly be run locally?
Given the large number of competitors already announced to ChatGPT, I fail to see how the space will be easily defensible or monetizable (despite large value add, competitors can easily undercut eachother)
The readme does not seem to be geared towards people not familiar with the topic. My questions: - Is this on the run on consumer GPU scale, or run on 8 A100 scale or you can’t run it yourself ever scale? - How does it compare to other language models in quality/abilities? - What is the training data?
The large model weights are 14B, so at 16 bits per weight, it won't quite fit on one 3090 or 4090.
So what's the level of effort to create ChatGPT equivalent products? Is it something where we'll have 100s of competing AIs, or is it gated to only a few large companies? Not up to date on current training/querying costs. Can these models feasibly be run locally? Given the large number of competitors already announced to ChatGPT, I fail to see how the space will be easily defensible or monetizable (despite large valu…