51–60 of 146 posts
Earlier quoted context omitted.
> RNNs famously have infinite context in theory and very short context in reality. any sources to read more about this please? its the first ive heard of it
Read about "RNN Vanishing Gradients". LSTMs help here, but see eg https://medium.com/analytics-vidhya/why-are-lstms-struggling... for the problems there.
Earlier quoted context omitted.
Two or three top GPUs? Thats basically nothing for a professinal project or even an investeded hobby
[flagged]
eg. https://github.com/fauxpilot/fauxpilot/blob/main/setup.sh#L5...
/shrug
If you don't want to run it locally, don't.
If the hardware you have doesn't support it, don't.
...but to say it's not possible is, again, bluntly, wrong.
THe RWKV model seems really cool. If you could get transformer-like performance with an RNN, the “hard coded” context length problem might go away. (That said, RNNs famously have infinite context in theory and very short context in reality.) Is there a primer for what RWKV does differently? According to the Github page it seems the key is multiple channels of state with different decaying rates, giving I assume, a co…
> RNNs famously have infinite context in theory and very short context in reality. any sources to read more about this please? its the first ive heard of it
Earlier quoted context omitted.
[flagged]
You details of what is required can be found per project. eg. https://github.com/fauxpilot/fauxpilot/blob/main/setup.sh#L5... /shrug If you don't want to run it locally, don't. If the hardware you have doesn't support it, don't. ...but to say it's not possible is, again, bluntly, wrong.
Earlier quoted context omitted.
Well yes, having no context memory, being slightly worse and requiring either a monster rig to run or paying per prompt made it completely and utterly irrelevant. Even now that it's improved and free to use its actual practical usability is marginal at best given the rate of blatantly wrong info being spewed with 105% confidence at the moment.
> blatantly wrong info being spewed with 105% confidence There are some approaches. For example in this paper they say truth has a certain logical consistency that is lacking in hallucinations and deception. So they find this latent direction that indicates truth in a frozen LLM. This actually works better than asking the model to self evaluate by text generation, or training with RLHF. "Discovering Latent Knowledge…
It makes you wonder what other abstract concepts current models may have had to learn to get as good as they are. If they're doing a good job of modelling when someone is speaking the truth, then what else have they learnt about us?
How complete of a "world model" can you learn purely in a passive way by consuming whatever online text is available to train on, or maybe by consuming all existent written material were it to be digitized? At some point I'm sure you need to be able to interact with the world to test hypothesis etc, but how far can predictive "intelligence" go without that?
Earlier quoted context omitted.
[flagged]
You details of what is required can be found per project. eg. https://github.com/fauxpilot/fauxpilot/blob/main/setup.sh#L5... /shrug If you don't want to run it locally, don't. If the hardware you have doesn't support it, don't. ...but to say it's not possible is, again, bluntly, wrong.
Name rolls of the tongue
So what's the level of effort to create ChatGPT equivalent products? Is it something where we'll have 100s of competing AIs, or is it gated to only a few large companies? Not up to date on current training/querying costs. Can these models feasibly be run locally? Given the large number of competitors already announced to ChatGPT, I fail to see how the space will be easily defensible or monetizable (despite large valu…
> Can these models feasibly be run locally? Bluntly, no. The models which are small enough to run locally perform so badly it’s not worth bothering. To run inference on the large models the perform decently you need the equivalent of two or three top end graphics cards. If you're serious about looking into it now, consider looking at this project that lets you run a bunch of independent machines as a cluster for infe…