Earlier quoted context omitted.
Wuh-wuh-wuh.
Yeah, some just skipped "http:" prefix and continued with "veh-veh-veh".
If you're german, yeah.
ChatRWKV, like ChatGPT but powered by the RWKV (RNN-based, open) language model
141–146 of 146 posts
Re: ChatRWKV, like ChatGPT but powered by the RWKV (RNN-based, open) language model
#142Earlier quoted context omitted.
> How much of this is the language vs the vast amount of passably accurate domain knowledge? LLMs don’t have domain knowledge, its all language.
That's what I meant by 'It's not clear where the language ends and the encyclopedic knowledge starts,' since the model (and perhaps our brains) make little distinction. But the model seems to be storing an absolutely vast amount of information, beyond the the capability of any individual person to accumulate and recall. This is clearly not a prerequisite for language, even if the information is represented linguistic…
> at age 20 I had read maybe 10-20% of what I've read since, but I was capable of reading comprehension and conversation...
This is because human intelligence is not just language, but lot of indirect context, "software" inside spinal cord (and other non-cortex parts of brain), and even human body itself.
But as I know, current LLMs working in plain flat structures. At the moment, nobody tried to use even neocortex-like structures, not even considered artificial spinal cord.
All these looks like, to teach table lamp, or something similar smart.
Re: ChatRWKV, like ChatGPT but powered by the RWKV (RNN-based, open) language model
#143Earlier quoted context omitted.
In 20 years I'll still be alive and enjoying myself. Does 20 years seem long to you?
That's one hell of an assumption. Many of my Russian friends were absolutely certain they'll be alive and well for at least the next 20 years not that long ago.
Sure, they will be alive if not come to Ukraine.
Re: ChatRWKV, like ChatGPT but powered by the RWKV (RNN-based, open) language model
#144So what's the level of effort to create ChatGPT equivalent products? Is it something where we'll have 100s of competing AIs, or is it gated to only a few large companies? Not up to date on current training/querying costs. Can these models feasibly be run locally? Given the large number of competitors already announced to ChatGPT, I fail to see how the space will be easily defensible or monetizable (despite large valu…
Datasets. The one with the largest, most personal, most obtrusive, invasive dataset will probably win. The one that has absorbed every podcast, every youtube video, every close-caption text in existence, will have the most "complete" answers.
Only legally gathered, absolutely "white" datasets could win, because gray/black methods of gathering lack feedback.
You have not methods to ensure, if gray/black really gather data or they faked it.
Re: ChatRWKV, like ChatGPT but powered by the RWKV (RNN-based, open) language model
#145Earlier quoted context omitted.
If it's a few minutes per token you might be waiting a lot longer for a full response: https://blog.quickchat.ai/post/tokens-entropy-question/ I also wonder how open.ai etc provides access to these for free. Reminds me of the adage from when Facebook rose to popularity: "if something is free, 'you' are the product". Perhaps to gather lots more conversational training data for fine tuning.
It would be remarkable and surprising if they weren’t doing that.
It's in their FAQ:
>> Who can view my conversations?
> As part of our commitment to safe and responsible AI, we review conversations to improve our systems and to ensure the content complies with our policies and safety requirements.
>> Will you use my conversations for training?
> Yes. Your conversations may be reviewed by our AI trainers to improve our systems.
Re: ChatRWKV, like ChatGPT but powered by the RWKV (RNN-based, open) language model
#146Earlier quoted context omitted.
Down that far, I start to wonder if trinary circuits might become useful again. fp4 with 1-3-0 would mean 27 values if the first bit were interpreted as binary. But--and an engineer should check me on this cause to me a transistor is a distant abstraction--I think you could double that to 54 values if you were clever with the sign bit and arithmetic circuitry. Maybe push it to 42 if only some of my intuition is wrong…
You're wrong on many levels. The basic reason for binary is because it's generally faster, especially as you scale to smaller transistors with more noise.
Here's how Brusentsov (who designed https://en.wikipedia.org/wiki/Setun) described the rationale for his choice of ternary:
"At that time [1955], transistors were not yet available, but it was clear that the machine should not use vacuum tubes. Tubes have a short lifespan, and tube-based machines were idle most of the time because they were always being repaired. A tube machine worked at best for several hours, then it was necessary to look for another malfunction. Yuli Izrailevich Gutenmakher built the LEM-1 machine on ferrite-diode elements. The thought occurred to me that since there are no transistors, then you can try to make a computer on these elements. Sobolev, whom everyone respected very much, arranged for me to go on an internship with Gutenmacher. I studied everything in detail. Since I am a radio engineer by education, I immediately saw that not everything should be done the way they did it. The first thing I noticed is that they use a pair of cores for each bit, one working and one compensating. And an idea came to my mind: what if we make the compensation core do work, as well? Then each cell becomes three-state. Consequently, the number of cores in Setun was seven times less than in LEM-1."
(https://notesofprogrammer.blogspot.com/2010/03/blog-post.htm...)