111–120 of 137 posts
why did they do this character by character? Would word by word make sense? Other than punctuation I'm not seeing why specific characters are meaningful units.
just wondering, how many AI programs (models with complete source code) OpenAI has released?
For context, Claude Shannon found that humans could model English text with an entropy of 0.6 to 1.3 bits per character (http://languagelog.ldc.upenn.edu/myl/Shannon1950.pdf)
Earlier quoted context omitted.
isn't it impressive? They trained a network to predict the next character and when the finished they trained the last layer to predict sentiment with only few examples. The network was able to learn sentiment somehow in the first training phase without telling it! For me this open my mind to new opportunities when training deep learning. For example I can do the same for images: train a network to recognize objects a…
Immediate thought - the network didn't learn sentiment in the training phase, it just clustered the data. The last phase was so quick to train because it was labeling already-clustered data, so a few data points in each cluster was enough to make it 'obvious'.
Earlier quoted context omitted.
The technique of training a model on a lot of data for a long time and then leveraging its sophisticated representation only learning the last layer(s) on small datasets to create accurate models is common practice.
Beating the state-of-the-art with one-shot learning is not common. Transfer learning for NLP is also quite unchartered. Also the technique is quite novel: This is not pre-trained nets on labeled data, it is an unsupervised generative model. Future research directions are exciting: Unsupervised prediction of the next frame in a video, and then being able to one-shot learn a wide range of visual tasks.
[0] https://www.youtube.com/watch?v=-yX1SYeDHbg&list=PLE6Wd9FR--...
Earlier quoted context omitted.
Is it wrong to be critical of research? Back in my previous life of doing basic research I scrutinized papers left and right. http://karpathy.github.io/2015/05/21/rnn-effectiveness/ towards the end has similar methodology and is 1.5 years old. Hype is an interesting thing especially when it comes from laymen.
As someone familiar with the field, you likely know this already, but the similarities between the Karpathy post from 2015 and this work from OpenAI is likely because Karpathy is a founder and lead researcher at OpenAI.
Earlier quoted context omitted.
Beating the state-of-the-art with one-shot learning is not common. Transfer learning for NLP is also quite unchartered. Also the technique is quite novel: This is not pre-trained nets on labeled data, it is an unsupervised generative model. Future research directions are exciting: Unsupervised prediction of the next frame in a video, and then being able to one-shot learn a wide range of visual tasks.
You might be interested in minute 50 onward [0], or this recent paper from Facebook [1]. [0] https://www.youtube.com/watch?v=-yX1SYeDHbg&list=PLE6Wd9FR--... [1] https://arxiv.org/abs/1703.07684
[1] https://arxiv.org/abs/1412.6056 "Predicting Deeper into the Future of Semantic Segmentation"
Is this linear combination between 2 different strings?
I'm trying to understand this statement: "The sentiment neuron within our model can classify reviews as negative or positive, even though the model is trained only to predict the next character in the text." If you look closely at the colorized paragraph in their paper/website, you can see that the major sentiment jumps (e.g. from green to light-green and from light-orangish to red) occur with period characters. Perh…
This seems to have to do with a pretty deep understanding of grammar; the model waits until it the low-level neurons have something to pass up (decoding of a complete unit of meaning) before using that to update its sentiment neuron.
A lot of next-character or next-word prediction ends up working like this - internally, the model keeps some state and makes big changes to its understanding at points that have to do with the structure of the stream.
Why are people being so critical about this work? Sure, the blog post provides a simplified picture about what the system is actually capable of, but it's still helpful for a non-ML audience to get a better understanding of the high-level motivation behind the work. The OpenAI folks are trying to educate the broader public as well, not just ML/AI researchers. Imagine if this discovery were made by some undergraduate…
I personally think they did great. They targeted the blog post at a more general audience so most people can follow and get an overhead view of the idea, and then put two giant buttons for "View code" and "Read paper" right at the top of the blog post for those who want more technical writing and working code.
We were very surprised that our model learned an interpretable feature, and that simply predicting the next character in Amazon reviews resulted in discovering the concept of sentiment. We believe the phenomenon is not specific to our model, but is instead a general property of certain large neural networks that are trained to predict the next step or dimension in their inputs
I think it says something very interesting about human language and information processing in general.