Fun hack. If anything, it highlights how compelling deep learning and RNNs are: no messing with NLP, no messing with building other features or adding up classifiers, etc. The manual feature engineering means it might work better on a smaller dataset, but even then probably not. For comparison with Andrej Karpathy's RNN code ( http://karpathy.github.io/2015/05/21/rnn-effectiveness/ ) training on the "HarryPotter(xxla…
I just can't agree that a simple, linear-time operation like "tokenizing words and basic n-gram models out of them" is a tedious problem like you seem to be implying, nor do I feel a solution to this very-solved problem is "compelling". Word tokenization and n-gram models are simple, unreasonably effective, and very fast. If character-based RNNs do better (albeit far more slowly during training), great, but nothing t…
Deep RNNs are simple and produce good results for a huge, diverse range of problems with no new domain information. As Andrej Karpathy wrote:
> Sometimes the ratio of how simple your model is to the quality of the results you get out of it blows past your expectations, and this was one of those times.
N-grams don't have nearly the power (eg longer-than-N-range structure like grammar) and don't generalize nearly as well, making them a lot less surprising.