Your implementation of momentum seems off, you just add a multiple of last error, instead of adding exponentially declining contributions from the past. I think you want double dW = alpha_ * val_[l][j] * D_[l+1][i] + beta_ * dW_[l+1][i][j]; W_[l+1][i][j] += dW; If you want to get an output class probability, softmax is the standard way. Minimize KL-divergence instead of squared error. You don't seem to be doing any r…
I think we used what is described in Artificial Intelligence: A Modern Approach... But I have to check because what you propose seems better.
> If you want to get an output class probability, softmax is the standard way. Minimize KL-divergence instead of squared error.
Thanks! We'll totally try that.
> You don't seem to be doing any regularization. It could maybe give you better generalization.
Thanks again. Someone mentioned that before as well. We'll have to experiment with that as well.
> Training on multiple threads and averaging is a nice touch. It would be interesting to hear if (how much) it improved your results.
Training was much faster and therefore tractable on a much larger set but we didn't manage to get our best results using this multi-threaded approach unfortunately as described in the post.
Maybe with a bigger training set we could have reach better results using multi-threaded training. That being said, the averaging phase disrupts a lot the overall backpropagation process, so I don't know how efficient it can be... Some advanced experimentation would probably be interesting here.