Implementing Neural Turing Machines
arxiv.org
Implementing Neural Turing Machines
1–10 of 16 posts
Re: Implementing Neural Turing Machines
#2“how the memory contents of a NTM are initialized may be a defining factor in the success of a NTM implementation”
Does it mean that one can expect better explainability in the future from these models?
Re: Implementing Neural Turing Machines
#3The associated github repo is at https://github.com/MarkPKCollier/NeuralTuringMachine
Re: Implementing Neural Turing Machines
#4Important and unsurprising sentence from the paper's abstract:
"A number
of open source implementations of NTMs exist but are unstable during
training and/or fail to replicate the reported performance of NTMs"
Re: Implementing Neural Turing Machines
#5Important and unsurprising sentence from the paper's abstract: " A number of open source implementations of NTMs exist but are unstable during training and/or fail to replicate the reported performance of NTMs "
Why do you deem it unsurprising? Do you think that the opensource neural network implementations are subpar?
Re: Implementing Neural Turing Machines
#6Important and unsurprising sentence from the paper's abstract: " A number of open source implementations of NTMs exist but are unstable during training and/or fail to replicate the reported performance of NTMs "
Why do you deem it unsurprising? Do you think that the opensource neural network implementations are subpar?
Anecdotally: I've tried to replicate some recent AI/ML papers and failed. So have some of my acquaintances.
Re: Implementing Neural Turing Machines
#7Important and unsurprising sentence from the paper's abstract: " A number of open source implementations of NTMs exist but are unstable during training and/or fail to replicate the reported performance of NTMs "
Why do you deem it unsurprising? Do you think that the opensource neural network implementations are subpar?
I would expect that it was just as unstable for DeepMind. But they can automatically run lots of experiments with different hyper params so that is less an issue for them.
Re: Implementing Neural Turing Machines
#8“how the memory contents of a NTM are initialized may be a defining factor in the success of a NTM implementation” Does it mean that one can expect better explainability in the future from these models?
The result was that constant/zero initialization was the best, which is the most natural choice anyway (for me at least), and definitely also the most simple option. I'm a bit surprised that they emphasise so much on that. Also, not really sure what to learn from this.
Re: Implementing Neural Turing Machines
#9It'd be interesting to see a Neural TM that can increase its Kolmogorov complexity.
Re: Implementing Neural Turing Machines
#10The author states in his github repository (https://github.com/MarkPKCollier/NeuralTuringMachine) that his work is based on (https://github.com/snowkylin/ntm). If that is the case then I find it kind of strange that he has relicensed the work to MIT from LGPL3 and removed any reference to the original author in the LICENSE. LGPL3 requires any derivative work to be licensed the same and retain mentions of the authors, more over it also requires a clear explanation of the modifications that were undertaken.
Also compared to the open source implementation (https://github.com/snowkylin/ntm) it seems like his main novel claim is that he looked at different memory initialisation patterns.
Edit:
compare the original: https://github.com/snowkylin/ntm/blob/master/ntm/ntm_cell.py
to the derivative work: https://github.com/MarkPKCollier/NeuralTuringMachine/blob/ma...
from what I can tell the main innovation is that the derivative work uses a named tuple instead of a dictionary for state keeping and there is new memory initialisation code. The original author apparently initialised the memory randomly. I also feel like the paper should cite the implementation they are basing their work on. The paper https://arxiv.org/pdf/1807.08518.pdf merely states that other implementations exist on page one and makes no mention of the fact that their implementation is based on one of those. Combine that with the fact that they are asking people in the Readme to cite their paper feels like not a very good idea.