Exploring LSTMs

#1

Post by deafcalculus » Sat, Jun 10, 2017, 5:43 AM UTC

Exploring LSTMs
blog.echen.me

Re: Exploring LSTMs

#2

Post by deepnotderp » Sat, Jun 10, 2017, 6:06 AM UTC

This is really nice but did we really have to use Java as an example language?

Re: Exploring LSTMs

#3

Post by minimaxir » Sat, Jun 10, 2017, 6:21 AM UTC

Is there code for the coloring of neurons per-character as in the post? I've seen that type of visualization on similar posts and am curious if there is a library for it. (the original char-rnn post [http://karpathy.github.io/2015/05/21/rnn-effectiveness/] indicates that it is custom Code/CSS/HTML)

Re: Exploring LSTMs

#4

Post by dirtyaura » Sat, Jun 10, 2017, 7:03 AM UTC

Really great work on visualizing neurons!

Is anyone working with LSTMs in a production setting? Any tips on what are the biggest challenges?

Jeremy Howard said in fast.ai course that in the applied setting, simpler GRUs work much better and has replaced LSTMs. Comments about this?

Re: Exploring LSTMs

#5

Post by trwoway » Sat, Jun 10, 2017, 7:05 AM UTC

minimaxir wrote: post #3

Is there code for the coloring of neurons per-character as in the post? I've seen that type of visualization on similar posts and am curious if there is a library for it. (the original char-rnn post [ http://karpathy.github.io/2015/05/21/rnn-effectiveness/ ] indicates that it is custom Code/CSS/HTML)

From Karpathy's blog post : http://cs.stanford.edu/people/karpathy/viscode.zip

Re: Exploring LSTMs

#6

Post by natch » Sat, Jun 10, 2017, 7:06 AM UTC

LSTM is "Long Short Term Memory," since the tutorial never mentions what it stands for.

https://en.wikipedia.org/wiki/Long_short-term_memory

Re: Exploring LSTMs

#7

Post by raarts » Sat, Jun 10, 2017, 7:23 AM UTC

Can someone provide a tl;dr ?

Re: Exploring LSTMs

#8

Post by jeffjose » Sat, Jun 10, 2017, 7:29 AM UTC

natch wrote: post #6

LSTM is "Long Short Term Memory," since the tutorial never mentions what it stands for. https://en.wikipedia.org/wiki/Long_short-term_memory

It's defined few sections in, but yes, your larger point stands.

Re: Exploring LSTMs

#9

Post by avaer » Sat, Jun 10, 2017, 7:38 AM UTC

raarts wrote: post #7

Can someone provide a tl;dr ?

Neural nets that update over time need a way to know what to keep and what to forget from last time. So let's learn what to keep and forget... by using more neural nets.

Disclaimer: I just learned what an LSTM was. But it's a good article.

Re: Exploring LSTMs

#10

Post by backpropaganda » Sat, Jun 10, 2017, 7:47 AM UTC

avaer wrote: post #9

raarts wrote: post #7

Can someone provide a tl;dr ?

Neural nets that update over time need a way to know what to keep and what to forget from last time. So let's learn what to keep and forget... by using more neural nets. Disclaimer: I just learned what an LSTM was. But it's a good article.

This is a surprisingly accurate tl;dr for someone who just learnt what LSTMs are.