Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Exploring LSTMs

blog.echen.me

Post reply on HN
Print view View original

11–20 of 48 posts

Re: Exploring LSTMs

#11

Post by agibsonccc »

dirtyaura wrote: post #4

Really great work on visualizing neurons! Is anyone working with LSTMs in a production setting? Any tips on what are the biggest challenges? Jeremy Howard said in fast.ai course that in the applied setting, simpler GRUs work much better and has replaced LSTMs. Comments about this?

Yes the bulk of our business is time series. This includes everything from hardware break downs to fraud detection. I think Jeremy has some good points but in general, but I wouldn't assume that everything is binary. (By this, I mean look at these kinds of terse statements with a bit of nuance)

Usually as long as you have a high amount of regularization and use truncated backprop through time in training you can learn some fairly useful classification and forecasting problems.

Beyond that standard neural net tuning applies. Eg: normalize your data, pay attention to your weight initialization, understand what loss function you're using,..

Re: Exploring LSTMs

#12

Post by msamwald »

dirtyaura wrote: post #4

Really great work on visualizing neurons! Is anyone working with LSTMs in a production setting? Any tips on what are the biggest challenges? Jeremy Howard said in fast.ai course that in the applied setting, simpler GRUs work much better and has replaced LSTMs. Comments about this?

A recent paper exploring various variatons of neural machine translation architectures found that LSTMs consistently outperformed GRUs in that task.

https://arxiv.org/abs/1703.03906

Re: Exploring LSTMs

#13

Post by dnautics »

avaer wrote: post #9
raarts wrote: post #7

Can someone provide a tl;dr ?

Neural nets that update over time need a way to know what to keep and what to forget from last time. So let's learn what to keep and forget... by using more neural nets. Disclaimer: I just learned what an LSTM was. But it's a good article.

One more key point:. Memory is additive, not multiplicative.

Re: Exploring LSTMs

#14

Post by denzil_correa »

dirtyaura wrote: post #4

Really great work on visualizing neurons! Is anyone working with LSTMs in a production setting? Any tips on what are the biggest challenges? Jeremy Howard said in fast.ai course that in the applied setting, simpler GRUs work much better and has replaced LSTMs. Comments about this?

> simpler GRUs

What does GRU stand for?

Re: Exploring LSTMs

#15

Post by olavgg »

dirtyaura wrote: post #4

Really great work on visualizing neurons! Is anyone working with LSTMs in a production setting? Any tips on what are the biggest challenges? Jeremy Howard said in fast.ai course that in the applied setting, simpler GRUs work much better and has replaced LSTMs. Comments about this?

Our chatbot uses LSTM(GRU) for prediction, we see about the same performance as LSTM, but training is more efficient.

Re: Exploring LSTMs

#16

Post by inlineint »

I personally find recurrent highway networks (RHNs) as described in [1] to be easier to understand and remember the formulas for than the original LSTM. Because as they are generalizations of LSTM, if one understands RHNs, one can understand LSTMs as just a particular case of RHN.

Instead of handwaving about "forgetting", it is IMO better to understand the problem of vanishing gradients and how can forget gates actually help with them.

And Jürgen Schmidhuber, the inventor of LSTM, is a co-author of the RHN paper.

[1] https://arxiv.org/abs/1607.03474

Re: Exploring LSTMs

#17

Post by mrplank »

LSTMs are on their retour in my opinion. They are a hack to make memory in recurrent networks more persistent. In practice they overfit too easy. They are being replaced with convolutional networks. Have a look at the latest paper from Facebook about translation for more details.

Re: Exploring LSTMs

#18

Post by visarga »

LSTMs are both amazing and not quite good enough. They seem to be too complicated for what they do well, and not quite complex enough for what they can't do so well. The main limitation is that they mix structure with style, or type with value. For example, if you want an LSTM to learn addition, if you taught it to operate on numbers of 6 digits it won't be able to generalize on numbers of 20 digits.

That's because it doesn't factorize the input into separate meaningful parts. The next step in LSTMs will be to operate over relational graphs so they only have to learn function and not structure at the same time. That way they will be able to generalize more between different situations and be much more useful.

Graphs can be represented as adjacency matrices and data as vectors. By multiplying vector with matrix, you can do graph computation. Recurring graph computations are a lot like LSTMs. That's why I think LSTMs are going to become more invariant to permutation and object composition in the future, by using graph data representation instead of flat euclidean vectors, and typed data instead of untyped data. So they are going to become strongly typed, graph RNNs. With such toys we can do visual and text based reasoning, and physical simulation.

Re: Exploring LSTMs

#19

Post by davik »

denzil_correa wrote: post #14
dirtyaura wrote: post #4

Really great work on visualizing neurons! Is anyone working with LSTMs in a production setting? Any tips on what are the biggest challenges? Jeremy Howard said in fast.ai course that in the applied setting, simpler GRUs work much better and has replaced LSTMs. Comments about this?

> simpler GRUs What does GRU stand for?

Gated recurrent unit, it has fewer gates than LSTMs usually.

Re: Exploring LSTMs

#20

Post by LukeB42 »

visarga wrote: post #18

LSTMs are both amazing and not quite good enough. They seem to be too complicated for what they do well, and not quite complex enough for what they can't do so well. The main limitation is that they mix structure with style, or type with value. For example, if you want an LSTM to learn addition, if you taught it to operate on numbers of 6 digits it won't be able to generalize on numbers of 20 digits. That's because i…

Any links to implementations?
Post reply on HN