Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Deep Forest: Towards an Alternative to Deep Neural Networks

arxiv.org

Post reply on HN
Print view View original

31–40 of 100 posts

Re: Deep Forest: Towards an Alternative to Deep Neural Networks

#31

Post by cgearhart »

DanielleMolloy wrote: post #15
kercker wrote: post #9

Earlier quoted context omitted.

In the conclusion section the authors said that : "If we had stronger computational facilities, we would like to try big data and deeper forest, which is left for future work." and that: "As a seminar study, we have only explored a little in this direction."

If someone proposes a method as an alternative for a field, they need to test this method on the accepted benchmark dataset for that field. For object recognition in static images this dataset is the ImageNet competition. Computing power can be bought from AWS if no cluster is available. The lack of it can't be an argument. Not saying that the paper has no reason to exist, I think it is generally well written and dec…

I agree their claim is a bit hyperbolic, but that's an unreasonably high bar to expect for the scope of this paper.

Re: Deep Forest: Towards an Alternative to Deep Neural Networks

#32

Post by mljoe »

paulsutter wrote: post #22

No Free Lunch theorem refesher: "if an algorithm performs well on a certain class of problems then it necessarily pays for that with degraded performance on the set of all remaining problems" https://en.m.wikipedia.org/wiki/No_free_lunch_theorem

"All remaining problems" in this case means the universe of all potential functions. Everything. Only a vanishingly small subset of every kind of function or dataset are of interest or even observable in nature.

Re: Deep Forest: Towards an Alternative to Deep Neural Networks

#34

Post by arjo129 »

rkaplan wrote: post #14

"In contrast to deep neural networks which require great effort in hyper-parameter tuning, gcForest is much easier to train." Hyperparameter tuning is not as much of an issue with deep neural networks anymore. Thanks to BatchNorm and more robust optimization algorithms, most of the time you can simply use Adam with a default learning rate of 0.001 and do pretty well. Dropout is not even necessary with many models tha…

They work well, just that you need a lot of patience (and know how) to work with them. Also GPUs are expensive. By the time you realize that you messed up you have wasted a lot of time. Of course this is true with any ml algorithm out there. But what I'm trying to say is it is possible that an as yet unknown method exists that may be less computationally complex.

One of the problems I see is that people abuse deep neural networks no end. One doesn't need to train a deep nn for recognizing structured objects like a coke can in a fridge. Simple hog/sift/other feature engineering may be a faster and better bet for small-scale object recognition. However expecting sift to out perform a deep neural net on imagenet is out of question. Thus when it comes to deploying systems in a short frame of time one should keep an open mind.

Re: Deep Forest: Towards an Alternative to Deep Neural Networks

#35

Post by arjo129 »

rkaplan wrote: post #14

"In contrast to deep neural networks which require great effort in hyper-parameter tuning, gcForest is much easier to train." Hyperparameter tuning is not as much of an issue with deep neural networks anymore. Thanks to BatchNorm and more robust optimization algorithms, most of the time you can simply use Adam with a default learning rate of 0.001 and do pretty well. Dropout is not even necessary with many models tha…

They work well, just that you need a lot of patience (and know how) to work with them. Also GPUs are expensive. By the time you realize that you messed up you have wasted a lot of time. Of course this is true with any ml algorithm out there. But what I'm trying to say is it is possible that an as yet unknown method exists that may be less computationally complex.

One of the problems I see is that people abuse deep neural networks no end. One doesn't need to train a deep nn for recognizing structured objects like a coke can in a fridge. Simple hog/sift/other feature engineering may be a faster and better bet for small-scale object recognition. However expecting sift to out perform a deep neural net on imagenet is out of question. Thus when it comes to deploying systems in a short frame of time one should keep an open mind.

Re: Deep Forest: Towards an Alternative to Deep Neural Networks

#36

Post by eutectic »

cgearhart wrote: post #29
eutectic wrote: post #26

Earlier quoted context omitted.

The No Free lunch theorem is basically a consequence of the fact that almost all problems 'look random'; it doesn't really apply to the tiny subset that are of interest to humans.

> it doesn't really apply to the tiny subset that are of interest to humans. That's a very misleading TL;DR. NFLT certainly _applies_ -- deep networks are not immune to NFLT -- it's just that NFLT isn't very useful because we can't use it as a basis for decisions. You can't detect that your algorithm's performance is being limited as a consequence of NFLT; and even if it were consequential, you would just see that th…

If you have two predictors p0 and p1 then it can be shown that the mixture p(data) = (p0(data) + p1(data)) / 2 incurs at most 1 bit of loss compared to the better of p0 and p1.

In other words, for any pair of machine learning algorithms there is an algorithm which only performs marginally worse than either of the two on any given problem, and which may perform arbitrarily better. NFL equalises these two cases by smearing tiny losses over vast regions of possibility-space that are vanishingly unlikely.

Re: Deep Forest: Towards an Alternative to Deep Neural Networks

#37

Post by ssivark »

eutectic wrote: post #36
cgearhart wrote: post #29

Earlier quoted context omitted.

> it doesn't really apply to the tiny subset that are of interest to humans. That's a very misleading TL;DR. NFLT certainly _applies_ -- deep networks are not immune to NFLT -- it's just that NFLT isn't very useful because we can't use it as a basis for decisions. You can't detect that your algorithm's performance is being limited as a consequence of NFLT; and even if it were consequential, you would just see that th…

If you have two predictors p0 and p1 then it can be shown that the mixture p(data) = (p0(data) + p1(data)) / 2 incurs at most 1 bit of loss compared to the better of p0 and p1. In other words, for any pair of machine learning algorithms there is an algorithm which only performs marginally worse than either of the two on any given problem, and which may perform arbitrarily better. NFL equalises these two cases by smea…

I often hear the claim that NFL just implies that different algorithms and learning strategies will succeed for different tasks. And that NFL has no practical consequences apart from that.

Is there a resource I can refer to, that is clear and explicit about the reasoning?

Re: Deep Forest: Towards an Alternative to Deep Neural Networks

#38

Post by Houshalter »

paulsutter wrote: post #22

No Free Lunch theorem refesher: "if an algorithm performs well on a certain class of problems then it necessarily pays for that with degraded performance on the set of all remaining problems" https://en.m.wikipedia.org/wiki/No_free_lunch_theorem

There is a theoretically optimal machine learning algorithm called Solomonoff Induction. Solomonoff Induction assumes a prior over all possible computer programs that could have produced the data. And it assumes that shorter, simpler programs are more likely than longer, more complex ones. Under such a prior, the no free lunch theorem doesn't apply.

NFLT assumes that all machine learning problems have infinite information. Infinite Kolmogorov complexity. That all positive and negative examples are labelled completely randomly without any underlying pattern or reason. Which is obviously untrue.

Re: Deep Forest: Towards an Alternative to Deep Neural Networks

#39

Post by eutectic »

ssivark wrote: post #37
eutectic wrote: post #36

Earlier quoted context omitted.

If you have two predictors p0 and p1 then it can be shown that the mixture p(data) = (p0(data) + p1(data)) / 2 incurs at most 1 bit of loss compared to the better of p0 and p1. In other words, for any pair of machine learning algorithms there is an algorithm which only performs marginally worse than either of the two on any given problem, and which may perform arbitrarily better. NFL equalises these two cases by smea…

I often hear the claim that NFL just implies that different algorithms and learning strategies will succeed for different tasks. And that NFL has no practical consequences apart from that. Is there a resource I can refer to, that is clear and explicit about the reasoning?

The fact that different algorithms are best for different tasks has more to do with computational complexity.

If we didn't care about resource usage then we could just pick a sufficiently expressive model class (e.g. all algorithms) to mix over, and perform Bayesian inference. You can look up AIXI and Solomonoff induction for more thoughts along these lines.

Re: Deep Forest: Towards an Alternative to Deep Neural Networks

#40

Post by KirinDave »

While optimizations to cost by ditching GPUs as a requirement are important (and presumably these systems benefit from GPU optimization as well, seems unclear from my skim of the paper), cheaper training is NOT just about saving your wallet.

A real emerging area of opportunity is having systems train new systems. This has numerous applications, including assisting DSEs in the construction of new systems or allowing expert systems to learn more over time and even integrate new techniques into a currently deployed system.

I'n not an expert here, but I'd like to be, so I'm definitely going to ask my expert friends more about this.

Post reply on HN