Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Deep Forest: Towards an Alternative to Deep Neural Networks

arxiv.org

Post reply on HN
Print view View original

61–70 of 100 posts

Re: Deep Forest: Towards an Alternative to Deep Neural Networks

#61

Post by deong »

Houshalter wrote: post #38
paulsutter wrote: post #22

No Free Lunch theorem refesher: "if an algorithm performs well on a certain class of problems then it necessarily pays for that with degraded performance on the set of all remaining problems" https://en.m.wikipedia.org/wiki/No_free_lunch_theorem

There is a theoretically optimal machine learning algorithm called Solomonoff Induction. Solomonoff Induction assumes a prior over all possible computer programs that could have produced the data. And it assumes that shorter, simpler programs are more likely than longer, more complex ones. Under such a prior, the no free lunch theorem doesn't apply. NFLT assumes that all machine learning problems have infinite inform…

Solomonoff Induction isn't really an "algorithm" in the way we normally think of algorithms, as it isn't computable. There are computable approximations, but at that point, you lose the claim of theoretically optimal.

Re: Deep Forest: Towards an Alternative to Deep Neural Networks

#62

Post by Cybiote »

eutectic wrote: post #26
paulsutter wrote: post #22

No Free Lunch theorem refesher: "if an algorithm performs well on a certain class of problems then it necessarily pays for that with degraded performance on the set of all remaining problems" https://en.m.wikipedia.org/wiki/No_free_lunch_theorem

The No Free lunch theorem is basically a consequence of the fact that almost all problems 'look random'; it doesn't really apply to the tiny subset that are of interest to humans.

An often missed point is that the NFL has a practical consequence and that is, learning doesn't happen if the model architecture is not well adapted to the structure of the data. There being structure is not on its own strong enough to allow for any algorithm to do well. The model itself needs to be aligned with the structure in the data. This applies to animals, humans too, by inserting structural priors, evolution sacrificed performance on certain kinds of problems to do well on the kind we met in the natural world.

You can read more in: Antipredictable Sequences: Harder to Predict Than Random Sequences by Huaiyu Zhu and Wolfgang Kinzel

Re: Deep Forest: Towards an Alternative to Deep Neural Networks

#63

Post by deong »

CuriouslyC wrote: post #24
paulsutter wrote: post #22

No Free Lunch theorem refesher: "if an algorithm performs well on a certain class of problems then it necessarily pays for that with degraded performance on the set of all remaining problems" https://en.m.wikipedia.org/wiki/No_free_lunch_theorem

That doesn't apply to an ensemble of algorithms where the weights of a given member of the ensemble are adapted based on observations from the given domain. If it did, humans wouldn't be able to choose a good algorithm for specific cases, and obviously we can. Deep neural networks can be thought of as ensembles of smaller neural networks, though of course each member of the ensemble is going to share some degree of a…

> If it did, humans wouldn't be able to choose a good algorithm for specific cases, and obviously we can.

This is a surprisingly commonly held fallacy in some AI circles. It's the idea that humans are mathematically perfect. When you phrase it that way, it's fairly obviously false, but you still see a lot of people argue things like "NFL doesn't apply to ensembles because humans..." or "machines can never be as intelligent as humans because...".

The reality is that humans are subject to the same mathematical laws as machines. It's far more likely that my brain can't solve an NP-hard problem in polynomial time either. My brain can't beat random search on the set of all possible problems.

Re: Deep Forest: Towards an Alternative to Deep Neural Networks

#64

Post by BickNowstrom »

ssivark wrote: post #37
eutectic wrote: post #36

Earlier quoted context omitted.

If you have two predictors p0 and p1 then it can be shown that the mixture p(data) = (p0(data) + p1(data)) / 2 incurs at most 1 bit of loss compared to the better of p0 and p1. In other words, for any pair of machine learning algorithms there is an algorithm which only performs marginally worse than either of the two on any given problem, and which may perform arbitrarily better. NFL equalises these two cases by smea…

I often hear the claim that NFL just implies that different algorithms and learning strategies will succeed for different tasks. And that NFL has no practical consequences apart from that. Is there a resource I can refer to, that is clear and explicit about the reasoning?

NFL is probably the worst mis-quoted theorem in ML.

The subtle elements of the theorems (both for inference and search) have had huge impact on the field of optimization theory. They touch upon (Kolmogorov) Complexity, Incompleteness, Halting Problems, and later related work on the physical limits of inference: One can not know everything about the universe, if you are a part of it, and philosophy: Is the universe itself a computer?

That different algorithms and search strategies will succeed for different tasks, does not imply that we have to try them all: We can use prior knowledge on what worked on related tasks. Also the tasks we care about, is only a very small (explorable) subset of all possible tasks.

Re: Deep Forest: Towards an Alternative to Deep Neural Networks

#65

Post by BickNowstrom »

throw_away_777 wrote: post #41

I've always found it curious that Neural Networks get so much hype when xgboost (gradient boosted decision trees) is by far the most popular and accurate algorithm for most Kaggle competitions. While neural networks are better for image processing types of problems, there are a wide variety of machine learning problems where decision tree methods perform better and are much easier to implement.

The hype for neural networks is deserved. Some major contributions to the field resulted in increases in accuracy for fields like NLP, computer vision, structured data, machine translation, style transfer, etc.

XGBoost did not change much from the "Greedy function approximation: A gradient boosting machine." paper, but uses a few tricks to be much much faster, allowing for better tuning.

XGBoost is popular for structured data competitions on Kaggle. Even there: The winner is often an ensemble of XGBoost and Keras. And some structured data competitions are won by neural nets alone: http://blog.kaggle.com/2012/11/01/deep-learning-how-i-did-it... and https://www.kaggle.com/c/higgs-boson/discussion/10425 (Neural nets won the Higgs Boson Detection Challenge, where XGBoost was introduced)

I'd say Tensorflow/Keras can handle a wider variety of problems, with the same, or improved accuracy, than tree-based methods can. NN's do well on structured problems (the domain of tree-based methods), but also own computer vision and, increasingly, NLP. I agree on the pitfalls of applying neural nets vs. forests.

It is true that tree-based methods are academically a bit out of vogue: The exciting stuff is happening in the neural network space. You have more chance of getting published with deep learning (this used to be the other way around).

Re: Deep Forest: Towards an Alternative to Deep Neural Networks

#66

Post by BickNowstrom »

nojvek wrote: post #44
throw_away_777 wrote: post #41

I've always found it curious that Neural Networks get so much hype when xgboost (gradient boosted decision trees) is by far the most popular and accurate algorithm for most Kaggle competitions. While neural networks are better for image processing types of problems, there are a wide variety of machine learning problems where decision tree methods perform better and are much easier to implement.

Any good links you recommend learning xgboost? I've never quite figured out how they work.

http://xgboost.readthedocs.io/en/latest/model.html

http://homes.cs.washington.edu/~tqchen/pdf/BoostedTree.pdf

https://www.youtube.com/watch?v=wPqtzj5VZus Trevor Hastie - Gradient Boosting Machine Learning

https://www.youtube.com/watch?v=sRktKszFmSk Ensembles (3): Gradient Boosting, Ihler

Re: Deep Forest: Towards an Alternative to Deep Neural Networks

#67

Post by PeterisP »

DanielleMolloy wrote: post #3

So if this works well why is there no comparison on ImageNet?

It doesn't do well - e.g. on the toy example of MNIST they report 98.96% which is, frankly, rather horrible.

They claim that deep neural approaches get 98.75% or 99.05% by referencing obsolete decade old results, while in fact state of art exceeds 99.8% (i.e. 0.2% error rate, which is five times lower than 1.0% error rate reported in this paper). I have seen MNIST given as a homework exercise for undergraduate students in ML/NN class, and getting 99.0% would indicate that your code has some serious bugs, a decent undergrad with no prior experience can get 99.7% on MNIST after a few lecture introduction to basics and a dozen hours of homework coding practice.

Re: Deep Forest: Towards an Alternative to Deep Neural Networks

#69

Post by cosminro »

rkaplan wrote: post #14

"In contrast to deep neural networks which require great effort in hyper-parameter tuning, gcForest is much easier to train." Hyperparameter tuning is not as much of an issue with deep neural networks anymore. Thanks to BatchNorm and more robust optimization algorithms, most of the time you can simply use Adam with a default learning rate of 0.001 and do pretty well. Dropout is not even necessary with many models tha…

No batch norm for LSTMs

Re: Deep Forest: Towards an Alternative to Deep Neural Networks

#70

Post by BickNowstrom »

plafl wrote: post #5
DanielleMolloy wrote: post #3

So if this works well why is there no comparison on ImageNet?

That's exactly what I was going to post, but you beat me. I found surprising that they claim competitive performance over neural networks just using a CPU. The advantage I think neural networks have over all other methods is how efficiently they run on GPUs. I would love to see other methods take advantage of modern hardware or parallelism and see if they can catch up with neural networks on image classification.

Random forests can be run on the GPU: https://github.com/EasonLiao/CudaTree
Post reply on HN