Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Deep Forest: Towards an Alternative to Deep Neural Networks

arxiv.org

Post reply on HN
Print view View original

21–30 of 100 posts

Re: Deep Forest: Towards an Alternative to Deep Neural Networks

#21

Post by ipunchghosts »

computerex wrote: post #17
rkaplan wrote: post #14

"In contrast to deep neural networks which require great effort in hyper-parameter tuning, gcForest is much easier to train." Hyperparameter tuning is not as much of an issue with deep neural networks anymore. Thanks to BatchNorm and more robust optimization algorithms, most of the time you can simply use Adam with a default learning rate of 0.001 and do pretty well. Dropout is not even necessary with many models tha…

I couldn't disagree more. The defaults don't just work, and the architecture of the network could also be considered a hyper parameter in which case what would be a reasonable default for all the types of problems ANN are used for?

Are you using batch normalization? If you are, an issue I see all the time is folks not setting the EMA filter coef correctly. In keras, it defaults to something like 0.99 which in my mind makes no sense. I use something around 0.6 and life is good. You want to get an overall good measurement of the statistics and in my mind the frequency cutoff when coef=0.99 is just way too high for most application. You usually want something that filters out just about everything except very close to DC.

Re: Deep Forest: Towards an Alternative to Deep Neural Networks

#23

Post by [deleted] »

computerex wrote: post #17
rkaplan wrote: post #14

"In contrast to deep neural networks which require great effort in hyper-parameter tuning, gcForest is much easier to train." Hyperparameter tuning is not as much of an issue with deep neural networks anymore. Thanks to BatchNorm and more robust optimization algorithms, most of the time you can simply use Adam with a default learning rate of 0.001 and do pretty well. Dropout is not even necessary with many models tha…

I couldn't disagree more. The defaults don't just work, and the architecture of the network could also be considered a hyper parameter in which case what would be a reasonable default for all the types of problems ANN are used for?

[deleted]

Re: Deep Forest: Towards an Alternative to Deep Neural Networks

#24

Post by CuriouslyC »

paulsutter wrote: post #22

No Free Lunch theorem refesher: "if an algorithm performs well on a certain class of problems then it necessarily pays for that with degraded performance on the set of all remaining problems" https://en.m.wikipedia.org/wiki/No_free_lunch_theorem

That doesn't apply to an ensemble of algorithms where the weights of a given member of the ensemble are adapted based on observations from the given domain. If it did, humans wouldn't be able to choose a good algorithm for specific cases, and obviously we can.

Deep neural networks can be thought of as ensembles of smaller neural networks, though of course each member of the ensemble is going to share some degree of algorithmic bias. This suggests that perhaps deep neural networks with heterogeneous activation functions and branching structures will perform better than homogeneous networks.

Re: Deep Forest: Towards an Alternative to Deep Neural Networks

#25

Post by paulsutter »

CuriouslyC wrote: post #24
paulsutter wrote: post #22

No Free Lunch theorem refesher: "if an algorithm performs well on a certain class of problems then it necessarily pays for that with degraded performance on the set of all remaining problems" https://en.m.wikipedia.org/wiki/No_free_lunch_theorem

That doesn't apply to an ensemble of algorithms where the weights of a given member of the ensemble are adapted based on observations from the given domain. If it did, humans wouldn't be able to choose a good algorithm for specific cases, and obviously we can. Deep neural networks can be thought of as ensembles of smaller neural networks, though of course each member of the ensemble is going to share some degree of a…

The human brain is exactly such an ensemble of algorithms, and look at all the cognitive biases we have[1]. That our brain was developed for foraging, hunting, and reproducing could explain why the code we write is so buggy.

Look I'm just kidding around. Every algorithm has its advocates, and the arguments get tedious after a while.

[1] https://en.m.wikipedia.org/wiki/List_of_cognitive_biases

Re: Deep Forest: Towards an Alternative to Deep Neural Networks

#26

Post by eutectic »

paulsutter wrote: post #22

No Free Lunch theorem refesher: "if an algorithm performs well on a certain class of problems then it necessarily pays for that with degraded performance on the set of all remaining problems" https://en.m.wikipedia.org/wiki/No_free_lunch_theorem

The No Free lunch theorem is basically a consequence of the fact that almost all problems 'look random'; it doesn't really apply to the tiny subset that are of interest to humans.

Re: Deep Forest: Towards an Alternative to Deep Neural Networks

#27

Post by levesque »

DanielleMolloy wrote: post #3

So if this works well why is there no comparison on ImageNet?

It might be an alternative for deep neural networks, but I doubt it will be an alternative for deep convolutional neural networks. The key breakthrough for deep learning was the stacking of many convolutional layers and being able to train those (key elements include SGD for large scale, dropout or batch norm, etc.). Perhaps if they build some convolutional features for the decision tree they will be able to achieve comparable results, but it will start to look a lot like a deep convolutional neural net (i.e., the same thing with a different name).

Re: Deep Forest: Towards an Alternative to Deep Neural Networks

#28

Post by IanCal »

CuriouslyC wrote: post #24
paulsutter wrote: post #22

No Free Lunch theorem refesher: "if an algorithm performs well on a certain class of problems then it necessarily pays for that with degraded performance on the set of all remaining problems" https://en.m.wikipedia.org/wiki/No_free_lunch_theorem

That doesn't apply to an ensemble of algorithms where the weights of a given member of the ensemble are adapted based on observations from the given domain. If it did, humans wouldn't be able to choose a good algorithm for specific cases, and obviously we can. Deep neural networks can be thought of as ensembles of smaller neural networks, though of course each member of the ensemble is going to share some degree of a…

The No Free Lunch theorem doesn't really care how your algorithm does it, and an ensemble of algorithms is just another algorithm. It certainly applies.

The reason it's not actually all that useful for real world applications is that problems we value are a subset of "all possible problems" for want of a better explanation. Problems are typically smooth, that is a small change in input variables results in a small change in 'fitness', which already limits how applicable the NFL theorem is.

[disclaimer, based on lectures 10 years ago now]

Re: Deep Forest: Towards an Alternative to Deep Neural Networks

#29

Post by cgearhart »

eutectic wrote: post #26
paulsutter wrote: post #22

No Free Lunch theorem refesher: "if an algorithm performs well on a certain class of problems then it necessarily pays for that with degraded performance on the set of all remaining problems" https://en.m.wikipedia.org/wiki/No_free_lunch_theorem

The No Free lunch theorem is basically a consequence of the fact that almost all problems 'look random'; it doesn't really apply to the tiny subset that are of interest to humans.

> it doesn't really apply to the tiny subset that are of interest to humans.

That's a very misleading TL;DR. NFLT certainly _applies_ -- deep networks are not immune to NFLT -- it's just that NFLT isn't very useful because we can't use it as a basis for decisions. You can't detect that your algorithm's performance is being limited as a consequence of NFLT; and even if it were consequential, you would just see that the algorithm wasn't working very well, and there are _much_ more likely causes for that than NFLT.

Re: Deep Forest: Towards an Alternative to Deep Neural Networks

#30

Post by kortex »

ipunchghosts wrote: post #21
computerex wrote: post #17

Earlier quoted context omitted.

I couldn't disagree more. The defaults don't just work, and the architecture of the network could also be considered a hyper parameter in which case what would be a reasonable default for all the types of problems ANN are used for?

Are you using batch normalization? If you are, an issue I see all the time is folks not setting the EMA filter coef correctly. In keras, it defaults to something like 0.99 which in my mind makes no sense. I use something around 0.6 and life is good. You want to get an overall good measurement of the statistics and in my mind the frequency cutoff when coef=0.99 is just way too high for most application. You usually wa…

When working with images, do you use mode 0, 1, or 2?
Post reply on HN