Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Deep Forest: Towards an Alternative to Deep Neural Networks

arxiv.org

Post reply on HN
Print view View original

51–60 of 100 posts

Re: Deep Forest: Towards an Alternative to Deep Neural Networks

#51

Post by gcr »

ipunchghosts wrote: post #21
computerex wrote: post #17

Earlier quoted context omitted.

I couldn't disagree more. The defaults don't just work, and the architecture of the network could also be considered a hyper parameter in which case what would be a reasonable default for all the types of problems ANN are used for?

Are you using batch normalization? If you are, an issue I see all the time is folks not setting the EMA filter coef correctly. In keras, it defaults to something like 0.99 which in my mind makes no sense. I use something around 0.6 and life is good. You want to get an overall good measurement of the statistics and in my mind the frequency cutoff when coef=0.99 is just way too high for most application. You usually wa…

The response to "the defaults should work just fine without any hyperparameter tuning" is "try fiddling with the EMA filter coefficient hyperparameter" ?

(Just poking fun. :P)

Re: Deep Forest: Towards an Alternative to Deep Neural Networks

#52

Post by argonaut »

jkbschwarz wrote: post #7
jre wrote: post #4

I don't know about the others, but the two visions dataset they compare to (MNIST and the face recognition one) are small datasets and the CNN they compare to doesn't seem very state of the art. It also seems each layer of random forest just concatenates a class distribution to the original feature vector. So this doesn't seem to get the same "hierarchy of features" benefit that you get in large-scale CNN and DNN.

To your point that they are comparing small datasets. I dont see that as a problem. If they achieve better results on small datasets that is a great achievement, as often the bottleneck is the size of the dataset rather than computation time.

They achieve substantially worse results than state of the art on (the toy dataset) MNIST.

Re: Deep Forest: Towards an Alternative to Deep Neural Networks

#53

Post by joe_the_user »

gcr wrote: post #51
ipunchghosts wrote: post #21

Earlier quoted context omitted.

Are you using batch normalization? If you are, an issue I see all the time is folks not setting the EMA filter coef correctly. In keras, it defaults to something like 0.99 which in my mind makes no sense. I use something around 0.6 and life is good. You want to get an overall good measurement of the statistics and in my mind the frequency cutoff when coef=0.99 is just way too high for most application. You usually wa…

The response to "the defaults should work just fine without any hyperparameter tuning" is "try fiddling with the EMA filter coefficient hyperparameter" ? (Just poking fun. :P)

It's like the joke of the mathematician giving an exposition of a complex proof. At one point he says "It is obvious that X", pauses, scratches his head, does a few calculations. Leaves room for twenty minutes and returns. Then continues "it is obvious that X" and goes to the next step.

Deep in the field, it's fine for machine learning experts to say "everything just works" [if you've mastered X, Y, Q esoteric fields and tuning methods] since they're welcome to "humble brag" as much as they want. But when this gets in the way of figuring out what really "just works" it's more of a problem.

Re: Deep Forest: Towards an Alternative to Deep Neural Networks

#54

Post by soVeryTired »

ungzd wrote: post #33

Was about to joke about Deep Support Vector Machines, but found out they exist too: https://www.esat.kuleuven.be/sista/ROKS2013/files/presentati... http://deeplearning.net/wp-content/uploads/2013/03/dlsvm.pdf

Deep linear regression?

Re: Deep Forest: Towards an Alternative to Deep Neural Networks

#55

Post by eanzenberg »

soVeryTired wrote: post #54
ungzd wrote: post #33

Was about to joke about Deep Support Vector Machines, but found out they exist too: https://www.esat.kuleuven.be/sista/ROKS2013/files/presentati... http://deeplearning.net/wp-content/uploads/2013/03/dlsvm.pdf

Deep linear regression?

That's quite literally deep learning so made me lol.

Re: Deep Forest: Towards an Alternative to Deep Neural Networks

#56

Post by modeless »

arjo129 wrote: post #34
rkaplan wrote: post #14

"In contrast to deep neural networks which require great effort in hyper-parameter tuning, gcForest is much easier to train." Hyperparameter tuning is not as much of an issue with deep neural networks anymore. Thanks to BatchNorm and more robust optimization algorithms, most of the time you can simply use Adam with a default learning rate of 0.001 and do pretty well. Dropout is not even necessary with many models tha…

They work well, just that you need a lot of patience (and know how) to work with them. Also GPUs are expensive. By the time you realize that you messed up you have wasted a lot of time. Of course this is true with any ml algorithm out there. But what I'm trying to say is it is possible that an as yet unknown method exists that may be less computationally complex. One of the problems I see is that people abuse deep ne…

> One doesn't need to train a deep nn for recognizing structured objects like a coke can in a fridge.

I disagree. Sure, you don't need a NN to recognize one Coke can in one fridge for your toy robot project. If you want to recognize all Coke cans in all fridges, for your real-world, consumer-ready Coke-fetching robot product? You're going to need a huge dataset of all the various designs of Coke cans out there, in all the different kinds of refrigerators, and your toy feature engineered approach is going to lose to a NN on that kind of varied dataset.

Re: Deep Forest: Towards an Alternative to Deep Neural Networks

#57

Post by amelius »

modeless wrote: post #56
arjo129 wrote: post #34

Earlier quoted context omitted.

They work well, just that you need a lot of patience (and know how) to work with them. Also GPUs are expensive. By the time you realize that you messed up you have wasted a lot of time. Of course this is true with any ml algorithm out there. But what I'm trying to say is it is possible that an as yet unknown method exists that may be less computationally complex. One of the problems I see is that people abuse deep ne…

> One doesn't need to train a deep nn for recognizing structured objects like a coke can in a fridge. I disagree. Sure, you don't need a NN to recognize one Coke can in one fridge for your toy robot project. If you want to recognize all Coke cans in all fridges, for your real-world, consumer-ready Coke-fetching robot product? You're going to need a huge dataset of all the various designs of Coke cans out there, in al…

But an NN can complete mess up when a new refrigerator is used, that wasn't part of the training set.

Also, the training is very asymmetric, since there are many more things NOT coke cans than there are coke cans.

Re: Deep Forest: Towards an Alternative to Deep Neural Networks

#58

Post by gwern »

gcr wrote: post #50
rkaplan wrote: post #14

"In contrast to deep neural networks which require great effort in hyper-parameter tuning, gcForest is much easier to train." Hyperparameter tuning is not as much of an issue with deep neural networks anymore. Thanks to BatchNorm and more robust optimization algorithms, most of the time you can simply use Adam with a default learning rate of 0.001 and do pretty well. Dropout is not even necessary with many models tha…

GAN training is still spooky mysterious and can easily fail in nonintuitive ways. Sometimes GANs converge or not depending on the random number seed, even with the same hyperparameters.

I'm not sure about that. The new GAN models over the past 2-3 months, like LS-GAN or WGAN, all seem to train much more stably. I've beaten up on WGAN with all sorts of strange tweaks and hyperparameter settings and while it may not work well, it's never catastrophically diverged on me the way DCGAN would at the drop of a hat.

Re: Deep Forest: Towards an Alternative to Deep Neural Networks

#59

Post by walrus »

dirtyaura wrote: post #42
ipunchghosts wrote: post #21

Earlier quoted context omitted.

Are you using batch normalization? If you are, an issue I see all the time is folks not setting the EMA filter coef correctly. In keras, it defaults to something like 0.99 which in my mind makes no sense. I use something around 0.6 and life is good. You want to get an overall good measurement of the statistics and in my mind the frequency cutoff when coef=0.99 is just way too high for most application. You usually wa…

Interesting, totally new concept for me: Where can I read more about EMA filter coefficient in Keras? My Google-fu is failing.

I think they're referring to the momentum parameter at [1]. The exponential moving average (EMA) of the batch mean/variance is used in the batch normalizing transform (Algorithm 1 in [2]).

The momentum ranges from 0 to 1. If it's close to 1, which the default of 0.99 is, the EMA of the batch mean/variance will change slowly across batches. If it's close to 0, the EMA will be close to the mean/variance of the current batch.

The EMA acts as a low-pass filter. With a momentum close to 1, the EMA changes slowly, filtering out high frequencies and leaving only frequencies close to DC. Note that this is opposite to what grandparent says: 0.99 has a lower frequency cutoff than 0.6 does. So I'm not really sure what they're getting at there.

[1] https://keras.io/layers/normalization/#batchnormalization

[2] https://arxiv.org/abs/1502.03167

Re: Deep Forest: Towards an Alternative to Deep Neural Networks

#60

Post by modeless »

amelius wrote: post #57
modeless wrote: post #56

Earlier quoted context omitted.

> One doesn't need to train a deep nn for recognizing structured objects like a coke can in a fridge. I disagree. Sure, you don't need a NN to recognize one Coke can in one fridge for your toy robot project. If you want to recognize all Coke cans in all fridges, for your real-world, consumer-ready Coke-fetching robot product? You're going to need a huge dataset of all the various designs of Coke cans out there, in al…

But an NN can complete mess up when a new refrigerator is used, that wasn't part of the training set. Also, the training is very asymmetric, since there are many more things NOT coke cans than there are coke cans.

> But an NN can complete mess up when a new refrigerator is used, that wasn't part of the training set

Not if your training set is representative. And this is just as true of feature engineered approaches, the only difference is that dealing with real world variation requires a lot less work with NNs because once you add the variation to your dataset you're done. With feature engineering that's only the first step because now you have to figure out where the new variation is breaking your features and how to modify them to fix it.

Post reply on HN