Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Deep Forest: Towards an Alternative to Deep Neural Networks

arxiv.org

Post reply on HN
Print view View original

71–80 of 100 posts

Re: Deep Forest: Towards an Alternative to Deep Neural Networks

#71

Post by Xcelerate »

deong wrote: post #61
Houshalter wrote: post #38

Earlier quoted context omitted.

There is a theoretically optimal machine learning algorithm called Solomonoff Induction. Solomonoff Induction assumes a prior over all possible computer programs that could have produced the data. And it assumes that shorter, simpler programs are more likely than longer, more complex ones. Under such a prior, the no free lunch theorem doesn't apply. NFLT assumes that all machine learning problems have infinite inform…

Solomonoff Induction isn't really an "algorithm" in the way we normally think of algorithms, as it isn't computable. There are computable approximations, but at that point, you lose the claim of theoretically optimal.

> There are computable approximations, but at that point, you lose the claim of theoretically optimal.

True, but I'm surprised more work isn't being done on "searching the space of programs that produce the data". There's very little research on this topic other than a few papers on minimum description length (MDL). I feel that this is probably the eventual route to AGI. We know what the "optimal" predictor is in theory; now work out the best time/memory approximation to it for practical purposes.

Re: Deep Forest: Towards an Alternative to Deep Neural Networks

#72

Post by mljoe »

soVeryTired wrote: post #68
eanzenberg wrote: post #55

Earlier quoted context omitted.

That's quite literally deep learning so made me lol.

Can you elaborate on that? I feel like I've missed something.

There is no "deep logistic regression" because such a construct is just called a "neural network". (It's not the only kind of neural network however.)

Re: Deep Forest: Towards an Alternative to Deep Neural Networks

#73

Post by eanzenberg »

soVeryTired wrote: post #68
eanzenberg wrote: post #55

Earlier quoted context omitted.

That's quite literally deep learning so made me lol.

Can you elaborate on that? I feel like I've missed something.

Linear regression on top of linear regression with non-linearities in between is a neural network.

Re: Deep Forest: Towards an Alternative to Deep Neural Networks

#74

Post by AlexCoventry »

gwern wrote: post #58
gcr wrote: post #50

Earlier quoted context omitted.

GAN training is still spooky mysterious and can easily fail in nonintuitive ways. Sometimes GANs converge or not depending on the random number seed, even with the same hyperparameters.

I'm not sure about that. The new GAN models over the past 2-3 months, like LS-GAN or WGAN, all seem to train much more stably. I've beaten up on WGAN with all sorts of strange tweaks and hyperparameter settings and while it may not work well, it's never catastrophically diverged on me the way DCGAN would at the drop of a hat.

Have you found any good ways to speed it up? The five-fold training on the Critic is very expensive.

Re: Deep Forest: Towards an Alternative to Deep Neural Networks

#75

Post by throw_away_777 »

BickNowstrom wrote: post #65
throw_away_777 wrote: post #41

I've always found it curious that Neural Networks get so much hype when xgboost (gradient boosted decision trees) is by far the most popular and accurate algorithm for most Kaggle competitions. While neural networks are better for image processing types of problems, there are a wide variety of machine learning problems where decision tree methods perform better and are much easier to implement.

The hype for neural networks is deserved. Some major contributions to the field resulted in increases in accuracy for fields like NLP, computer vision, structured data, machine translation, style transfer, etc. XGBoost did not change much from the "Greedy function approximation: A gradient boosting machine." paper, but uses a few tricks to be much much faster, allowing for better tuning. XGBoost is popular for struct…

I agree that neural nets are state-of-the-art and do quite well on certain types of problems (NLP and vision, which are important problems). But a lot of data is structured (sales, churn, recommendations, etc), and it is so much easier to train an xgboost model than a neural net model. You need a very expensive computer or expensive cloud computing to train neural nets, and even then it is not easy. Ease of implementation is an important factor that gets overlooked in academia. And on non-NLP and non-image datasets, usually the single best Kaggle model is an xgboost model, which was probably developed in 1/10th the time it took to make a good neural net model. Xgboost has come a long way since it was first introduced, with early stopping being an example of a significant improvement.

Re: Deep Forest: Towards an Alternative to Deep Neural Networks

#76

Post by snovv_crash »

modeless wrote: post #56
arjo129 wrote: post #34

Earlier quoted context omitted.

They work well, just that you need a lot of patience (and know how) to work with them. Also GPUs are expensive. By the time you realize that you messed up you have wasted a lot of time. Of course this is true with any ml algorithm out there. But what I'm trying to say is it is possible that an as yet unknown method exists that may be less computationally complex. One of the problems I see is that people abuse deep ne…

> One doesn't need to train a deep nn for recognizing structured objects like a coke can in a fridge. I disagree. Sure, you don't need a NN to recognize one Coke can in one fridge for your toy robot project. If you want to recognize all Coke cans in all fridges, for your real-world, consumer-ready Coke-fetching robot product? You're going to need a huge dataset of all the various designs of Coke cans out there, in al…

Which is why you should do stereo or SfM, make a 3d reconstruction, and then do HOG or some 3D feature to recognise the coke can.

Trying to do it from images with a NN that doesn't comprehend 3D space is just silly.

Re: Deep Forest: Towards an Alternative to Deep Neural Networks

#77

Post by sgt101 »

snovv_crash wrote: post #76
modeless wrote: post #56

Earlier quoted context omitted.

> One doesn't need to train a deep nn for recognizing structured objects like a coke can in a fridge. I disagree. Sure, you don't need a NN to recognize one Coke can in one fridge for your toy robot project. If you want to recognize all Coke cans in all fridges, for your real-world, consumer-ready Coke-fetching robot product? You're going to need a huge dataset of all the various designs of Coke cans out there, in al…

Which is why you should do stereo or SfM, make a 3d reconstruction, and then do HOG or some 3D feature to recognise the coke can. Trying to do it from images with a NN that doesn't comprehend 3D space is just silly.

Errm, hang on, are you saying that if you have a task of classifying unseen images given a labelled training set you should get a stereo camera or video camera and create another problem?

Which you can solve?

Because the problem is silly>

What if I say : "I will give you $10m to solve it, and if you fail, I will kill this very kind old monkey?"

Re: Deep Forest: Towards an Alternative to Deep Neural Networks

#78

Post by Cybiote »

Xcelerate wrote: post #71
deong wrote: post #61

Earlier quoted context omitted.

Solomonoff Induction isn't really an "algorithm" in the way we normally think of algorithms, as it isn't computable. There are computable approximations, but at that point, you lose the claim of theoretically optimal.

> There are computable approximations, but at that point, you lose the claim of theoretically optimal. True, but I'm surprised more work isn't being done on "searching the space of programs that produce the data". There's very little research on this topic other than a few papers on minimum description length (MDL). I feel that this is probably the eventual route to AGI. We know what the "optimal" predictor is in the…

There's lots of work that's being done related to program synthesis and inductive logic programming. You can even view the more sophisticated recurrent Neural nets, with more complex memory structures, as differentiable programs. Then you're searching the space of programs guided by gradients. SGD effectively acts as an additional prior (assumption) by the kind of solutions it tends towards.

The reason (the general) you don't hear much about them if you don't go looking is that the state of the art hasn't budged much for the past couple decades. It's the same graph algorithms, searching and sorting toy problems. The search space over programs is difficult to traverse and it remains to be seen what the added compute power + gradients gets us.

On a more practical level, the learning 2 search paradigm can be viewed as also searching for a particular program under certain strict constraints that make search tractable. Probabilistic programming where the priors and likelihoods are themselves complex programs instead of simple distributions from the exponential family are effectively also searching for programs.

Re: Deep Forest: Towards an Alternative to Deep Neural Networks

#79

Post by sgt101 »

throw_away_777 wrote: post #41

I've always found it curious that Neural Networks get so much hype when xgboost (gradient boosted decision trees) is by far the most popular and accurate algorithm for most Kaggle competitions. While neural networks are better for image processing types of problems, there are a wide variety of machine learning problems where decision tree methods perform better and are much easier to implement.

Because in the real world we need robust classifiers not optimised ones.

Problem : robust && optimized are very vague terms in ML.

Re: Deep Forest: Towards an Alternative to Deep Neural Networks

#80

Post by ska »

modeless wrote: post #60
amelius wrote: post #57

Earlier quoted context omitted.

But an NN can complete mess up when a new refrigerator is used, that wasn't part of the training set. Also, the training is very asymmetric, since there are many more things NOT coke cans than there are coke cans.

> But an NN can complete mess up when a new refrigerator is used, that wasn't part of the training set Not if your training set is representative. And this is just as true of feature engineered approaches, the only difference is that dealing with real world variation requires a lot less work with NNs because once you add the variation to your dataset you're done. With feature engineering that's only the first step be…

"Not if your training set is representative."

And herein lies a prominent failure mode of a huge amount of this sort of work that I've seen - hard to just "add the variation to your dataset" when your data set is one or more orders of magnitude too small to contain it. At that point all that remains is the handwaving.

The right response to insufficient data is usually simplifying the modeling.

Post reply on HN