Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Deep Forest: Towards an Alternative to Deep Neural Networks

arxiv.org

Post reply on HN
Print view View original

81–90 of 100 posts

Re: Deep Forest: Towards an Alternative to Deep Neural Networks

#81

Post by CuriouslyC »

deong wrote: post #63
CuriouslyC wrote: post #24

Earlier quoted context omitted.

That doesn't apply to an ensemble of algorithms where the weights of a given member of the ensemble are adapted based on observations from the given domain. If it did, humans wouldn't be able to choose a good algorithm for specific cases, and obviously we can. Deep neural networks can be thought of as ensembles of smaller neural networks, though of course each member of the ensemble is going to share some degree of a…

> If it did, humans wouldn't be able to choose a good algorithm for specific cases, and obviously we can. This is a surprisingly commonly held fallacy in some AI circles. It's the idea that humans are mathematically perfect. When you phrase it that way, it's fairly obviously false, but you still see a lot of people argue things like "NFL doesn't apply to ensembles because humans..." or "machines can never be as intel…

It doesn't imply that humans are mathematically perfect, but our brains are basically algorithm generating algorithms - we're not just weighting a set of preexisting solutions. To generalize quite a bit, saying NFL applies to general intelligence ends up implying there are problems for which efficient algorithms exist, but intelligence is literally incapable of discovering. That seems pretty absurd to me.

Re: Deep Forest: Towards an Alternative to Deep Neural Networks

#82

Post by photoJ »

throw_away_777 wrote: post #75
BickNowstrom wrote: post #65

Earlier quoted context omitted.

The hype for neural networks is deserved. Some major contributions to the field resulted in increases in accuracy for fields like NLP, computer vision, structured data, machine translation, style transfer, etc. XGBoost did not change much from the "Greedy function approximation: A gradient boosting machine." paper, but uses a few tricks to be much much faster, allowing for better tuning. XGBoost is popular for struct…

I agree that neural nets are state-of-the-art and do quite well on certain types of problems (NLP and vision, which are important problems). But a lot of data is structured (sales, churn, recommendations, etc), and it is so much easier to train an xgboost model than a neural net model. You need a very expensive computer or expensive cloud computing to train neural nets, and even then it is not easy. Ease of implement…

How can you say that ease of implementation is overlooked in academia, when academia created the exact tools your are speaking of?

Re: Deep Forest: Towards an Alternative to Deep Neural Networks

#83

Post by igul222 »

gwern wrote: post #58
gcr wrote: post #50

Earlier quoted context omitted.

GAN training is still spooky mysterious and can easily fail in nonintuitive ways. Sometimes GANs converge or not depending on the random number seed, even with the same hyperparameters.

I'm not sure about that. The new GAN models over the past 2-3 months, like LS-GAN or WGAN, all seem to train much more stably. I've beaten up on WGAN with all sorts of strange tweaks and hyperparameter settings and while it may not work well, it's never catastrophically diverged on me the way DCGAN would at the drop of a hat.

Try removing BN from the critic :)

Re: Deep Forest: Towards an Alternative to Deep Neural Networks

#84

Post by BickNowstrom »

photoJ wrote: post #82
throw_away_777 wrote: post #75

Earlier quoted context omitted.

I agree that neural nets are state-of-the-art and do quite well on certain types of problems (NLP and vision, which are important problems). But a lot of data is structured (sales, churn, recommendations, etc), and it is so much easier to train an xgboost model than a neural net model. You need a very expensive computer or expensive cloud computing to train neural nets, and even then it is not easy. Ease of implement…

How can you say that ease of implementation is overlooked in academia, when academia created the exact tools your are speaking of?

Academia does not have to run the model in production. It has few computational constraints, and most datasets do not have a feedback loop, requiring combating drift, debugging, and retraining. Papers are often accepted when they equal or beat state-of-the-art. Not many academics have to deal with the business side of running models in prod.

All of this leads to ease of implementation being overlooked. Especially on NLP, you see a lot of overengineering with deep neural nets (where the feature engineering is hidden inside the architecture). These models are hard to implement/reuse.

But yeah: academia/theoretical machine learning creates the very tools for applied machine learning.

Re: Deep Forest: Towards an Alternative to Deep Neural Networks

#85

Post by edixon »

None of these experiments actually do anything to show feature learning - if this is the claim, I would like to see a transfer learning experiment. I would be surprised if this works well, since they can't jointly optimize their layers (so you can't just use ImageNet to induce a good representation). Not quite clear why we should think that trees will turn out to be inherently cheaper that a DNN with similar accuracy, unless perhaps the model structure encodes a prior which matches the distribution of the problem?

Re: Deep Forest: Towards an Alternative to Deep Neural Networks

#88

Post by sdenton4 »

snovv_crash wrote: post #76
modeless wrote: post #56

Earlier quoted context omitted.

> One doesn't need to train a deep nn for recognizing structured objects like a coke can in a fridge. I disagree. Sure, you don't need a NN to recognize one Coke can in one fridge for your toy robot project. If you want to recognize all Coke cans in all fridges, for your real-world, consumer-ready Coke-fetching robot product? You're going to need a huge dataset of all the various designs of Coke cans out there, in al…

Which is why you should do stereo or SfM, make a 3d reconstruction, and then do HOG or some 3D feature to recognise the coke can. Trying to do it from images with a NN that doesn't comprehend 3D space is just silly.

I'm not sure if you're serious or throwing some very excellent shade.

Re: Deep Forest: Towards an Alternative to Deep Neural Networks

#89

Post by Houshalter »

deong wrote: post #61
Houshalter wrote: post #38

Earlier quoted context omitted.

There is a theoretically optimal machine learning algorithm called Solomonoff Induction. Solomonoff Induction assumes a prior over all possible computer programs that could have produced the data. And it assumes that shorter, simpler programs are more likely than longer, more complex ones. Under such a prior, the no free lunch theorem doesn't apply. NFLT assumes that all machine learning problems have infinite inform…

Solomonoff Induction isn't really an "algorithm" in the way we normally think of algorithms, as it isn't computable. There are computable approximations, but at that point, you lose the claim of theoretically optimal.

This is pedantic and I'm not even certain correct. Full SI will run forever without returning an answer, sure. But as it runs it will get closer and closer to the true answer. Eventually it will approach an answer within whatever degree of precision you want. And it will be more accurate than any alternative algorithm, so it's still optimal.
Post reply on HN