Live data from Hacker News

Deep Forest: Towards an Alternative to Deep Neural Networks

arxiv.org

91–100 of 100 posts

Re: Deep Forest: Towards an Alternative to Deep Neural Networks

#91
post #58

Earlier quoted context omitted.

I'm not sure about that. The new GAN models over the past 2-3 months, like LS-GAN or WGAN, all seem to train much more stably. I've beaten up on WGAN with all sorts of strange tweaks and hyperparameter settings and while it may not work well, it's never catastrophically diverged on me the way DCGAN would at the drop of a hat.

Have you found any good ways to speed it up? The five-fold training on the Critic is very expensive.

No, not yet. I suspect that increasing the discriminator-only learning rate might help but haven't tried.

Re: Deep Forest: Towards an Alternative to Deep Neural Networks

#92
Please note the CPUs they have used are pretty advanced: 2x Intel E5 2670 v3 CPU (24 cores) - approx. price $1.5k per unit (http://ark.intel.com/products/81709/Intel-Xeon-Processor-E5-...).

Looking forward to try the code (especially on CIFAR or ImageNet), Zhi-Hua Zhou, one of the authors, said they are going to publish it soon.

Re: Deep Forest: Towards an Alternative to Deep Neural Networks

#93

Earlier quoted context omitted.

Which is why you should do stereo or SfM, make a 3d reconstruction, and then do HOG or some 3D feature to recognise the coke can. Trying to do it from images with a NN that doesn't comprehend 3D space is just silly.

I'm not sure if you're serious or throwing some very excellent shade.

I'm serious. If you have to rely on mono, single image inputs then yeah ImageNet is going to do better. But it will also mistake every picture of a coke can as the real thing. It will be horrifically sensitive to malicious inputs. Much better would be to use 2 calibrated lenses and do 3D reconstruction. Even if you're just doing the reconstruction as a sanity check for a NN to weed out the false positives.

Re: Deep Forest: Towards an Alternative to Deep Neural Networks

#94
post #77

Earlier quoted context omitted.

Which is why you should do stereo or SfM, make a 3d reconstruction, and then do HOG or some 3D feature to recognise the coke can. Trying to do it from images with a NN that doesn't comprehend 3D space is just silly.

Errm, hang on, are you saying that if you have a task of classifying unseen images given a labelled training set you should get a stereo camera or video camera and create another problem? Which you can solve? Because the problem is silly> What if I say : "I will give you $10m to solve it, and if you fail, I will kill this very kind old monkey?"

Object recognition doesn't only exist in the subspace of labelled 2D images. It tends to be derived from a 3D space, which is a whole extra orthogonal data source that the "NN all the things" crowd is fastidiously ignoring.

Why, I'm not sure, but I'm guessing because it is hard/inaccurate to do with just NNs and parameter/network architecture tweaking. Possibly also because benchmarks with single mono images are much easier to make.

Just because it is hard with method A, and is harder to make benchmarks, doesn't mean method B isn't better.

Re: Deep Forest: Towards an Alternative to Deep Neural Networks

#95
post #77

Earlier quoted context omitted.

Errm, hang on, are you saying that if you have a task of classifying unseen images given a labelled training set you should get a stereo camera or video camera and create another problem? Which you can solve? Because the problem is silly> What if I say : "I will give you $10m to solve it, and if you fail, I will kill this very kind old monkey?"

Object recognition doesn't only exist in the subspace of labelled 2D images. It tends to be derived from a 3D space, which is a whole extra orthogonal data source that the "NN all the things" crowd is fastidiously ignoring. Why, I'm not sure, but I'm guessing because it is hard/inaccurate to do with just NNs and parameter/network architecture tweaking. Possibly also because benchmarks with single mono images are much…

Yes but am I missing something when I say that if the problem is to deal with labelled 2d images declaring that you should be working with 3d images or short video sequences doesn't help.

Sure, if you are building a Robot and I say "use this camera and a deep network" and you say "It'll work better with stereo" well... yes super do that!

But if we are working with mono images I don't understand how the observation helps?

Re: Deep Forest: Towards an Alternative to Deep Neural Networks

#96
post #14

"In contrast to deep neural networks which require great effort in hyper-parameter tuning, gcForest is much easier to train." Hyperparameter tuning is not as much of an issue with deep neural networks anymore. Thanks to BatchNorm and more robust optimization algorithms, most of the time you can simply use Adam with a default learning rate of 0.001 and do pretty well. Dropout is not even necessary with many models tha…

No batch norm for LSTMs

Totally agree, but RF's are more related to "CNN-ish problems" (image classification and...?), not RNNs, or generally, any graphical sequence model.

EDIT: to clarify: "j/k" with the thing in parenthesis ;-)

Re: Deep Forest: Towards an Alternative to Deep Neural Networks

#97
post #95

Earlier quoted context omitted.

Object recognition doesn't only exist in the subspace of labelled 2D images. It tends to be derived from a 3D space, which is a whole extra orthogonal data source that the "NN all the things" crowd is fastidiously ignoring. Why, I'm not sure, but I'm guessing because it is hard/inaccurate to do with just NNs and parameter/network architecture tweaking. Possibly also because benchmarks with single mono images are much…

Yes but am I missing something when I say that if the problem is to deal with labelled 2d images declaring that you should be working with 3d images or short video sequences doesn't help. Sure, if you are building a Robot and I say "use this camera and a deep network" and you say "It'll work better with stereo" well... yes super do that! But if we are working with mono images I don't understand how the observation he…

Quoting from GP of your original reply:

> If you want to recognize all Coke cans in all fridges, for your real-world, consumer-ready Coke-fetching robot product?

If you're stuck with a mono dataset, post collection, then sure use NN and call it a day. But even if you have video you can do 3D reconstruction just from baseline movement. You won't know scale, so you can't differentiate between big coke cans and little coke cans, but at least you can rule out pictures of coke cans.

Re: Deep Forest: Towards an Alternative to Deep Neural Networks

#98

Earlier quoted context omitted.

That's quite literally deep learning so made me lol.

Can you elaborate on that? I feel like I've missed something.

Neural networks are just networks of logistic regressors.

Re: Deep Forest: Towards an Alternative to Deep Neural Networks

#99
post #63

Earlier quoted context omitted.

> If it did, humans wouldn't be able to choose a good algorithm for specific cases, and obviously we can. This is a surprisingly commonly held fallacy in some AI circles. It's the idea that humans are mathematically perfect. When you phrase it that way, it's fairly obviously false, but you still see a lot of people argue things like "NFL doesn't apply to ensembles because humans..." or "machines can never be as intel…

It doesn't imply that humans are mathematically perfect, but our brains are basically algorithm generating algorithms - we're not just weighting a set of preexisting solutions. To generalize quite a bit, saying NFL applies to general intelligence ends up implying there are problems for which efficient algorithms exist, but intelligence is literally incapable of discovering. That seems pretty absurd to me.

The problem is that if our brains are not "magic" (essentially making this a religious argument), then they're operating via some algorithmic principles. If our brains are "algorithm generating algorithms", then I can in principle write an "algorithm generating algorithm" in silicon that does the same thing. And I know with mathematical certainty that my digital version is subject to NFL. So either my brain is too or we go back to religion to explain what non-physical process is responsible for our super-turing capabilities.

The second point is that NFL is often interpreted in a weird way, where we only think about "interesting problems". It is defined on the set of all search or optimization problems. It says if your algorithm is better than exhaustive search over some subset of problems, it must be worse on the complement of that set.

What does it mean for an algorithm to be efficient? Well, it's roughly speaking the number of steps it needed to take (assuming each step is the same amount of work, blah blah). OK, so an "efficient" algorithm must, by definition, prioritize some steps over others -- it's picking the "best" steps to take each time it has the choice. OK, so I'll just make up some instances of the problem that are custom-tailored so that what the algorithm thinks are the "best" steps always lead me in the wrong direction. You algorithm will then be worse than exhaustive search on my set of problems, precisely because it's choosing to avoid the steps I know to be good -- I defined the problem to make that happen.

That is true of any algorithm you can conceive of. There will be problems for which the bias that makes the algorithm good on the problems you intended it to work on will be exactly the wrong bias.

It doesn't matter if you say, "Aha, I'll let my algorithm generate new algorithms! Gotcha!" I'll just design a set of problems for which your algorithm generating algorithm will generate the wrong algorithms.

Search is always about bias. Without bias, you have random search -- that's literally the textbook machine learning definition of bias. All NFL says is that if you have to worry about every possible problem, any bias you choose will be worse than random sometimes. There's no escape clause here. Your algorithm generating algorithm is still a search algorithm with its own biases, and it will still be worse than random on some subset of all possible problems.

Re: Deep Forest: Towards an Alternative to Deep Neural Networks

#100

No Free Lunch theorem refesher: "if an algorithm performs well on a certain class of problems then it necessarily pays for that with degraded performance on the set of all remaining problems" https://en.m.wikipedia.org/wiki/No_free_lunch_theorem

There is a theoretically optimal machine learning algorithm called Solomonoff Induction. Solomonoff Induction assumes a prior over all possible computer programs that could have produced the data. And it assumes that shorter, simpler programs are more likely than longer, more complex ones. Under such a prior, the no free lunch theorem doesn't apply. NFLT assumes that all machine learning problems have infinite inform…

Isn't Solomonoff Induction only given a Free Lunch if its priors are true? What if its priors are false?
Post reply on HN