“The Unreasonable Effectiveness of Deep Learning Representations” • Hacker News Board

Re: “The Unreasonable Effectiveness of Deep Learning Representations”

#31

Post by bglazer » Fri, Jul 06, 2018, 5:29 PM UTC

kujaomega wrote: post #26

I see a good explanation of the problem and a good evolution of the done steps. But I see a problem in the approach. When you are getting the most similar result, you are supposed to compute high cosine similarity between all the embeddings. If you have more than a billion of embeddings and the embeddings have 1k dimensions, it will take a lot of time. How would you solve this problem? Clustering the embeddings?

This is generally called the k-Nearest Neighbors problem. You should check out the various data structures for doing this, like the ball tree and kd tree.

Re: “The Unreasonable Effectiveness of Deep Learning Representations”

#32

Post by yorwba » Fri, Jul 06, 2018, 5:34 PM UTC

bjourne wrote: post #24

nerdponx wrote: post #8

Earlier quoted context omitted.

It's because there is no formal definition of dependence in statistics. Let that sink in for a minute. What? Statistical dependence (of random variables) is defined clearly and precisely. Data has to be changed and manipulated into i.i.d. form, or the algorithms won't work Neural networks don't use the iid assumption. I downvoted you because it seems like you don't really know what you're talking about and you're cur…

All statistical learning, which neural nets are a form of, use the iid assumption. See https://stats.stackexchange.com/questions/213464/on-the-impo...

That kind of iid assumption could be summarized as "the training data is representative of the data we want to apply the model to", and if it doesn't hold, that's indeed a problem.

But "Data has to be changed and manipulated into i.i.d. form, or the algorithms won't work. How does an independent set of random variables give us a model of the actual dataset which is a very limited representation of the real world?" strongly implies that the data itself should be decomposed into iid variables. While whitening ("manipulating into iid form") is a common preprocessing technique because it's simple and effective, that doesn't mean that learning algorithms wouldn't work without it. They'd just take a bit longer to arrive at the same result.

Re: “The Unreasonable Effectiveness of Deep Learning Representations”

#33

Post by bunderbunder » Fri, Jul 06, 2018, 5:53 PM UTC

kujaomega wrote: post #26

I see a good explanation of the problem and a good evolution of the done steps. But I see a problem in the approach. When you are getting the most similar result, you are supposed to compute high cosine similarity between all the embeddings. If you have more than a billion of embeddings and the embeddings have 1k dimensions, it will take a lot of time. How would you solve this problem? Clustering the embeddings?

There are off-the-shelf libraries like ANNOY and nmslib that index the vectors in a way that allows for fast (possibly approximate) nearest neighbors searches.

Re: “The Unreasonable Effectiveness of Deep Learning Representations”

#34

Post by kgwgk » Fri, Jul 06, 2018, 7:39 PM UTC

kgwgk wrote: post #29

chriselrod wrote: post #21

Earlier quoted context omitted.

Your "3" is a Bayesian view. Specifically, from the Jaynesian school, which views probability as ignorance. When we can't calculate which of those world's we're in, we express our remaining uncertainty with probability. The connection to subjective "beliefs" is recognizing that these probabilities are all in our own heads. Believing otherwise is the "mind projection fallacy"; in reality -- as you noted -- these thing…

> Your "3" is a Bayesian view. Except for the infinite number of universes nonsense :-)

Dear downvoters: Even if we assume an infinite number of (real or imaginary) universes, what does “in exactly half of the worlds" mean? This definition doesn’t seem at all less problematic that the usual ones.

Re: “The Unreasonable Effectiveness of Deep Learning Representations”

#35

Post by perl4ever » Sat, Jul 07, 2018, 4:30 AM UTC

kgwgk wrote: post #34

kgwgk wrote: post #29

Earlier quoted context omitted.

> Your "3" is a Bayesian view. Except for the infinite number of universes nonsense :-)

Dear downvoters: Even if we assume an infinite number of (real or imaginary) universes, what does “in exactly half of the worlds" mean? This definition doesn’t seem at all less problematic that the usual ones.

I think "infinite" is just sloppy language. If every possible universe exists in some sense, that is a large number, but not infinite - because nothing about a universe has infinite precision. Thus, "half" would still mean something.

Re: “The Unreasonable Effectiveness of Deep Learning Representations”

#36

Post by perl4ever » Sat, Jul 07, 2018, 4:32 AM UTC

reitanqild wrote: post #18

akvadrako wrote: post #17

Earlier quoted context omitted.

Probability is far from clear. Very briefly, there are two main camps: 1. Bayesian probability is about degrees of belief. But that's always subjective and belief about what , if not probability? It's circular. 2. Frequentist probability is about, after X >> 1 runs of an experiment, an outcome with odds of Y occurs Y/X times. But it's only exact with an infinite number of runs, which never happens. And what's the odd…

> Probability is far from clear. Very briefly, there are two main camps: Isn't this a bit like saying there are two main camps when it comes to coins: 1. "heads" 2. and "tails" ? At least to me it felt like the different forms of statistics where only different techniques.

I don't even understand how the frequentist view is a valid alternative. It always seemed to me like either you are honest about your priors, and use Bayesian logic to take them into account, or you sweep it under the rug. Lying to yourself always produces bad results, is my overriding heuristic. But I'm not good at math.