Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

“The Unreasonable Effectiveness of Deep Learning Representations”

blog.insightdatascience.com

Post reply on HN
Print view View original

21–30 of 36 posts

Re: “The Unreasonable Effectiveness of Deep Learning Representations”

#21

Post by chriselrod »

akvadrako wrote: post #17
meikos wrote: post #13

Earlier quoted context omitted.

what do you mean by the notion of probability itself? probability was mastered far before computers were a thing

Probability is far from clear. Very briefly, there are two main camps: 1. Bayesian probability is about degrees of belief. But that's always subjective and belief about what , if not probability? It's circular. 2. Frequentist probability is about, after X >> 1 runs of an experiment, an outcome with odds of Y occurs Y/X times. But it's only exact with an infinite number of runs, which never happens. And what's the odd…

Your "3" is a Bayesian view. Specifically, from the Jaynesian school, which views probability as ignorance. When we can't calculate which of those world's we're in, we express our remaining uncertainty with probability. The connection to subjective "beliefs" is recognizing that these probabilities are all in our own heads. Believing otherwise is the "mind projection fallacy"; in reality -- as you noted -- these things are certain from the god's eye view, and we fall somewhere in between that and total ignorance/entropy. (I'm not a physicist, but I know some use the Many Worlds interpretation to apply this determinism even to quantum physics.)

E.T. Jaynes fleshes out his worldview in "Probability Theory: The Logic of Science", which was published posthumously in 2003.

Re: “The Unreasonable Effectiveness of Deep Learning Representations”

#22

Post by JoeAltmaier »

Looking at pictures and categorizing them by appearance has very limited application. The human context can't be in grasped that way, can it?

For instance, 'wedding pictures'. A cake being cut; a cute kid throwing flower petals; a black-clad clergyman; a hand with a ring on it. Any human could categorize a pile of pictures into those that are in the 'wedding' category, and those that aren't. But no strategy based on weighting pixels is ever going to get there.

Likewise, 'cute' or 'scary' or 'funny'. And on and on.

Re: “The Unreasonable Effectiveness of Deep Learning Representations”

#23

Post by beagle3 »

meikos wrote: post #13
wadkar wrote: post #4

Earlier quoted context omitted.

> So the math needs work Finally! I thought I was alone (and stupid) for thinking like this. Is there any literature or any meta-work that discusses the notion of probability itself? What is expectation? What is dependence?

what do you mean by the notion of probability itself? probability was mastered far before computers were a thing

Probability was understood long before computers, true, but it waited for Kolmogorov’s axiomatic formulation to actually become the coherent field of math that it is today, rather than a hodge lodge of definitions, tricks and theorems.

And that only happened in 1933, which is around the time that computers became a thing. Not general purpose ones yet - I agree it was before computers were widespread, but definitely not far before they were a thing.

Re: “The Unreasonable Effectiveness of Deep Learning Representations”

#24

Post by bjourne »

nerdponx wrote: post #8
robius wrote: post #2

What I find unreasonable is doing all this without knowing what the model is doing. It's blind with no way to steer and correct it. That is what feed forward networks and back propagation do for us. So why do we keep using them? Then there's the statistics of it all.. what are we actually modeling? 'The real world' you say? Think again. Data has to be changed and manipulated into i.i.d. form, or the algorithms won't…

It's because there is no formal definition of dependence in statistics. Let that sink in for a minute. What? Statistical dependence (of random variables) is defined clearly and precisely. Data has to be changed and manipulated into i.i.d. form, or the algorithms won't work Neural networks don't use the iid assumption. I downvoted you because it seems like you don't really know what you're talking about and you're cur…

All statistical learning, which neural nets are a form of, use the iid assumption. See https://stats.stackexchange.com/questions/213464/on-the-impo...

Re: “The Unreasonable Effectiveness of Deep Learning Representations”

#25

Post by jchw »

JoeAltmaier wrote: post #22

Looking at pictures and categorizing them by appearance has very limited application. The human context can't be in grasped that way, can it? For instance, 'wedding pictures'. A cake being cut; a cute kid throwing flower petals; a black-clad clergyman; a hand with a ring on it. Any human could categorize a pile of pictures into those that are in the 'wedding' category, and those that aren't. But no strategy based on…

There are definitely networks in our brain that classify what we're seeing. An uneducated guess, though, is that the networks in our brains have many, many intermediate representations and don't go directly from image -> words, but rather go to abstract classifiers that can go back to words.

Re: “The Unreasonable Effectiveness of Deep Learning Representations”

#26

Post by kujaomega »

I see a good explanation of the problem and a good evolution of the done steps. But I see a problem in the approach. When you are getting the most similar result, you are supposed to compute high cosine similarity between all the embeddings. If you have more than a billion of embeddings and the embeddings have 1k dimensions, it will take a lot of time. How would you solve this problem? Clustering the embeddings?

Re: “The Unreasonable Effectiveness of Deep Learning Representations”

#27

Post by bunderbunder »

bjourne wrote: post #24
nerdponx wrote: post #8

Earlier quoted context omitted.

It's because there is no formal definition of dependence in statistics. Let that sink in for a minute. What? Statistical dependence (of random variables) is defined clearly and precisely. Data has to be changed and manipulated into i.i.d. form, or the algorithms won't work Neural networks don't use the iid assumption. I downvoted you because it seems like you don't really know what you're talking about and you're cur…

All statistical learning, which neural nets are a form of, use the iid assumption. See https://stats.stackexchange.com/questions/213464/on-the-impo...

But they use it in different ways. For example, an ARMA model is specifically looking for dependencies among the data points, so there assuming iid among them would be an absurdity. In time series analysis, you're looking for the model's residuals, not the source data, to be independent and identically distributed.

Also, in real-world statistical modeling, there's nuance. Just like for any assumption of a parametric model, the data not being iid doesn't mean that the model is 100% crap, it means that you can't draw specific conclusions about the quality of the model.

Which is fine, because maybe you don't care to draw those conclusions, anyway. One of the key differences between machine learning and traditional statistical analysis is that you aren't so worried about developing parsimonious models with well-defined parameters. You're typically just empirically interested in the model's predictive or descriptive utility. This difference isn't a result of one school being more principled and the other being more lackadaisical. It's reflective of differing goals: One approach was developed for use in scientific hypothesis testing, where your primary deliverable is (in the case of something like regression, anyway) the model's parameters, and its estimates are a means to evaluate those parameters. The other approach is used for modeling processes, where the primary deliverable is the estimates, and the parameters are a means to get those estimates.

Re: “The Unreasonable Effectiveness of Deep Learning Representations”

#28

Post by eli_gottlieb »

akvadrako wrote: post #17
meikos wrote: post #13

Earlier quoted context omitted.

what do you mean by the notion of probability itself? probability was mastered far before computers were a thing

Probability is far from clear. Very briefly, there are two main camps: 1. Bayesian probability is about degrees of belief. But that's always subjective and belief about what , if not probability? It's circular. 2. Frequentist probability is about, after X >> 1 runs of an experiment, an outcome with odds of Y occurs Y/X times. But it's only exact with an infinite number of runs, which never happens. And what's the odd…

Probability is just measure theory where total measure norms to 1.0 ;-).

Re: “The Unreasonable Effectiveness of Deep Learning Representations”

#29

Post by kgwgk »

chriselrod wrote: post #21
akvadrako wrote: post #17

Earlier quoted context omitted.

Probability is far from clear. Very briefly, there are two main camps: 1. Bayesian probability is about degrees of belief. But that's always subjective and belief about what , if not probability? It's circular. 2. Frequentist probability is about, after X >> 1 runs of an experiment, an outcome with odds of Y occurs Y/X times. But it's only exact with an infinite number of runs, which never happens. And what's the odd…

Your "3" is a Bayesian view. Specifically, from the Jaynesian school, which views probability as ignorance. When we can't calculate which of those world's we're in, we express our remaining uncertainty with probability. The connection to subjective "beliefs" is recognizing that these probabilities are all in our own heads. Believing otherwise is the "mind projection fallacy"; in reality -- as you noted -- these thing…

> Your "3" is a Bayesian view.

Except for the infinite number of universes nonsense :-)

Re: “The Unreasonable Effectiveness of Deep Learning Representations”

#30

Post by breatheoften »

eli_gottlieb wrote: post #28
akvadrako wrote: post #17

Earlier quoted context omitted.

Probability is far from clear. Very briefly, there are two main camps: 1. Bayesian probability is about degrees of belief. But that's always subjective and belief about what , if not probability? It's circular. 2. Frequentist probability is about, after X >> 1 runs of an experiment, an outcome with odds of Y occurs Y/X times. But it's only exact with an infinite number of runs, which never happens. And what's the odd…

Probability is just measure theory where total measure norms to 1.0 ;-).

Is there a way to confirm this experimentally?
Post reply on HN