Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

“The Unreasonable Effectiveness of Deep Learning Representations”

blog.insightdatascience.com

Post reply on HN
Print view View original

11–20 of 36 posts

Re: “The Unreasonable Effectiveness of Deep Learning Representations”

#11

Post by etaioinshrdlu »

robius wrote: post #2

What I find unreasonable is doing all this without knowing what the model is doing. It's blind with no way to steer and correct it. That is what feed forward networks and back propagation do for us. So why do we keep using them? Then there's the statistics of it all.. what are we actually modeling? 'The real world' you say? Think again. Data has to be changed and manipulated into i.i.d. form, or the algorithms won't…

I would try to remember... Just because you don't like it, doesn't mean it doesn't work. Deep learning is creating an awful lot of actual value right now, and I think we're just getting started.

Re: “The Unreasonable Effectiveness of Deep Learning Representations”

#12

Post by srean »

nerdponx wrote: post #8
robius wrote: post #2

What I find unreasonable is doing all this without knowing what the model is doing. It's blind with no way to steer and correct it. That is what feed forward networks and back propagation do for us. So why do we keep using them? Then there's the statistics of it all.. what are we actually modeling? 'The real world' you say? Think again. Data has to be changed and manipulated into i.i.d. form, or the algorithms won't…

It's because there is no formal definition of dependence in statistics. Let that sink in for a minute. What? Statistical dependence (of random variables) is defined clearly and precisely. Data has to be changed and manipulated into i.i.d. form, or the algorithms won't work Neural networks don't use the iid assumption. I downvoted you because it seems like you don't really know what you're talking about and you're cur…

Strongly agreed. It seems robius really is clue less when he/she's talking about modeling independence or modeling the lack of independence.

Re: “The Unreasonable Effectiveness of Deep Learning Representations”

#13

Post by meikos »

wadkar wrote: post #4
robius wrote: post #2

What I find unreasonable is doing all this without knowing what the model is doing. It's blind with no way to steer and correct it. That is what feed forward networks and back propagation do for us. So why do we keep using them? Then there's the statistics of it all.. what are we actually modeling? 'The real world' you say? Think again. Data has to be changed and manipulated into i.i.d. form, or the algorithms won't…

> So the math needs work Finally! I thought I was alone (and stupid) for thinking like this. Is there any literature or any meta-work that discusses the notion of probability itself? What is expectation? What is dependence?

what do you mean by the notion of probability itself?

probability was mastered far before computers were a thing

Re: “The Unreasonable Effectiveness of Deep Learning Representations”

#14

Post by yorwba »

robius wrote: post #7
kornish wrote: post #3

Earlier quoted context omitted.

Which company, and what do they have?

The company is a small startup with an amazing breakthrough called Optimizing Mind. They have magical ways of 'explaining' black box models. But it's not what DARPA is pushing (box remains black), rather the opposite, illuminating what's inside the box, making it a transparent open box. So much so, that the models they make you can edit by hand, since they make sense (to mere humans). Has rather immense implications.…

I guess by DARPA you mean https://www.darpa.mil/program/explainable-artificial-intelli... ?

It might be their crappy website, but I don't feel like Optimizing Mind is likely to create a better solution than that DARPA project. Their single "Static Demo" shows the importance of various factors in a linear regression model ... but that isn't exactly revolutionary. There might be some value in nicely packaging this for decision makers who use linear regression models but don't know how they work, but I doubt that it scales to much larger models.

Re: “The Unreasonable Effectiveness of Deep Learning Representations”

#15

Post by soVeryTired »

wadkar wrote: post #4
robius wrote: post #2

What I find unreasonable is doing all this without knowing what the model is doing. It's blind with no way to steer and correct it. That is what feed forward networks and back propagation do for us. So why do we keep using them? Then there's the statistics of it all.. what are we actually modeling? 'The real world' you say? Think again. Data has to be changed and manipulated into i.i.d. form, or the algorithms won't…

> So the math needs work Finally! I thought I was alone (and stupid) for thinking like this. Is there any literature or any meta-work that discusses the notion of probability itself? What is expectation? What is dependence?

> What is expectation?

There is a formal mathematical definition:

Let (\Omega, \mathcal{F}, P) be a probability space, and let X: \Omega -> S be a random variable taking values in some measurable space (S, \mathcal{S}).

Then the expectation is \int X(\omegs)dP

In computer science terms, do an experiment with every possible random seed and average the outcome (set \Omega to be the set of all seeds, and set P to be the uniform measure on them).

Re: “The Unreasonable Effectiveness of Deep Learning Representations”

#16

Post by avaku »

nerdponx wrote: post #8
robius wrote: post #2

What I find unreasonable is doing all this without knowing what the model is doing. It's blind with no way to steer and correct it. That is what feed forward networks and back propagation do for us. So why do we keep using them? Then there's the statistics of it all.. what are we actually modeling? 'The real world' you say? Think again. Data has to be changed and manipulated into i.i.d. form, or the algorithms won't…

It's because there is no formal definition of dependence in statistics. Let that sink in for a minute. What? Statistical dependence (of random variables) is defined clearly and precisely. Data has to be changed and manipulated into i.i.d. form, or the algorithms won't work Neural networks don't use the iid assumption. I downvoted you because it seems like you don't really know what you're talking about and you're cur…

Agree. I can't downvote, so I just agree :)

Re: “The Unreasonable Effectiveness of Deep Learning Representations”

#17

Post by akvadrako »

meikos wrote: post #13
wadkar wrote: post #4

Earlier quoted context omitted.

> So the math needs work Finally! I thought I was alone (and stupid) for thinking like this. Is there any literature or any meta-work that discusses the notion of probability itself? What is expectation? What is dependence?

what do you mean by the notion of probability itself? probability was mastered far before computers were a thing

Probability is far from clear. Very briefly, there are two main camps:

1. Bayesian probability is about degrees of belief. But that's always subjective and belief about what, if not probability? It's circular.

2. Frequentist probability is about, after X >> 1 runs of an experiment, an outcome with odds of Y occurs Y/X times. But it's only exact with an infinite number of runs, which never happens. And what's the odds of exactly Y x 1000 outcomes after 1000 runs? Again, that's circular.

My favourite way to think about probability is the multiverse kind:

3. Assuming there are an infinite number of fungible identical worlds, if a coin flip has 50% of heads, it means observers in exactly half the worlds see heads. However, this isn't actually probability at all - from a god's eye view it's objectively certain what happens.

Re: “The Unreasonable Effectiveness of Deep Learning Representations”

#18

Post by reitanqild »

akvadrako wrote: post #17
meikos wrote: post #13

Earlier quoted context omitted.

what do you mean by the notion of probability itself? probability was mastered far before computers were a thing

Probability is far from clear. Very briefly, there are two main camps: 1. Bayesian probability is about degrees of belief. But that's always subjective and belief about what , if not probability? It's circular. 2. Frequentist probability is about, after X >> 1 runs of an experiment, an outcome with odds of Y occurs Y/X times. But it's only exact with an infinite number of runs, which never happens. And what's the odd…

> Probability is far from clear. Very briefly, there are two main camps:

Isn't this a bit like saying there are two main camps when it comes to coins:

1. "heads"

2. and "tails"

?

At least to me it felt like the different forms of statistics where only different techniques.

Re: “The Unreasonable Effectiveness of Deep Learning Representations”

#19

Post by akvadrako »

reitanqild wrote: post #18
akvadrako wrote: post #17

Earlier quoted context omitted.

Probability is far from clear. Very briefly, there are two main camps: 1. Bayesian probability is about degrees of belief. But that's always subjective and belief about what , if not probability? It's circular. 2. Frequentist probability is about, after X >> 1 runs of an experiment, an outcome with odds of Y occurs Y/X times. But it's only exact with an infinite number of runs, which never happens. And what's the odd…

> Probability is far from clear. Very briefly, there are two main camps: Isn't this a bit like saying there are two main camps when it comes to coins: 1. "heads" 2. and "tails" ? At least to me it felt like the different forms of statistics where only different techniques.

As far as we know, there is no underlying theory of probability for them to be techniques of. So maybe they are equivalent in some sense, but on the face of it, they are separate ideas.

Re: “The Unreasonable Effectiveness of Deep Learning Representations”

#20

Post by wadkar »

akvadrako wrote: post #17
meikos wrote: post #13

Earlier quoted context omitted.

what do you mean by the notion of probability itself? probability was mastered far before computers were a thing

Probability is far from clear. Very briefly, there are two main camps: 1. Bayesian probability is about degrees of belief. But that's always subjective and belief about what , if not probability? It's circular. 2. Frequentist probability is about, after X >> 1 runs of an experiment, an outcome with odds of Y occurs Y/X times. But it's only exact with an infinite number of runs, which never happens. And what's the odd…

Lets not forget the non Kolmogorovian notion of probability - or quantum mechanics. I personally believe that we would want to accommodate a more generalized notion of probability to significantly improve our statistical models of the world. You certainly hint at it in #3
Post reply on HN