Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why does deep and cheap learning work so well?

arxiv.org

Post reply on HN
Print view View original

41–50 of 53 posts

Re: Why does deep and cheap learning work so well?

#41

Post by bcherny »

chatmasta wrote: post #33

I really like this kind of cross-disciplinary research and knowledge transfer. It should happen more often. It’s interesting that so many equations governing different laws in different fields actually share quite a few properties, and on deeper analysis, can be explained by a single mathematical property. It makes me wonder how many insights we are missing simply because they were discovered in another field, under…

I recall a project (out of MIT I think) that used category theory and knowledge mapping to discover isomorphism between the knowledge graphs of different fields like math, physics, and economics.

Re: Why does deep and cheap learning work so well?

#42

Post by mitchtbaum »

bcherny wrote: post #41
chatmasta wrote: post #33

I really like this kind of cross-disciplinary research and knowledge transfer. It should happen more often. It’s interesting that so many equations governing different laws in different fields actually share quite a few properties, and on deeper analysis, can be explained by a single mathematical property. It makes me wonder how many insights we are missing simply because they were discovered in another field, under…

I recall a project (out of MIT I think) that used category theory and knowledge mapping to discover isomorphism between the knowledge graphs of different fields like math, physics, and economics.

I found something similar with topology, graph theory, ...

links??

Re: Why does deep and cheap learning work so well?

#43

Post by mitchtbaum »

chatmasta wrote: post #33

I really like this kind of cross-disciplinary research and knowledge transfer. It should happen more often. It’s interesting that so many equations governing different laws in different fields actually share quite a few properties, and on deeper analysis, can be explained by a single mathematical property. It makes me wonder how many insights we are missing simply because they were discovered in another field, under…

As above, so below.

As the Universe, so the Soul.

https://www.youtube.com/watch?v=I3eancQx5pI

GnosticMedia - Alchemy and the Endocrine System - an interview with Jose Barrera

Re: Why does deep and cheap learning work so well?

#44

Post by yodon »

electricslpnsld wrote: post #28

I am not a Physicist (IANAP), so maybe I’m misrecalling some definitions, but isn’t the restriction to Hamiltonians a bit, well, restrictive? This limits the results to path independent potentials, which is basically nothing in the non-spherical cow world. Are the authors working from a different definition of Hamiltonian? Does it matter if your Hamiltonian is smooth or if you are working from a discrete theory?

If you are asking the neural network to make a discrete classification or discrete prediction about some set of input data, you are almost by definition asking it a Hamiltonian class of question.

By way of a counter example, you could say “but it depends whether the person walked or took the bus!” In that case either you need to provide the data on whether they walked or took the bus (in which case your question collapses to a simple Hamiltonian form with the additional data) or you don’t provide the data and the question is fundamentally unanswerable in a simple discrete way.

Re: Why does deep and cheap learning work so well?

#45

Post by jdonaldson »

ralusek wrote: post #24
jdonaldson wrote: post #21

Love to see more papers like this. I remember a recent paper that showed decent model performance when the model was allowed to pick its own activation functions. It was picking wacky things like sine waves. You'd like to learn something about the model by the way it configures itself, but right now we can only understand simple model features.

That's so weird, I can't help but feel like that would only ever result in it coming up with something that initially got it a better result and the rest of the network adjusted around it and is able to carry the weight of that weirdness. What IS interesting is that it shows just how resilient the networks that come out of this really are that they can support such weirdo activations. But there are other elements of…

Yeah, I agree about smarter parameter search and training. That was a big theme at ICLR this year.

Fwiw, here's the paper on RNN architecture generation referenced earlier.

https://arxiv.org/abs/1712.07316

Re: Why does deep and cheap learning work so well?

#46

Post by throwaway98321 »

mitchtbaum wrote: post #43
chatmasta wrote: post #33

I really like this kind of cross-disciplinary research and knowledge transfer. It should happen more often. It’s interesting that so many equations governing different laws in different fields actually share quite a few properties, and on deeper analysis, can be explained by a single mathematical property. It makes me wonder how many insights we are missing simply because they were discovered in another field, under…

As above, so below. As the Universe, so the Soul. https://www.youtube.com/watch?v=I3eancQx5pI GnosticMedia - Alchemy and the Endocrine System - an interview with Jose Barrera

Despite the predictable down-votes, I wanted to say that I greatly enjoyed that interview.

Re: Why does deep and cheap learning work so well?

#48

Post by zerostar07 »

crb002 wrote: post #16

Learning a transformation from the full transformation semigroup is the most general case. Consider a mystery unary CPU operation M on a 64 bit register. How deep of a circuit do you need to calculate any such transformation M? According to Kolmogorov just writing out a random transformation function takes (2^64)*(64) bits to make the lookup table. log of that to get a result out efficiently. Results here were alread…

it's not the same results, their point was to justify it on physical grounds

Re: Why does deep and cheap learning work so well?

#49

Post by sitkack »

vanderZwan wrote: post #37
akvadrako wrote: post #34

Earlier quoted context omitted.

Indeed, the connections seem profound. It seems to be a general-purpose optimal algorithm for, well, optimisation. And that would explain why the universe, our brains and AIs all trend toward it. It could also be just that intelligence tends to mirror the outside world, but that seems a bit arbitrary.

> It could also be just that intelligence tends to mirror the outside world, but that seems a bit arbitrary. We are part of the universe, so why would it be arbitrary if our brains were structured in ways that match typical structures found in this universe?

https://en.wikipedia.org/wiki/Panpsychism

Re: Why does deep and cheap learning work so well?

#50

Post by vanderZwan »

sitkack wrote: post #49
vanderZwan wrote: post #37

Earlier quoted context omitted.

> It could also be just that intelligence tends to mirror the outside world, but that seems a bit arbitrary. We are part of the universe, so why would it be arbitrary if our brains were structured in ways that match typical structures found in this universe?

https://en.wikipedia.org/wiki/Panpsychism

I was thinking more along the lines that if our minds process outside information in a way that makes sense of that information, partially through simulating it, it does not seem so strange if the structures end up matching the outside structures through some form of convergent evolution.

From a cursory reading of that article I do not see it argue the same thing.

Post reply on HN