Michael I. Jordan was a mentor to Andrew Ng. Probabilistic Graphical Models are the next frontier in AI after deep learning.
An Introduction to Probabilistic Graphical Models (2003) [pdf]
11–20 of 37 posts
Re: An Introduction to Probabilistic Graphical Models (2003) [pdf]
#12Earlier quoted context omitted.
Not an expert but PGMs are mostly a superset of NN, so it's kinda understandable.
What is the connection other than "they both have graphs somewhere in them"? I sort of see what you mean since NNs transport information across a graph in a straightforward (non-loopy) way, and PGMs can propagate information in crazy (neverending loops) ways when doing belief propagation...
Not just graphs but also probability.
Re: An Introduction to Probabilistic Graphical Models (2003) [pdf]
#13Earlier quoted context omitted.
Not an expert but PGMs are mostly a superset of NN, so it's kinda understandable.
What is the connection other than "they both have graphs somewhere in them"? I sort of see what you mean since NNs transport information across a graph in a straightforward (non-loopy) way, and PGMs can propagate information in crazy (neverending loops) ways when doing belief propagation...
From my limited understanding, pure NNs are not able to express confidence in predictions. By adding probability to the NN, we can have both predictions and confidence scores. In practice, noise is being injected, or connections dropped out randomly, then predictions averaged over multiple runs.
Re: An Introduction to Probabilistic Graphical Models (2003) [pdf]
#14A few comments have mentioned neural nets in this post. adamnemecek mentions in this thread that PGMs are a superset of neural networks, and and Thomas Wiecki has a few excellent blog posts on creating bayesian neural networks using pymc3.[0][1][2] If you're curious about how these two concepts can be brought together I highly recommend reading through these three posts. [0] http://twiecki.github.io/blog/2016/06/01/b…
Do they train with backpropagation efficiently?
Re: An Introduction to Probabilistic Graphical Models (2003) [pdf]
#15PGM's are great, but my experience from Koller's course is that it is very hard to identify cases where they can be used. Part of the reason is that you need a-priori knowledge of the causal relationships (coarse grained I.e direction) between your variables. Presumably if you're doing ML you don't know those causal relationships to begin with. Particularly good fits are things like physics where laws are known.
A heuristic guide...If you have more data than knowledge about the domain, and you want to do classification/prediction, then NNs are a good. PGMs are good if you have knowledge that is important to encode; you need modularity i.e. you don't want to embed priors in the model; you have strong causal relationships e.g. diagnostic models with 'explaining away'; you want to integrate value of information e.g. what test s…
by complexity of inference do you mean the complexity of learning the structure of a model ? because inference on an existing PGM is linear in the number of edges with belief-propagation isn't it ?
Re: An Introduction to Probabilistic Graphical Models (2003) [pdf]
#16Good article: "Big-data boondoggles and brain-inspired chips are just two of the things we’re really getting wrong" - Michael I. Jordan ref: http://spectrum.ieee.org/robotics/artificial-intelligence/ma...
Re: An Introduction to Probabilistic Graphical Models (2003) [pdf]
#17Earlier quoted context omitted.
A heuristic guide...If you have more data than knowledge about the domain, and you want to do classification/prediction, then NNs are a good. PGMs are good if you have knowledge that is important to encode; you need modularity i.e. you don't want to embed priors in the model; you have strong causal relationships e.g. diagnostic models with 'explaining away'; you want to integrate value of information e.g. what test s…
by complexity of inference do you mean the complexity of learning the structure of a model ? because inference on an existing PGM is linear in the number of edges with belief-propagation isn't it ?
It's linear in singly connected networks, not for multiply connected graphs. However NP-hard is the worst case as I mentioned (p288 Koller & Friedman)
Re: An Introduction to Probabilistic Graphical Models (2003) [pdf]
#18PGM's are great, but my experience from Koller's course is that it is very hard to identify cases where they can be used. Part of the reason is that you need a-priori knowledge of the causal relationships (coarse grained I.e direction) between your variables. Presumably if you're doing ML you don't know those causal relationships to begin with. Particularly good fits are things like physics where laws are known.
> Part of the reason is that you need a-priori knowledge of the causal relationships (coarse grained I.e direction) between your variables.
Isn't there a whole book, titled Causality by Dr. Pearl, on teasing causal relationships out of data, explicitly for this purpose?
Re: An Introduction to Probabilistic Graphical Models (2003) [pdf]
#19From: http://spectrum.ieee.org/robotics/artificial-intelligence/ma...
Jordan: Well, humans are able to deal with cluttered scenes. They are able to deal with huge numbers of categories. They can deal with inferences about the scene: “What if I sit down on that?” “What if I put something on top of something?” These are far beyond the capability of today’s machines. Deep learning is good at certain kinds of image classification. “What object is in this scene?”
I think Jordan refers here to Bayesian models that incorporate gravity, occlusion, and other such concepts.
http://www.cv-foundation.org/openaccess/content_cvpr_2013/ht... e.g. postulates entire humans to improve scene understanding.
What I get out of this: Deep learning has to be enriched with progress from other machine learning fields
Re: An Introduction to Probabilistic Graphical Models (2003) [pdf]
#20Earlier quoted context omitted.
by complexity of inference do you mean the complexity of learning the structure of a model ? because inference on an existing PGM is linear in the number of edges with belief-propagation isn't it ?
It's linear in singly connected networks, not for multiply connected graphs. However NP-hard is the worst case as I mentioned (p288 Koller & Friedman)
by multiply connected graphs do you mean graphs with cycles ?