Live data from Hacker News

Foundations of probability theory

terrytao.wordpress.com

21–30 of 54 posts

Re: Foundations of probability theory

#21
post #18
post #13

Earlier quoted context omitted.

No, he's just being sloppy. That can be a real pain for students trying to learn. But, be warned: A lot of people work with probability, but only a tiny fraction ever had a course in graduate probability . So, eventually have to learn to put up with, and sometimes rewrite, some of what is written that is not very precise. Here are the accepted definitions: Take a non-empty set, usually denoted by Omega, and call it t…

We want all that for probability theory. We want countably infinite so that we can discuss, say, the event that a coin never or always comes up heads. We don't want uncountably infinite because it would create a big mess in the theory. Who should I read for an introduction to that "big mess"? So many of the cases I'm actually interested in require real valued (continuous) inputs and outcomes. While one can quantize t…

All of this is fine for real-valued inputs and outcomes. It's the number of events (coin flips, measurements, etc) that we're restricting to be countable.

Re: Foundations of probability theory

#22
post #15
post #9

Earlier quoted context omitted.

> In the standard foundations of probability theory, as laid out by Kolmogorov, we can then model these events and random variables by introducing a sample space (which will be a probability space) to capture all the ambient sources of randomness; events are then modeled as measurable subsets of this sample space, and random variables are modeled as measurable functions on this sample space. This matches the definiti…

" ... a sample space (which will be a probability space)" is, in a word, wrong. The sample space will be part of a probability space but will not "be a probability space".

Nonsense. Tao is using perfectly idiomatic language here - "The sample space will be a probability space (once we have endowed it with some additional structure)".

Re: Foundations of probability theory

#24

I don't know what mathematicians think of it, but I enjoyed "Probability Theory: The Logic of Science". https://www.google.com/#q=probability+the+logic+of+science

Its take on probability as a way of modelling our brain is refreshing, for instance, the beginning of chapter 1:

Suppose some dark night a policeman walks down a street, apparently deserted; but suddenly he hears a burglar alarm, looks across the street, and sees a jewelry store with a broken window. Then a gentleman wearing a mask comes crawling out through the broken window, carrying a bag which turns out to be full of expensive jewelry. The policeman doesn't hesitate at all in deciding that this gentleman is dishonest. But by what reasoning process does he arrive at this conclusion? Let us first take a leisurely look at the general nature of such problems.

A moment's thought makes it clear that our policeman's conclusion was not a logical deduction from the evidence; for there may have been a perfectly innocent explanation for everything. It might be, for example, that this gentleman was the owner of the jewelry store and he was coming home from a masquerade party, and didn't have the key with him. But just as he walked by his store a passing truck threw a stone through the window; and he was only protecting his own property. Now while the policeman's reasoning process was not logical deduction, we will grant that it had a certain degree of validity. The evidence did not make the gentleman's dishonesty certain, but it did make it extremely plausible. This is an example of a kind of reasoning in which we haveall become more or less proficient, necessarily, long before studying mathematical theories. We are hardly able to get through one waking hour without facing some situation (e.g. will it rain or won't it?) where we do not have enough information to permit deductive reasoning; but still we must decide immediately what to do.

Re: Foundations of probability theory

#25
post #18
post #13

Earlier quoted context omitted.

No, he's just being sloppy. That can be a real pain for students trying to learn. But, be warned: A lot of people work with probability, but only a tiny fraction ever had a course in graduate probability . So, eventually have to learn to put up with, and sometimes rewrite, some of what is written that is not very precise. Here are the accepted definitions: Take a non-empty set, usually denoted by Omega, and call it t…

We want all that for probability theory. We want countably infinite so that we can discuss, say, the event that a coin never or always comes up heads. We don't want uncountably infinite because it would create a big mess in the theory. Who should I read for an introduction to that "big mess"? So many of the cases I'm actually interested in require real valued (continuous) inputs and outcomes. While one can quantize t…

No problem. For the foundations I outlined, can work just fine with continuous functions, measurable functions, stochastic processes, random variables taking values on the real line, in the complex plane, in finite dimensional real or complex vector spaces with, say, the usual topology, Hilbert and Banach spaces, etc. Can do multi-dimensional Markov processes, and much more.

And you can have each point on the real line an event. Fine. But you just can't take the uncountable union of any set of such events and assume that the result is also an event.

As for the event a random variable takes a value >= 0? Fine.

Or, let the Borel subsets of the real line be the smallest sigma algebra that contains all the open sets, e.g., all the open intervals. Then for Borel set A and real valued random variable X, can ask for the probability X is in A.

I believe you will find that you will have a solid foundation for what you want.

To see all this stuff, need more than just the sparse definitions and, instead, need an actual text and maybe a course. Recently looked at the on-line materials from MIT and didn't see such a course. Graduate probability is not all that popular in the US; stochastic processes in continuous time is still less popular.

To study graduate probability, I'd recommend a good undergraduate major in pure math with good coverage of, say, W. Rudin, Principles of Mathematical Analysis. Then good coverage of linear algebra from more than one of the best known texts. Likely also spend as much time as you can in Halmos, Finite Dimensional Vector Spaces. E.g., at one time, Halmos, Rudin, and Spivak, Calculus on Manifolds were the three main texts for Harvard's famous Math 55.

Get good with proving the theorems.

I also recommend Fleming, Functions of Several Variables.

Then, sure, Royden, Real Analysis. Couldn't be prettier.

If not in a hurry, then the real half of Rudin's Real and Complex Analysis. Especially if you like Fourier theory!

Then of the probability books, I believe that the nicest, first book is L. Breiman, Probability. He wrote that before he went consulting and came back and did CART and random forests.

Next, K. Chung, A Course in Probability Theory. Next, J. Neveu, Mathematical Foundations of the Calculus of Probability. Then, Loeve, Probability Theory.

Loeve is huge -- mostly just use it for reference or browse. E.g., it has sufficient statistics and stationary stochastic processes (the EEs love that) IIRC not in the other books.

IIRC, both Breiman and Neveu were Loeve students at Berkeley.

If do well with Breiman, then for graduate probability, likely can stop there. Else, Chung will then be fast and easy reading and reinforce what you learned in Breiman. Neveu is elegant; my favorite, but deserve extra credit for each workable exercise you can find, not actually work, you understand, just find! Sure, some of the exercises are terrific, half a course in a few lines of an exercise. E.g., he has one of those on statistical decision theory or some such. And see the Tulcea material in the back.

Then there's more that you can do on stochastic processes, potential theory via Brownian motion, e.g., for mathematical finance, stochastic optimal control, and more.

Re: Foundations of probability theory

#26

Earlier quoted context omitted.

Out of curiosity, what's the background that lets you be so dismissive of a Fields Medal winner? At this point I see a blog post written by a well respected mathematician whom I feel comfortable trusting and it's being brushed aside by I don't know who.

Being a great mathematician doesn't necessarily make a great math educator. It is odd Tao chose Durret, but I assume it's due to the book being freely available online.

As the blog comments suggest there's a difference between a good self-study book and a good textbook for a class. The book will be accompanied by (at least) what look to be a good set of course notes.

Some googling around suggests his students are quite pleased with him. He has, according to Wikipedia's intro for him, the undisputed king of math blogs. Both of these point to him being at least a good or above average educator.

Re: Foundations of probability theory

#27
post #18

Earlier quoted context omitted.

We want all that for probability theory. We want countably infinite so that we can discuss, say, the event that a coin never or always comes up heads. We don't want uncountably infinite because it would create a big mess in the theory. Who should I read for an introduction to that "big mess"? So many of the cases I'm actually interested in require real valued (continuous) inputs and outcomes. While one can quantize t…

All of this is fine for real-valued inputs and outcomes. It's the number of events (coin flips, measurements, etc) that we're restricting to be countable.

No, the number of events is necessarily also finite or uncountable. Indeed, it is a nice exercise that there are no countably infinite sigma algebras (extra credit for a solution!).

It's just can't take uncountably many events, take their union, and assume that the result is also an event.

Re: Foundations of probability theory

#28
The fact that Shannon entropy is still relevant in numerous modern mathematics research is amazing.

Apart from minor typos that make him a "sloppy mathematician", he is a good educator based on my personal experience taking grad. courses from him. He's not the passionate high school STEM teacher type, but he offers great insights. I think which textbook he used is somewhat secondary. For courses he had taught before, he usually pick a standard text and teach based on the material he wrote in his blog post, including exercises.

Re: Foundations of probability theory

#29
post #8
post #2

I have a question that might be related to the topic. I have found that often, when solving problems related to probability theory, it is more convenient to think in terms of combinations than to think in terms of probabilities (perhaps because I'm a programmer). Would it be possible to devise a programming language that allows me to program a filter that selects the desired outcomes out of the complete set of combin…

This is possible, and in fact probably implemented in some probabilistic programming languages, but I think you are looking at the wrong direction. The point is that even for fairly simple real use cases, the computation complexity is so huge, that all computers in the world couldn't compute it in your lifetime if you don't employ some approximation or optimization and stick to naive algorithms. So, that is what the…

All those probabilistic programming languages will become exponentially faster once we have feasible quantum computers, since BPP \in BQP. We currently use a weaker inclusion, BPP \in PSPACE, as the core execution model.

Re: Foundations of probability theory

#30
post #16

Earlier quoted context omitted.

Out of curiosity, what's the background that lets you be so dismissive of a Fields Medal winner? At this point I see a blog post written by a well respected mathematician whom I feel comfortable trusting and it's being brushed aside by I don't know who.

Are you advocating "proof by authority "?

Of course. Appeal to authority was never a fallacy; the fallacy is "appeal to false authority". "Terry Tao knows much more about math in general and this in particular than I do, so I'll trust what he says here" is completely valid.
Post reply on HN