Foundations of probability theory • Hacker News Board

Re: Foundations of probability theory

#41

Post by ahelwer » Sat, Oct 03, 2015, 1:28 AM UTC

joseraul wrote: post #24

GregBuchholz wrote: post #19

I don't know what mathematicians think of it, but I enjoyed "Probability Theory: The Logic of Science". https://www.google.com/#q=probability+the+logic+of+science

Its take on probability as a way of modelling our brain is refreshing, for instance, the beginning of chapter 1: Suppose some dark night a policeman walks down a street, apparently deserted; but suddenly he hears a burglar alarm, looks across the street, and sees a jewelry store with a broken window. Then a gentleman wearing a mask comes crawling out through the broken window, carrying a bag which turns out to be ful…

This is called abductive reasoning, FYI.

I sometimes joke Sherlock Holmes is really all about "The Science of Abduction" not deduction; a rather more threatening tagline.

Re: Foundations of probability theory

#42

Post by Retra » Sat, Oct 03, 2015, 1:44 AM UTC

thaumasiotes wrote: post #30

copperx wrote: post #16

Earlier quoted context omitted.

Are you advocating "proof by authority "?

Of course. Appeal to authority was never a fallacy; the fallacy is "appeal to false authority". "Terry Tao knows much more about math in general and this in particular than I do, so I'll trust what he says here" is completely valid.

A fields medal doesn't indicate general authority in mathematics, only authority with regards to what was necessary for getting the medal.

Re: Foundations of probability theory

#43

Post by grayclhn » Sat, Oct 03, 2015, 1:49 AM UTC

nkurz wrote: post #18

graycat wrote: post #13

Earlier quoted context omitted.

No, he's just being sloppy. That can be a real pain for students trying to learn. But, be warned: A lot of people work with probability, but only a tiny fraction ever had a course in graduate probability . So, eventually have to learn to put up with, and sometimes rewrite, some of what is written that is not very precise. Here are the accepted definitions: Take a non-empty set, usually denoted by Omega, and call it t…

We want all that for probability theory. We want countably infinite so that we can discuss, say, the event that a coin never or always comes up heads. We don't want uncountably infinite because it would create a big mess in the theory. Who should I read for an introduction to that "big mess"? So many of the cases I'm actually interested in require real valued (continuous) inputs and outcomes. While one can quantize t…

I think you're misunderstanding. Standard basic (measure-theoretic) probability theory is designed to handle common non-discrete real-world cases: continuous random variables like height, temperature, etc. They're not approximated by something countable; instead theorems proving that they have the sort of behavior you'd want are established by proving them for a countable approximation, then taking limits. It is exactly like integration: prove things for step functions, then make the steps infinitely thin. The theory is clean and straightforward.

Here's where it gets less basic: say you want to look at temperature over time, but you don't want to model temperature as a variable that's measured daily, or hourly, or even every second (secondly?), but you want to model it as a process that evolves in continuous time. That's where the theory gets messy. Not necessarily at the level of a user of this theory, but definitely at the level of proving that the math you want to use is allowed.

If you actually need an introduction to that sort of probability, Lawrence Evans (Berkeley) has some old lecture notes aimed at undergrads [1] that he turned into a book [2]. If (more likely) you want standard measure-theoretic probability theory (as opposed to what's taught to undergraduates), David Pollard's book is pretty good [3].

I'm sure that @graycat will scoff at those recommendations, but his reading list would be considered excessively hardcore and time consuming even for a graduate student in math, which I'm assuming you're not.

[1]: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.416...

[2]: http://www.amazon.com/An-Introduction-Stochastic-Differentia...

[3]: http://www.amazon.com/Theoretic-Probability-Statistical-Prob...

ps: after looking at it again, the intro in Evans's notes is as gentle as it's going to get, so start there. And (as you'll find out, unless you're some sort of savant) this shit's hard. If you actually want to understand this stuff, graduate coursework is probably the only practical way to do it.

Re: Foundations of probability theory

#44

Post by raincom » Sat, Oct 03, 2015, 1:57 AM UTC

grayclhn wrote: post #43

nkurz wrote: post #18

Earlier quoted context omitted.

We want all that for probability theory. We want countably infinite so that we can discuss, say, the event that a coin never or always comes up heads. We don't want uncountably infinite because it would create a big mess in the theory. Who should I read for an introduction to that "big mess"? So many of the cases I'm actually interested in require real valued (continuous) inputs and outcomes. While one can quantize t…

I think you're misunderstanding. Standard basic (measure-theoretic) probability theory is designed to handle common non-discrete real-world cases: continuous random variables like height, temperature, etc. They're not approximated by something countable; instead theorems proving that they have the sort of behavior you'd want are established by proving them for a countable approximation, then taking limits. It is exac…

Thanks for your references. I have always thought that many statistics/probability based explanations are adhoc. They are adhoc because they explain pre-selected facts; and their predictions are just a confirming instances (cf. positive vs confirming instance from Larry Laudan, a philosopher of science). Your point "definitely at the level of proving that the math you want to use is allowed" hints in that direction.

Re: Foundations of probability theory

#45

Post by grayclhn » Sat, Oct 03, 2015, 3:02 AM UTC

raincom wrote: post #44

grayclhn wrote: post #43

Earlier quoted context omitted.

I think you're misunderstanding. Standard basic (measure-theoretic) probability theory is designed to handle common non-discrete real-world cases: continuous random variables like height, temperature, etc. They're not approximated by something countable; instead theorems proving that they have the sort of behavior you'd want are established by proving them for a countable approximation, then taking limits. It is exac…

Thanks for your references. I have always thought that many statistics/probability based explanations are adhoc. They are adhoc because they explain pre-selected facts; and their predictions are just a confirming instances (cf. positive vs confirming instance from Larry Laudan, a philosopher of science). Your point "definitely at the level of proving that the math you want to use is allowed" hints in that direction.

Probability's hard to teach. You can give informal statements and kind of wave your hands at the underlying theory, or you can give a rigorous well-founded treatment that's intellectually satisfying. But the rigorous foundation uses math that's a step or two beyond what undergraduate math majors learn. It's not necessarily harder than what math majors see, but it's a ton of extra material to teach, when the payoff is that you can now (after half a year) prove that the conditional probability is well-defined as

Pr(A | B) = Pr(A and B) / Pr(B)

instead of just telling it to students and drawing a few diagrams that drive the point home.

But I think it's more pragmatic than ad hoc. Any deep theory of probability that doesn't deliver

Pr(A | B) = Pr(A and B) / Pr(B)

is basically useless since that's how random phenomena seem to behave in real life. Having a deeper theory is useful because it allows you to derive other implications of that theory and makes certain calculations much easier. But if the theory disagrees with phenomena that we want to model, that can be a problem.

Re: Foundations of probability theory

#46

Post by Double_Cast » Sat, Oct 03, 2015, 4:08 AM UTC

amelius wrote: post #2

I have a question that might be related to the topic. I have found that often, when solving problems related to probability theory, it is more convenient to think in terms of combinations than to think in terms of probabilities (perhaps because I'm a programmer). Would it be possible to devise a programming language that allows me to program a filter that selects the desired outcomes out of the complete set of combin…

I think you're looking for Pascal's Triangle.

  let P = probability of 2 heads given 4 coins
  let omega = (x + y)^n     # probability space
  let x = 50%               # P(heads)
  let y = 50%               # P(tails)
  let (n, k) = (4, 2)       # 4 coins, 2 heads

  P = (n, k)(x  )^2(y  )^2
      (4, 2)(50%)^2(50%)^2
      (6   )(25%)  (25%)    # (4, 2) = 6; see link
      (6   )(6.25%)
      (37.5%)

https://en.wikipedia.org/wiki/Pascal%27s_triangle

Re: Foundations of probability theory

#47

Post by yawaramin » Sat, Oct 03, 2015, 6:34 AM UTC

Retra wrote: post #42

thaumasiotes wrote: post #30

Earlier quoted context omitted.

Of course. Appeal to authority was never a fallacy; the fallacy is "appeal to false authority". "Terry Tao knows much more about math in general and this in particular than I do, so I'll trust what he says here" is completely valid.

A fields medal doesn't indicate general authority in mathematics, only authority with regards to what was necessary for getting the medal.

OK, but being a math professor does indicate general authority in mathematics.

Re: Foundations of probability theory

#48

Post by eruditely » Sat, Oct 03, 2015, 8:20 AM UTC

Retra wrote: post #42

thaumasiotes wrote: post #30

Earlier quoted context omitted.

Of course. Appeal to authority was never a fallacy; the fallacy is "appeal to false authority". "Terry Tao knows much more about math in general and this in particular than I do, so I'll trust what he says here" is completely valid.

A fields medal doesn't indicate general authority in mathematics, only authority with regards to what was necessary for getting the medal.

Appeal to authority is a dumb logical fallacy in reality geniuses are better at everything and we should prioritize their opinions by some decent weight over everyone else. Obviously.

Re: Foundations of probability theory

#49

Post by nkurz » Sat, Oct 03, 2015, 9:45 AM UTC

grayclhn wrote: post #43

nkurz wrote: post #18

Earlier quoted context omitted.

We want all that for probability theory. We want countably infinite so that we can discuss, say, the event that a coin never or always comes up heads. We don't want uncountably infinite because it would create a big mess in the theory. Who should I read for an introduction to that "big mess"? So many of the cases I'm actually interested in require real valued (continuous) inputs and outcomes. While one can quantize t…

I think you're misunderstanding. Standard basic (measure-theoretic) probability theory is designed to handle common non-discrete real-world cases: continuous random variables like height, temperature, etc. They're not approximated by something countable; instead theorems proving that they have the sort of behavior you'd want are established by proving them for a countable approximation, then taking limits. It is exac…

You're right, and I misunderstood. I'm a computer programmer trying to rapidly learn enough about probability theory to be able to communicate with some theoretical statisticians regarding causality, confounding, and longitudinal data analysis. I have a decent intuitive grasp of what's happening, but no ability to convey anything with proper terminology. I could certainly use a better grasp of the basics, and I'm trying to figure out where to start. Thanks for the links.

Re: Foundations of probability theory

#50

Post by graycat » Sat, Oct 03, 2015, 2:26 PM UTC

grayclhn wrote: post #43

nkurz wrote: post #18

Earlier quoted context omitted.

We want all that for probability theory. We want countably infinite so that we can discuss, say, the event that a coin never or always comes up heads. We don't want uncountably infinite because it would create a big mess in the theory. Who should I read for an introduction to that "big mess"? So many of the cases I'm actually interested in require real valued (continuous) inputs and outcomes. While one can quantize t…

I think you're misunderstanding. Standard basic (measure-theoretic) probability theory is designed to handle common non-discrete real-world cases: continuous random variables like height, temperature, etc. They're not approximated by something countable; instead theorems proving that they have the sort of behavior you'd want are established by proving them for a countable approximation, then taking limits. It is exac…

> I'm sure that @graycat will scoff at those recommendations, but his reading list would be considered excessively hardcore and time consuming even for a graduate student in math, which I'm assuming you're not.

Probability and stochastic processes based on measure theory are not very popular in the US, even in graduate math departments.

Uh, scoff, scoff. Okay?

The full measure theoretic details of stochastic processes in continuous time can be a bit of a challenge. That topic can be important, e.g., for Brownian motion and stochastic differential equations used in mathematical finance. Of course, there is

Karatzas and Shreve, Brownian Motion and Stochastic Calculus and Chung and Williams, Introduction to Stochastic Integration. And there's much more, especially from Russia and France.

But, otherwise, usually in practice, what people are interested in is either (1) second order stationary stochastic processes, e.g., as in electronic or acoustical signals and noise. There are commonly interested in power spectral estimation, digital filtering, maybe Wiener filtering, the fast Fourier transform, etc. or (2) what is in, say, Cinlar, Introduction to Stochastic Processes.

In Cinlar, for the continuous time case, get a good introduction to the Poisson process (the vanilla arrival process, e.g., like clicks at a Geiger counter, new sessions at a Web site, and much more). Also get what else people are mostly interested in in practice, Markov processes in discrete time with a discrete state space (that is, the values are discrete).

The case of Markov processes in continuous time and discrete state space is not so tough if the jumps are driven by just a Poisson process. But there is still more in Cinlar.

And there are other good texts on stochastic processes.

For (1), look at some of the texts used by EEs. The measure theory approach is in Doob, Stochastic Processes, Loeve, Probability Theory, and several more texts by quite good authors. E.g., without measure theory, can just dive in via Blackman and Tukey, The Measurement of Power Spectra ....

With all these sources, are able to get by without measure theory. Yes, without measure theory, at some places will have to not ask to understand too much and just skip over some details to get back to the applied stuff.

But for measure theory, the Durrett text seems to get a student to that unusually quickly.

For more, at the MIT Web site, there is an on-line course in mathematical finance that avoids measure theory. They want to use the Radon-Nikodym theorem and Ito integration but still avoid measure theory. Uh, the Radon-Nikodym theorem is a generalization of the fundamental theorem of calculus. Once see it, it's dirt simple, but a good proof takes a bit or follow von Neumann's proof that knocks it all off in one stroke (it's in Rudin, Real and Complex Analysis).