Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Bayes’ Rule Emerged Triumphant from Two Centuries of Controversy

mcgrayne.com

Post reply on HN
Print view View original

41–50 of 83 posts

Re: How Bayes’ Rule Emerged Triumphant from Two Centuries of Controversy

#41

Post by hacker42 »

twanvl wrote: post #26
hacker42 wrote: post #19

I could never quite understand the divide between Bayesian statistics and frequentist statistics. Both seem to be ultimately about counting the frequency by which something occurs and normalizing this frequency with respect to the number of all possible outcomes. Bayesian statistics essentially is concerned with the application of the Bayesian updating technique by which one can iteratively improve a distribution ove…

Bayesian and frequentist approaches ultimately have a different notion of probability. In the frequentist approach, a probability of 10% means that if you repeat an experiment many times, roughly 1 out of 10 times you will observe an event. In Baysian statistics, a probability of 10% means that you are that certain about the event happening. So you would be willing to bet at 10 to 1 odds on the event happening. There…

But it seems to me one can still define this in terms of frequencies using a more general definition of what is meant by repeating an experiment. For example, when you use a probability P as a degree of belief or certainty about whether a patient X with symptoms W has a particular disease Y, one would define the universe as containing all possible realities in which X has the same symptoms W but with different underlying causal factors that lead to the same symptoms. An observation is a uniform sample from this universe and the belief is that a fraction of P of these realities has the cause Y.

Re: How Bayes’ Rule Emerged Triumphant from Two Centuries of Controversy

#42

Post by [deleted] »

kareemsabri wrote: post #39
danbruc wrote: post #38

Earlier quoted context omitted.

That means it is totally impossible to get H H H H... or H T T H T T... ad infinitum? I really have no well-founded opinion on that, it just seems very counterintuitive that the coin is not allowed to yield any such sequence, they same as good as any other sequence to me.

Not at all! It yields that sequence all the time, in small numbers. But, it is impossible to get only heads, if you could truly flip a coin infinitely many times, which of course you can't. The larger you make the sequence, the smaller you make the probability of all heads. In your example: P(H,H,H,H) = 0.5 ^ 4 = 0.0625 P(H,H,H,H,H,H,H,H) = 0.5 ^ 8 = 0.00390625 As you can see it's getting pretty small already. Howeve…

[deleted]

Re: How Bayes’ Rule Emerged Triumphant from Two Centuries of Controversy

#43

Post by kareemsabri »

danbruc wrote: post #40
kareemsabri wrote: post #39

Earlier quoted context omitted.

Not at all! It yields that sequence all the time, in small numbers. But, it is impossible to get only heads, if you could truly flip a coin infinitely many times, which of course you can't. The larger you make the sequence, the smaller you make the probability of all heads. In your example: P(H,H,H,H) = 0.5 ^ 4 = 0.0625 P(H,H,H,H,H,H,H,H) = 0.5 ^ 8 = 0.00390625 As you can see it's getting pretty small already. Howeve…

But now you are contradicting yourself, aren't you? We agree that for any finite sequence of tosses it is unlikely but possible to get all heads and therefore it is possible to not converge to 0.5. The question is what difference it makes to go from a large but finite to an infinite number of tosses. Either it is impossible to get only heads an infinite number of times, then I have a problem understanding why that is…

> The question is what difference it makes going from a large but finite to an infinite number of tosses.

Convergence is only guaranteed as N -> ∞. The difference between large but finite and infinite is.. well, infinite :) So that's a pretty significant difference.

> Either is impossible to get only heads an infinite number of times, then I have a problem understanding why that is, or all heads is still a possible outcome even in the infinite case, then the process does not necessarily yield 0.5 even in the limit of an infinite number of tosses.

It's the former. It is impossible to get only heads an infinite number of times. It is possible, but increasingly unlikely, to get only heads a REALLY LARGE, but finite, number of times.

Re: How Bayes’ Rule Emerged Triumphant from Two Centuries of Controversy

#44

Post by danbruc »

jordigh wrote: post #32
danbruc wrote: post #18

Is there any (uncontroversial) theory that rigorously defines what a 50 % probability for heads and tails means? It certainly doesn't mean that in the long run you will obtain the same number of heads and tails because there is a (vanishing) chance that you will always get heads even though the coin is actually fair. And just saying that you will obtain the same or at least similar number of heads and tails with high…

> Is there any (uncontroversial) theory that rigorously defines what a 50 % probability for heads and tails means? Yeah, Kolmogorov's axioms: https://en.wikipedia.org/wiki/Probability_axioms To interpret these axioms for 50% probability means that the measure underneath the density function corresponding to the event "heads" is one-half. But "rigourous" doesn't have anything to do with the natural world. You can't ma…

I know the Kolmogorov's axioms but I am really more interested in that part they avoid - what is the meaning of a probability of 0.5? It is surly nice that we can operate with probabilities in a (hopefully) self-consistent way, but it bugs me quite a bit that I don't really precisely unterstand what the result of a calculation implies for the real world.

Re: How Bayes’ Rule Emerged Triumphant from Two Centuries of Controversy

#45

Post by mikekchar »

hacker42 wrote: post #23
mikekchar wrote: post #11

Earlier quoted context omitted.

The one place I've used Bayes (hopefully properly!) is in a spaced repetition flash card program. Usually spaced repetition algorithms wait a certain amount of time based on how many times you have seen and remembered a card. The more times you have remembered it, the longer you wait. It then creates a schedule for each day. You review the cards that have "expired" their wait time. I wanted to turn this upside down.…

That basically seems to ignore the exponential Ebbinghausian forgetting curve. I doubt it results in better scheduling of the cards.

It's not exponential. It has a gamma distribution. The nice thing about the gamma distribution is that within a short part of the tail it is nearly linear. So while it's hard to explain the math in an HN posting, below a certain probability your odds of getting a 90-95% confidence of a 90% recall rate (10% forgetting) rate is very, very low. So you will have to review all of those cards anyway. Once you get into the range where false positives are more likely (say above 80%) the curve starts to get more linear. This is especially true for cards with a very shallow curve (those that you have seen many times). So even when you get it wrong, the ones that are likely to be problematic are those that you will review again quickly anyway.

As for whether it results in a better scheduling of cards, it's hard to say. It has the advantage of being self adjusting and containing far fewer magic numbers than something like SM(insert-any-version-here).

Re: How Bayes’ Rule Emerged Triumphant from Two Centuries of Controversy

#46

Post by jordigh »

danbruc wrote: post #44
jordigh wrote: post #32

Earlier quoted context omitted.

> Is there any (uncontroversial) theory that rigorously defines what a 50 % probability for heads and tails means? Yeah, Kolmogorov's axioms: https://en.wikipedia.org/wiki/Probability_axioms To interpret these axioms for 50% probability means that the measure underneath the density function corresponding to the event "heads" is one-half. But "rigourous" doesn't have anything to do with the natural world. You can't ma…

I know the Kolmogorov's axioms but I am really more interested in that part they avoid - what is the meaning of a probability of 0.5? It is surly nice that we can operate with probabilities in a (hopefully) self-consistent way, but it bugs me quite a bit that I don't really precisely unterstand what the result of a calculation implies for the real world.

The modern approach to mathematics is that there is no "meaning", just like "2" or "derivative" has no meaning. We just say how it behaves, or define it in terms of other things, which eventually bottoms out with undefined terms, such as sets and set membership. This is formalism.

How you apply mathematics to the world is not the business of formal mathematics. Whatever you want to do with it is "mere" philosophy. ;-)

Re: How Bayes’ Rule Emerged Triumphant from Two Centuries of Controversy

#47

Post by danbruc »

kareemsabri wrote: post #43
danbruc wrote: post #40

Earlier quoted context omitted.

But now you are contradicting yourself, aren't you? We agree that for any finite sequence of tosses it is unlikely but possible to get all heads and therefore it is possible to not converge to 0.5. The question is what difference it makes to go from a large but finite to an infinite number of tosses. Either it is impossible to get only heads an infinite number of times, then I have a problem understanding why that is…

> The question is what difference it makes going from a large but finite to an infinite number of tosses. Convergence is only guaranteed as N -> ∞. The difference between large but finite and infinite is.. well, infinite :) So that's a pretty significant difference. > Either is impossible to get only heads an infinite number of times, then I have a problem understanding why that is, or all heads is still a possible o…

Okay, assuming that is true, is there an intuitive way to understand that? And the fact that lim[n-> ∞] 0.5^n = 0 unfortunately won't do for me, that is true for any specific infinite sequence, even those containing 50/50 heads and tails. I think most of the sequences - hand-waving, most of an infinite set - are 50/50 heads and tails just because there are more possibilities - hand-waving again - to arrange 50/50 heads and tails versus 100/0 or 40/60 heads and tails. So can I sum over all the 50/50 sequences and get 1 and sum over the rest and get 0? I still would not really understand what forces my coin to show tails eventually, but if there were a measure showing that those two sets have measure 1 respectively 0 it would already be easier to swallow.

Re: How Bayes’ Rule Emerged Triumphant from Two Centuries of Controversy

#48

Post by danbruc »

jordigh wrote: post #46
danbruc wrote: post #44

Earlier quoted context omitted.

I know the Kolmogorov's axioms but I am really more interested in that part they avoid - what is the meaning of a probability of 0.5? It is surly nice that we can operate with probabilities in a (hopefully) self-consistent way, but it bugs me quite a bit that I don't really precisely unterstand what the result of a calculation implies for the real world.

The modern approach to mathematics is that there is no "meaning", just like "2" or "derivative" has no meaning. We just say how it behaves, or define it in terms of other things, which eventually bottoms out with undefined terms, such as sets and set membership. This is formalism. How you apply mathematics to the world is not the business of formal mathematics. Whatever you want to do with it is "mere" philosophy. ;-…

That is what I meant - I am more asking for a solid philosophical interpretation than a mathematical theory, after all frequentism, Bayesianism and all the other interpretations seem to carry quite a bit of philosophy.

On the other hand in another comment it just boiled down to the question whether there is a measure that gives 1 for the set of all infinite binary sequences with 50/50 zeros and ones and 0 for the set of all the other sequences. So it is not pure philosophy what I am interested in.

Re: How Bayes’ Rule Emerged Triumphant from Two Centuries of Controversy

#49

Post by mikekchar »

mtrimpe wrote: post #14
mikekchar wrote: post #11

Earlier quoted context omitted.

The one place I've used Bayes (hopefully properly!) is in a spaced repetition flash card program. Usually spaced repetition algorithms wait a certain amount of time based on how many times you have seen and remembered a card. The more times you have remembered it, the longer you wait. It then creates a schedule for each day. You review the cards that have "expired" their wait time. I wanted to turn this upside down.…

That sounds fascinating. Is there anywhere you could look at the result (or code ;) of this work?

Umm... The code is horrific as I was experimenting with a few different ideas. It is written in Ruby for GTK+ and is not at all idiomatic Ruby. It's also somewhere between slightly and completely broken at the moment... But with all that in mind: https://github.com/mikekchar/JLDrill

Probably more interesting is simply my description of the scheduling algorithm: https://github.com/mikekchar/JLDrill/blob/master/web/src/Str...

There is also one detail missing which is forgetting. Because the items are sorted by the ratio of time waited to "ideal schedule", we can easily stick anything over a certain amount into a separate set (called the forgotten set). That way if you don't study for a long time you can "forget" those items and they are treated like a high priority "new" set until the set is empty. Probably that makes no sense, but if you read my strategy document, you will hopefully be able to understand.

Re: How Bayes’ Rule Emerged Triumphant from Two Centuries of Controversy

#50

Post by mikekchar »

kybernetikos wrote: post #13
mikekchar wrote: post #11

Earlier quoted context omitted.

The one place I've used Bayes (hopefully properly!) is in a spaced repetition flash card program. Usually spaced repetition algorithms wait a certain amount of time based on how many times you have seen and remembered a card. The more times you have remembered it, the longer you wait. It then creates a schedule for each day. You review the cards that have "expired" their wait time. I wanted to turn this upside down.…

That sounds very interesting. Is your program available anywhere?

See my reply to mtrimpe below. I maintained this program for quite a long time, but realistically my choice of development platform was a poor one ;-) Also my code was pretty awful as I was experimenting with several strange ideas and also writing Ruby code is if I had spent the last 20 years writing C++ code (which... um... might have been true...)

You can likely get it to work for some definitions of "work" on a Linux box, but anything else would require serious effort ;-)

Link in case you don't see the other message: https://github.com/mikekchar/JLDrill

Post reply on HN