I think there is a simple solution to the thought experiment in the beginning, ignoring the paragraphs upon paragraphs of EY liking the sound of his own voice: The information content of each experiment consists of more than just the stated number of patients tested and success rate. In particular, each experiment report I notice is strong evidence that someone actually used humanity's limited resources to perform that experiment, and slightly less strong evidence that they actually followed the stated procedure. Therefore, the completion of the "stop when I have a high enough success rate" experiment should cause me to update in favour of people with the means to actually running such an experiment, and hence make it more likely that at this very moment there are other research groups out there that are like 1000 patients in and have not yet gotten their 60% success rate.
Beautiful Probability
51–60 of 67 posts
Re: Beautiful Probability
#52This essay is so weird to read. The author is extremely passionate, yet also claiming to be simply rational. He’s throwing terminology around (Dutch book, ZF) but seems unaware of the limits of the approach he advocates.
There are so many cracks in the Bayesian edifice promoted in TFA!
These problems are well-known in the Theories of Probability community [1] (which is only a subset of the larger set of theorists recognizing the limits of mechanical Bayesian reasoning in decision problems).
Here are a couple.
(1)
Bayesian approaches force you to assign a sharp probability to every event. How do we map any event to a sharp probability? E.g., I need to give a number for the probability of rain tomorrow, a non-repeating event. How do I map that to a number? Not through relative frequencies- it’s non-repeating. If two people give different numbers, how do we decide who is right?
This problem is what Peter Walley has called the “Bayesian dogma of precision.” [2]
(2)
As noted above in an aside, we have a hard time computing probabilities. This is a practical problem that we all are aware of, but often discount.
In what we could call CMP (Conventional Mathematical Probability - Kolmogorov’s axioms) we typically can’t even correctly enumerate the sample space. We’re always forgetting something, so our models are too confident. (In the “Dutch book” analogy alluded to in TFA, we are following the axioms but are somehow always losing money, in a very real sense.)
Related to this problem of computing probabilities, we don’t have a rigorous way to determine when two real-world events are independent. Yet we constantly invoke independence to construct models. Kolmororov’s 1933 manuscript was clear on this problem. [3]
Not satisfied with this, we go on to hypothesize conditional independence relationships in order to feed our complex “rational” Bayesian machine. It’s thirsty for numbers, and we just make them up!
*
This all sounds somewhat hypothetical. It’s not. In my day job, I compute supposed Bayesian credible intervals for various physical variables.
The people downstream who use those variables to assimilate into physical models typically multiply our credible intervals by 2. My friend across lab has it even worse, they multiply his Bayesian intervals by 3.
This is not a well-functioning machine.
[1] E.g., https://isipta23.sipta.org/, or https://plato.stanford.edu/entries/imprecise-probabilities/#...
[2] https://issuu.com/impreciseprobabilities/docs/imprecise_prob..., first paragraph, although the whole short article is on-point
[3] from memory, the quote is something like, “determining the conditions under which events may be judged independent is one of the major outstanding problems in theory of probability“
Re: Beautiful Probability
#53Bayesian approach sounds like a religion (one true way). There is nothing unusual about different mathematical methods/models producing different results e.g., the number of roots even for the same quadratic equation may depend on "private" thoughts such as whether complex roots are of interest (sometimes they do/sometimes they don't). All models are wrong some are useful.
> the number of roots even for the same quadratic equation may depend on "private" thoughts such as whether complex roots are of interest You are confusing ambiguity in a problem statement due to human language being imprecise with two well-specified identical experimental results having different results due to the intentions of the human carrying them out. Is arithmetic a religion because there's "one true way" of…
It is not about human language being imprecise. I can formulate the questions using the math language precisely with the exact same result (different number of roots are possible for different formulations of the problem for the same "physical" (coefficients of the quadratic equation) setup).
The Map is not the Territory.
Different maps can be useful. No true map.
Re: Beautiful Probability
#54As other commenters have pointed out any given introductory chapter in a book on Bayesian statistics, including Jaynes’, is better exposition than this. I found _Probability Theory: The Logic of Science_ very easy to follow and very well-written. I had a similar experience when I finally found a copy of Barbour’s _The End of Time_ and discovered, much to my chagrin, that it wasn’t nearly as mystical or complicated as…
Jaynes is great, but The Logic of Science is a bit rough around the edges, with lots of errata. Jaynes died when the book was really just a very rough draft plus notes. Bretthorst had to go in and turn it into something publishable, not an enviable task by any means.
Here's a list of errata and commentary, collected by a fan: https://ksvanhorn.com/bayes/jaynes/index.html.
Re: Beautiful Probability
#55>Think laws, not tools
But laws are tools, and the esthetical intellectual elegance is an epiphenomenal bonus or a mean to keep human psychism motivated to keep its focus away from all the other attention sinks that life throw at it.
And that apply for both law in judiciary and sciences parlances.
Re: Beautiful Probability
#56I'm confused in that I don't see how this is troubling. Yes, the two experimenters rolled dice and got the same result, but it's as if one of them was rolling a 6 sided die and the other a 20 sided one. Each experiment is not a result per se but a sample from a distribution. How you infer the shape of that distribution based on the experiment is a function of the distribution of all courses your experiment could have…
I had the same reaction?
We don't actually care at all about what happened in the two experiments per se, we care about the information provided by the experiments about future or other events.
If somehow we learned that both experiments were totally unreplicable and a product purely of that time and location with no implications for anything else ever before or since we wouldn't care about them except maybe as a historical curiosity.
Intentionally is a red herring; what matters is our expectation about what might be observed if we were to repeat the experiments again.
In that sense, there's variability in the second experiment's results due to sample size being random. So we interpret and infer based on that potential experiment we could do, not what happened to be observed at a particular moment.
I'm also confused about what this has to do with Bayesian versus non-Bayesian inference as you could approach either experiment from either paradigm, and there are different forms of Bayesianism, including nonsubjective Bayesianism.
Re: Beautiful Probability
#57Bayesian approach sounds like a religion (one true way). There is nothing unusual about different mathematical methods/models producing different results e.g., the number of roots even for the same quadratic equation may depend on "private" thoughts such as whether complex roots are of interest (sometimes they do/sometimes they don't). All models are wrong some are useful.
> the number of roots even for the same quadratic equation may depend on "private" thoughts such as whether complex roots are of interest No. The number of roots that you care about might depend on your private thoughts; but the number of roots itself does not. It's a mathematical fact. It just might not be a mathematical fact that you actually care about. But what you care about is not part of math.
Why are you ignoring the quaternion roots? 3x3 matrix roots?
Re: Beautiful Probability
#58I use Bayesian methods often, but this a just religious. Bayesian methods are just that, tools, methods for approaching a problem.
There are no laws for applying probability to the real world. To think so puts too much faith in your models. Remember, all models are wrong. Applying probability to the real world requires a host of assumptions, regardless of the methods you use.
Frequentist and Bayesian methods have different goals, both have there place.
For a counterweight to the strong likelihood principle find discussions of Larry Wasserman: https://youtu.be/Z-YvWyM6dRQ?si=qwzRiaPbj9ruiUEv
And for a balanced discussion for why both are great see Michael Jordan: https://youtu.be/HUAE26lNDuE?si=cwg6wpRS1gXL6r1Y
Re: Beautiful Probability
#59Earlier quoted context omitted.
This seems to assume that intentions "don't count" in some way, as if they were nonphysical, whereas unless you presume a supernatural soul, they are as physical as any other part of the experiment.
> This seems to assume that intentions "don't count" in some way, as if they were nonphysical Not nonphysical: just not part of the physical degrees of freedom that can affect things like how die rolls come up or how well a given treatment works on a patient. The experimenter's intentions (not about the stopping criterion, but about other things) can of course be upstream physical causes, so to speak, of things like…
> Not nonphysical: just not part of the physical degrees of freedom that can affect things like how die rolls come up or how well a given treatment works on a patient.
For a dice that is not a concern (unless animism is taken into consideration), but when humans are on both side of the equation, how do you get rid of all the social and psychological effects that imply, including placebo and the desire to see the study bend in some direction, be it at some unconscious level?
Re: Beautiful Probability
#60Earlier quoted context omitted.
P(B|I saw E, P) = P(I saw E|B,P) * P(B|P) / P(I saw E|P) P(B|E was false, I saw E, P) = P(E was false|B,I saw E,P) * P(B|P,I saw E) / P(E was false|P, I saw E) This is a pretty basic application of Bayes' theorem.
Love it: p(I saw E) and p(I didn’t really see E). Just move the argument one level down: “I saw E is false” and it turns out so is “E is false” . So then? Add “E was false was false”? Turtles all the way down. At some point something has to be “true” in order to conditionalise on it.
Yes sure, here are a few truths that never disapointed me:
There is an absolute universal truth.
Absolute universal truth, as a whole, is unreachable even to the most intelligent and resourceful human that will ever exist.