Bayesian statistics for confused data scientists • Hacker News Board

Re: Bayesian statistics for confused data scientists

#41

Post by _alternator_ » Sun, Mar 22, 2026, 1:19 PM UTC

zozbot234 wrote: post #32

kgwgk wrote: post #29

Earlier quoted context omitted.

Not true. In frequentist statistics, from the perspective of Bayesians and non-Bayesians alike, there are no priors. —- Dear ChatGPT, are there priors in frequentist statistics? (Please answer with a single sentence.) No — unlike Bayesian statistics, frequentist statistics do not use priors, as they treat parameters as fixed and rely solely on the likelihood derived from the observed data.

There's always priors, they're just "flat", uniform priors (for maximum likelihood methods). But what "flat" means is determined by the parameterization you pick for your model. which is more or less arbitrary. Bayesians would call this an uninformative prior. And you can most likely account for stronger, more informative priors within frequentist statistics by resorting to so-called "robust" methods.

First, there is not such thing as a ‘uninformative’ prior; it’s a misnomer. They can change drastically based on your paramerization (cf change of variables in integration).

Second, I think the nod to robust methods is what’s often called regularization in frequentist statistics. There are cases where regularization and priors lead to the same methodology (cf L1 regularized fits and exponential priors) but the interpretation of the results is different. Bayesian claim they get stronger results but that’s because they make what are ultimately unjustified assumptions. My point is that if they were fully justified, they have to use frequentist methods.

Re: Bayesian statistics for confused data scientists

#42

Post by 7777777phil » Sun, Mar 22, 2026, 1:25 PM UTC

Most ML practitioners use L1/L2 daily without realizing they're making Bayesian prior assumptions. Gaussian prior = Ridge, Laplace prior = Lasso. Once you see it that way, "choosing a regularization strength" is really "choosing how informative your prior is."

Re: Bayesian statistics for confused data scientists

#43

Post by JHonaker » Sun, Mar 22, 2026, 1:29 PM UTC

statskier wrote: post #3

I went through grad school in a very frequentist environment. We “learned” Bayesian methods but we never used them much. In my professional life I’ve never personally worked on a problem that I felt wasn’t adequately approached with frequentist methods. I’m sure other people’s experiences are different depending on the problems you gravitate towards. In fact, I tend to get pretty frustrated with Bayesian approaches b…

I’m not sure what your professional experience is in, but as a counterpoint, I’ve never been in a situation where I hadn’t wished for a system I’m working with to already be in a Bayesian framework. Having said that, I only occasionally am building things from scratch instead of modifying existing systems, so I’m not always lucky enough to be able to work with them.

The pain points around getting a sampler/model pairing working in a reasonable timeframe is definitely a valid complaint. In my experience, inference methods in Bayesian stats are much less forgiving of poorly specified models (or said another way, don’t let you get away with ignoring important structural components of the phenomena of interest). A poorly performing model (in terms of sampler speed/mixing) is often a sign of a problem with the geometry of the parameter space. Frustratingly this can sometimes be a result of conceptually equivalent, but computationally different parameterizations (e.g. centered vs non-centered multi level effects).

The struggles are worth it IMO because it is helpful feedback that helps guide design, and the ease with which I can compute meaningful uncertainty bounds on pretty much any quantity of interest is invaluable.

Re: Bayesian statistics for confused data scientists

#44

Post by kgwgk » Sun, Mar 22, 2026, 1:59 PM UTC

kgwgk wrote: post #29

uoaei wrote: post #26

Earlier quoted context omitted.

Not true. In frequentist statistics, from the perspective of Bayesians, your prior is a point distribution derived empirically. It doesn't have the same confidence / uncertainty intervals but it does have an unnecessarily overconfident assumption of the nature of the data generating process.

Not true. In frequentist statistics, from the perspective of Bayesians and non-Bayesians alike, there are no priors. —- Dear ChatGPT, are there priors in frequentist statistics? (Please answer with a single sentence.) No — unlike Bayesian statistics, frequentist statistics do not use priors, as they treat parameters as fixed and rely solely on the likelihood derived from the observed data.

If you want to say that when you do a frequentist analysis which doesn’t include any concept of prior you get a result that has a similar form to the result of a completely different conceptually Bayesian analysis which uses a flat prior (definitely not “a point distribution derived empirically”) that may be correct. It remains true that there is no prior in the frequentist analysis because they are not part of frequentist inference at all.

Re: Bayesian statistics for confused data scientists

#45

Post by kgwgk » Sun, Mar 22, 2026, 2:07 PM UTC

zozbot234 wrote: post #32

kgwgk wrote: post #29

Earlier quoted context omitted.

Not true. In frequentist statistics, from the perspective of Bayesians and non-Bayesians alike, there are no priors. —- Dear ChatGPT, are there priors in frequentist statistics? (Please answer with a single sentence.) No — unlike Bayesian statistics, frequentist statistics do not use priors, as they treat parameters as fixed and rely solely on the likelihood derived from the observed data.

There's always priors, they're just "flat", uniform priors (for maximum likelihood methods). But what "flat" means is determined by the parameterization you pick for your model. which is more or less arbitrary. Bayesians would call this an uninformative prior. And you can most likely account for stronger, more informative priors within frequentist statistics by resorting to so-called "robust" methods.

It’s not true that “there are always priors”. There are no priors when you calculate the area of a triangle, because priors are not a thing in geometry. Priors are not a thing in frequentist inference either.

You may do a Bayesian calculation that looks similar to a frequentist calculation but it will be conceptually different. The result is not really comparable: a frequentist confidence interval and a Bayesian credible interval are completely different things even if the numerical values of the limits coincide.

Re: Bayesian statistics for confused data scientists

#46

Post by kgwgk » Sun, Mar 22, 2026, 2:12 PM UTC

_alternator_ wrote: post #41

zozbot234 wrote: post #32

Earlier quoted context omitted.

There's always priors, they're just "flat", uniform priors (for maximum likelihood methods). But what "flat" means is determined by the parameterization you pick for your model. which is more or less arbitrary. Bayesians would call this an uninformative prior. And you can most likely account for stronger, more informative priors within frequentist statistics by resorting to so-called "robust" methods.

First, there is not such thing as a ‘uninformative’ prior; it’s a misnomer. They can change drastically based on your paramerization (cf change of variables in integration). Second, I think the nod to robust methods is what’s often called regularization in frequentist statistics. There are cases where regularization and priors lead to the same methodology (cf L1 regularized fits and exponential priors) but the interp…

One standard way to get uninformative priors is to make them invariant under the transformation groups which are relevant given the symmetries in the problem.

Re: Bayesian statistics for confused data scientists

#47

Post by zozbot234 » Sun, Mar 22, 2026, 2:27 PM UTC

kgwgk wrote: post #45

zozbot234 wrote: post #32

Earlier quoted context omitted.

There's always priors, they're just "flat", uniform priors (for maximum likelihood methods). But what "flat" means is determined by the parameterization you pick for your model. which is more or less arbitrary. Bayesians would call this an uninformative prior. And you can most likely account for stronger, more informative priors within frequentist statistics by resorting to so-called "robust" methods.

It’s not true that “there are always priors”. There are no priors when you calculate the area of a triangle, because priors are not a thing in geometry. Priors are not a thing in frequentist inference either. You may do a Bayesian calculation that looks similar to a frequentist calculation but it will be conceptually different. The result is not really comparable: a frequentist confidence interval and a Bayesian cred…

Frequentist confidence intervals as generally interpreted are not even compatible with the likelihood principle. There's really not much of a proper foundation for that interpretation of the "numerical values".

Re: Bayesian statistics for confused data scientists

#48

Post by kgwgk » Sun, Mar 22, 2026, 3:23 PM UTC

zozbot234 wrote: post #47

kgwgk wrote: post #45

Earlier quoted context omitted.

It’s not true that “there are always priors”. There are no priors when you calculate the area of a triangle, because priors are not a thing in geometry. Priors are not a thing in frequentist inference either. You may do a Bayesian calculation that looks similar to a frequentist calculation but it will be conceptually different. The result is not really comparable: a frequentist confidence interval and a Bayesian cred…

Frequentist confidence intervals as generally interpreted are not even compatible with the likelihood principle. There's really not much of a proper foundation for that interpretation of the "numerical values".

What does “as generally interpreted” mean? There is one valid way to interpret confidence intervals. The point is that it’s not based on a posterior probability and there is no prior probability there either.

Re: Bayesian statistics for confused data scientists

#49

Post by NeutralCrane » Sun, Mar 22, 2026, 3:48 PM UTC

algolint wrote: post #31

[flagged]

The exact opposite is true. Virtually everyone’s intuition is aligned with the Bayesian model. That intuition has to be hammered out of people in their stats classes because for decades frequentist approaches were computationally more feasible, even if they don’t align with how most humans interpret probability.

Re: Bayesian statistics for confused data scientists

#50

Post by therobots927 » Sun, Mar 22, 2026, 4:08 PM UTC

jmalicki wrote: post #13

therobots927 wrote: post #8

Earlier quoted context omitted.

That’s Bayesian propaganda

Huh? Are there really any pure frequentists post Stein's paradox? At least ones that are aware of it and maintain objections to fusing the fields?

Downvote me all you want. Bayesianism is misapplied much more frequently than frequentism. It just makes it way too easy to fudge p values. Sorry not sorry.