Earlier quoted context omitted.
When I was in academia, I used to fit highly sophisticated models (think many-parameters, multi-level non-linear mixed effect models) who were giving not only point estimate but also confidence and predictive intervals ("please explain to me the difference between the two" is one of my favorite interview questions and I still have not heard a correct answer). When I tried to bring an "uncertainty mindset" over when I…
What do "multi-level" and "mixed effects" mean? There are tons of non-linear models with lots of parameters, but I've never heard these other terms.
Modeling Uncertainty with PyTorch
11–20 of 23 posts
Re: Modeling Uncertainty with PyTorch
#12The field of ML is largely focused on just getting predictions with fancy models. Estimating the uncertainty, unexpectedness and perplexity of specific predictions is highly underappreciated in common practice. Even though it is highly economically valuable to be able to tell to what extent you can trust a prediction, the modelling of uncertainty of ML pipelines remains an academic affair in my experience.
When I was in academia, I used to fit highly sophisticated models (think many-parameters, multi-level non-linear mixed effect models) who were giving not only point estimate but also confidence and predictive intervals ("please explain to me the difference between the two" is one of my favorite interview questions and I still have not heard a correct answer). When I tried to bring an "uncertainty mindset" over when I…
Great answer. It prompts a bunch of followup questions!
most DS/ML scientists use ML models that typically don't provide an easy way to estimate uncertainty intervals
Not an DS/ML scientist but a data engineer. The models I've used have been pretty much "slap it into XGBoost with k-fold CV, call it done" — an easy black box. Is there any model or approach you like to estimate uncertainty with similar ease?
I've seen uncertainty interval / quantile regression done using XGBoost, but it isn't out of the box. I've also been trying to learn some Bayesian modeling, but definitely don't feel handy enough to apply it to random problems needing quick answers at work.
Re: Modeling Uncertainty with PyTorch
#13The field of ML is largely focused on just getting predictions with fancy models. Estimating the uncertainty, unexpectedness and perplexity of specific predictions is highly underappreciated in common practice. Even though it is highly economically valuable to be able to tell to what extent you can trust a prediction, the modelling of uncertainty of ML pipelines remains an academic affair in my experience.
When I was in academia, I used to fit highly sophisticated models (think many-parameters, multi-level non-linear mixed effect models) who were giving not only point estimate but also confidence and predictive intervals ("please explain to me the difference between the two" is one of my favorite interview questions and I still have not heard a correct answer). When I tried to bring an "uncertainty mindset" over when I…
> the difference between the two
One is bigger than the other as far as I remember which means that the standard error of the prediction interval is bigger?
Re: Modeling Uncertainty with PyTorch
#14Earlier quoted context omitted.
When I was in academia, I used to fit highly sophisticated models (think many-parameters, multi-level non-linear mixed effect models) who were giving not only point estimate but also confidence and predictive intervals ("please explain to me the difference between the two" is one of my favorite interview questions and I still have not heard a correct answer). When I tried to bring an "uncertainty mindset" over when I…
Great answer. It prompts a bunch of followup questions! most DS/ML scientists use ML models that typically don't provide an easy way to estimate uncertainty intervals Not an DS/ML scientist but a data engineer. The models I've used have been pretty much "slap it into XGBoost with k-fold CV, call it done" — an easy black box. Is there any model or approach you like to estimate uncertainty with similar ease? I've seen…
Correct, quantile regression is an option. Another is "pure" bootstrapping (you can see by googling something like uncertainty + machine learning + bootstrapping that this is a very active area of current research).
The major problem with bootstrapping is the computational time for big models, since many models need to be fit to obtain a representative distribution of predictions.
Now, if you want more "rigorous" quantification of uncertainty, one option is to go Bayesian using probabilistic programming (PyMC, Stan, TMB), but computational time for large models can be prohibitive. Another option is to "scale down" the complexity to models that might be (on average) a bit less accurate, but provide rigorous uncertainty intervals and good interpretability of results, for example Generalized Additive Models.
A note here is that I saw certain quantification of uncertainty by people who were considered very capable in the ML community that gave me goosebumps, for example since the lower bound of the interval was a negative number and the response variable modeled could not be negative, the uncertainty interval was "cut" at zero (one easy way to deal with it, although it depends on the variable modeled and the model itself, is log-transforming the response—but pay attention to intervals when exp(log(y)) to get back to the natural scale. Another useful interview question.)
Re: Modeling Uncertainty with PyTorch
#15Earlier quoted context omitted.
When I was in academia, I used to fit highly sophisticated models (think many-parameters, multi-level non-linear mixed effect models) who were giving not only point estimate but also confidence and predictive intervals ("please explain to me the difference between the two" is one of my favorite interview questions and I still have not heard a correct answer). When I tried to bring an "uncertainty mindset" over when I…
> the difference between the two One is bigger than the other as far as I remember which means that the standard error of the prediction interval is bigger?
From a good SO answer, see https://stats.stackexchange.com/questions/16493/difference-b...
"A confidence interval gives a range for E[y∣x], as you say. A prediction interval gives a range for y itself.".
In the vast majority of the cases, what we want it the range for y (prediction interval), that is, given x = 3, what is the expected distribution of y?. For example, say we train a model to estimate how the 100-m dash time varies with age. The uncertainty we want is, "at age 48, 90% of Master Athletes run the 100-m dash between 10.2 and 12.4 seconds" (here there would be another difference to point out between Frequentist and Bayesian intervals, but let's make things simple).
We are generally not interested in, given x = 3, what is the uncertainty of the expected value of y (that is, the confidence interval)? In this case, the uncertainty we get (we might want it, but often we do not), is, "at age 48, we are 90% confident that the expected time to complete the 100-m dash for Master Athletes is between 11.2 and 11.6 seconds".
Re: Modeling Uncertainty with PyTorch
#16The field of ML is largely focused on just getting predictions with fancy models. Estimating the uncertainty, unexpectedness and perplexity of specific predictions is highly underappreciated in common practice. Even though it is highly economically valuable to be able to tell to what extent you can trust a prediction, the modelling of uncertainty of ML pipelines remains an academic affair in my experience.
Uncertainty estimates in traditional parametric statistics are facilitated by strong assumptions on the distribution of the data being analyzed.
In traditional nonparametric statistics, uncertainty estimates are obtained by a process called bootstrapping. But there's a trade-off. There's no free lunch!) If you want to eschew strong distributional hypotheses, you need to pay for it with more data and more compute. The "more compute" typically involves fitting variants of the model in question to many subsets of the original dataset. In deep learning applications in which each fit of the model is extremely expensive, this is impractical.
Re: Modeling Uncertainty with PyTorch
#17The field of ML is largely focused on just getting predictions with fancy models. Estimating the uncertainty, unexpectedness and perplexity of specific predictions is highly underappreciated in common practice. Even though it is highly economically valuable to be able to tell to what extent you can trust a prediction, the modelling of uncertainty of ML pipelines remains an academic affair in my experience.
When I was in academia, I used to fit highly sophisticated models (think many-parameters, multi-level non-linear mixed effect models) who were giving not only point estimate but also confidence and predictive intervals ("please explain to me the difference between the two" is one of my favorite interview questions and I still have not heard a correct answer). When I tried to bring an "uncertainty mindset" over when I…
That is really an effect of CS rather than math people dominating ML both in applications and management. My background is in engimeering but always hire a percentage people witb math and business background. In reality there are very few ML applications that don't need confidence estimation and estimation of monetary costs. Else each company will end up having the equivalent of the google graveyard of useless applications.
It really is not that hard.
Re: Modeling Uncertainty with PyTorch
#18Earlier quoted context omitted.
When I was in academia, I used to fit highly sophisticated models (think many-parameters, multi-level non-linear mixed effect models) who were giving not only point estimate but also confidence and predictive intervals ("please explain to me the difference between the two" is one of my favorite interview questions and I still have not heard a correct answer). When I tried to bring an "uncertainty mindset" over when I…
That is really an effect of CS rather than math people dominating ML both in applications and management. My background is in engimeering but always hire a percentage people witb math and business background. In reality there are very few ML applications that don't need confidence estimation and estimation of monetary costs. Else each company will end up having the equivalent of the google graveyard of useless applic…
I agree that statisticians would better than CS people appreciate the importance of uncertainty intervals--it is mostly cultural--but that "In reality there are very few ML applications that don't need confidence estimation and estimation of monetary costs" is empirically false.
If ML application require uncertainty attached to point estimate, we would see plenty more uncertainty intervals attached to point estimates, but in industry, outside of niches (e.g., banking, bio, actuary to name a few), very few bother dealing with them.
I am currently part of a large team (we are talking hundreds) of ML specialists, and I have yet to see a single presentation in which a point estimate was associate with some uncertainty interval. And in my previous company it was the same and when I interview candidates (dozens? hundreds?) I never get a satisfactory answer to the confidence interval vs predictive interval question I ask about.
Re: Modeling Uncertainty with PyTorch
#19Earlier quoted context omitted.
That is really an effect of CS rather than math people dominating ML both in applications and management. My background is in engimeering but always hire a percentage people witb math and business background. In reality there are very few ML applications that don't need confidence estimation and estimation of monetary costs. Else each company will end up having the equivalent of the google graveyard of useless applic…
I agree that statisticians would better than CS people appreciate the importance of uncertainty intervals--it is mostly cultural--but that "In reality there are very few ML applications that don't need confidence estimation and estimation of monetary costs" is empirically false. If ML application require uncertainty attached to point estimate, we would see plenty more uncertainty intervals attached to point estimates…
Let me rephrase your empirical observation in probabilistic terms. If the a random sample of data scientists from startups had the same distribution of mathematicians and CS people than a ramdom sample of data scientists from banking then we could compare empirically whether confidence intervals are equally useful in both industries.
Given that historically regulators figured out that when playing with other people's assets you need to assess your confidence, the volatility of the outcomes in non banking industries that lack such oversight can be greatly attributed to people DunningKrugering after a couple of Andrew NG's courses.
That is my claim and based on my experience working in projects accross many industries accross many countries.
Re: Modeling Uncertainty with PyTorch
#20Earlier quoted context omitted.
> Or is it lacking scalability in practice? Only speaking from my own little perspective in bioinformatics, lack of scalability above all else, both for BNNs and GPs. Sure, the library support could be better, but that was not the main hurdle, more of a friction.
Do you have an anecdotal guess on the scalability barrier maybe? Like does it take too long with more than 10,000 data points having 100 features? Just to get a feel.
The main bottleneck in GP models is the inversion of an NxN covariance matrix, so training with the most straightforward algorithm has cubic complexity (and quadratic memory complexity). 10k instance is what I’ve seen as the limit of tractability.
The input dimensionally doesn’t necessarily matter since it’s kernel method, but if you have many features and want to do feature selection or optimize parameters things can really stack up.
There are scalable approximate inference algorithms, and pretty good library support (gpflow, gpytorch, etc), but it seems like they are not widely known, and there are definitely tradeoffs to consider among the different methods.