131–140 of 165 posts
I have, in my life as a web developer, had multiple "academics" urgently demand that i remove error bands, bars, notes about outliers, confidence intervals etc from graphics at the last minute so people are not "confused" Its depressing
Earlier quoted context omitted.
Agreed! I also think it's extremely important as practitioners to know what we're even trying to estimate. Expected value (i.e. least squares regression) is the usual first thing to go for, does that even matter? We're probably actually interested in something like an upper quantile for planning purposes. And then the whole model component of it, the interval that's being simultaneously estimated is model driven and…
> We're probably actually interested in something like an upper quantile for planning purposes. True. But a conditional quantile is much harder to accurately estimate from data than a conditional expectation (particularly if you are talking about extreme quantiles).
Earlier quoted context omitted.
They can also mean pushed forward uncertainty from input parameters which isn't exactly the same as model error
I'm not sure I see the distinction. Would you mind clarifying?
Input: temperature is measured at -1° C.
Prediction: water will freeze.
Actual: water didn't freeze.
Actual temperature: 2° C.
The model isn't broken, it gives an incorrect result because of input error.
Earlier quoted context omitted.
> is it, loosely speaking, an expectation about a mean/true value or about the distribution of unobserved values If you don't mind typing it out, what do you mean formally here?
I think they mean either what is E[x| y] (standard regression point estimate) along with a confidence interval (this assumes that the mean is a meaningful quantity), or the interval s.t. F(x | y) -- the PDF of x -- is between .025 and .975 (the 95% predictive interval centered around .5). The point is that the width of the confidence interval around the point estimate of the mean converges to 0 as you add more data b…
If you think about linear regression, it makes sense, given the assumptions of linear regression, that confidence interval E[x|y] is narrower around the mean of x and y.
If I had to choose between the two, confidence intervals in a forecasting context are less useful in the context of decision-making, while prediction intervals are, in my opinion, always needed.
Earlier quoted context omitted.
It is a very common misconception and one of my technical crusades. I keep fighting, but I think I have lost. Not knowing what the "uncertainty interval" represents (is it, loosely speaking, an expectation about a mean/true value or about the distribution of unobserved values?) could be even more dangerous, in theory, than using no uncertainty interval at all. I say in theory because, in my experience in the tech ind…
Agreed! I also think it's extremely important as practitioners to know what we're even trying to estimate. Expected value (i.e. least squares regression) is the usual first thing to go for, does that even matter? We're probably actually interested in something like an upper quantile for planning purposes. And then the whole model component of it, the interval that's being simultaneously estimated is model driven and…
I have more doubts when it comes to actions taken when considering properly estimated predictive intervals. Even I, who have a good knowledge of statistical modeling, after hearing "the median survival time for this disease is 5 years," do not stop to think that the median is calculated/estimated on an empirical distribution, so there are people who presumably die after 2 years, others after 8. Well, that depends on the variance.
But if I am so strongly drawn to a central estimate, is there any chance for others not so used to thinking about distributions?
Uncertainty quantification is a neglected aspect of data science and especially machine learning. Practitioners do not always have the statistical background, and the ML crowd generally has a "predict first and asks questions later" mindset that precludes such niceties. I always demand error bars.
You can demand error bars but they aren't always possible or meaningful. You can more or less "fudge" some sort of normally distributed IID error estimate onto any method, but that doesn't necessarily mean anything. Generating error bars (or generally error distributions) that actually describe the common sense idea of uncertainty can be quite theoretically and computationally demanding for a general nonlinear model…
For instance, if you look at https://blog.tensorflow.org/2019/03/regression-with-probabil... until the case 4 it's easy to follow and digest, but if you look at the _Tabula rasa_ section I am pretty sure that such content isn't understandable by many. Where you get stuck because the ideas become too complex depends on your math skills.
Two things I think are interesting here, one discussed by the author and one not. (1) As mentioned at the bottom, forecasting usually should lead to decisionmaking, and when it gets disconnected, it can be unclear what the value is. It sounds like Rosenfield is trying to use forecasting to give added weight to his statistical conclusions about past data, which I agree sounds suspect. (2) it's not clear what the "erro…
A position espoused by Bill Phillips [1], and to which I now adhere: "You should be willing to take either side of the bet that confidence interval implies." (paraphrasing; he says it better). For a concrete example, with a 95% confidence interval, you should be as willing to accept the 19:1 odds that the true value is outside the interval as you are the 1:19 odds that the true value is inside the interval. Aside fro…
Earlier quoted context omitted.
I'm a sense, that makes my point. Why do they persist with models that don't represent reality despite knowing it ? Eventually you must realise that adding epicycles isn't going to cut it, yet still the sage voices echo the standard dogma when economies are dragged into the doldrums by policy posed by useless models. Bought into is not the same as believing.
> Why do they persist with models that don't represent reality despite knowing it? Why do physicists ignore friction whenever possible? In general, for any task, you take the simplest model that represents the aspects of reality that you care about. But you stay aware of the limits. That's true in physics or engineering just as much as in economics. That's why NASA uses Newtonian mechanics for all their rocket scienc…
For example you say a basic policy is a tax on guns instead of a ban. First of all I dispute that is even orthodox economics. Second there is some strong evidence that gun bans reduce violence.
Free migration is another one. It is an insanely complicated issue in the real world. No country has 100% free migration or they wouldn’t be a country. There are all kinds of very complex rules and effects of these rules. And it is not clear that “free migration” is “good”. (I am sure the native americans probably didn’t like the free migration)
Earlier quoted context omitted.
It's worse than that. Oftentimes the skin in the game provides a motivation to mislead. C.f. most of the economics profession.
How do economists have skin in the game? Many of them eg work in universities and some even have tenure. There's not much skin in the game between any forecasts they might make and their academic prospects. Economists working for companies often have to help them understand micro and macro-economics. Eg (some of) Google's economists help them design the ad auctions. It's relatively easy to figure out for Google how w…