Macroeconomics resembles a science in exactly two ways: it looks at history, and it makes predictions (or prescribes courses of action; these are equivalent).
Greek mythology resembled a science in those same two ways.
Why Economic Models are Always Wrong
41–50 of 61 posts
Re: Why Economic Models are Always Wrong
#42Macroeconomics resembles a science in exactly two ways: it looks at history, and it makes predictions (or prescribes courses of action; these are equivalent). Greek mythology resembled a science in those same two ways.
Economics ... makes predictions (or prescribes courses of action; these are equivalent).
This is absolutely false, except at the micro level. In terms of policy, economics can only inform us of the relative costs of various alternatives. It cannot tell us which alternative is right.
Consider the question of free trade. Virtually every economist agrees that free trade improves total efficiency (viz the Law of Comparative Advantage). However, at the margins, it may harm some individuals. Economics cannot tell us if it is morally right to incur those individualized harms in order to improve the lot of the whole, nor what if anything we should do to make whole those who were affected.
Economics makes predictions, giving us insights. Our morals are then needed to prescribe courses of action.
Re: Why Economic Models are Always Wrong
#43Earlier quoted context omitted.
I've thought there was more opportunity in fundamentals up until Warren Buffet and Ben Graham's the intelligent investor became well known. More people tried to use these methods, thereby increasing demand and decreasing the upside on securities that meet Graham and Buffets criteria. The stock market today is very different from when they got going, although long term I don't know that anything has fundamentally chan…
Wait, if there was more "opportunity in fundamentals" back then it would mean that stocks were further away from their fundamentals, right? That's pretty much the opposite of what the OP is complaining about.
This was probably the most insightful thing I've seen all week. Thanks for making me smile.
Re: Why Economic Models are Always Wrong
#44Great discussion! The author doesn't seem to introduce the concept of training/testing datasets which absolutely critical to obtaining any reasonable model. So I don't buy the author's thesis that economic models are always wrong. The solution to the hypothetical problem posed in the article is to separate the historical dataset into training and testing groups. The models should be generated while only 'seeing' the…
"training/testing datasets which absolutely critical to obtaining any reasonable model"
This is partly correct but, in general, too strong.
Am I commenting on the OP? Not really!
Why too strong? Because it assumes too little and sometimes more information is available and with the extra information a 'testing data set' may not be needed.
Why are 'testing data sets' important? If about all you have to go on is the 'historical data' and then are just searching for a 'model' based mostly just on what 'fits' the data, then, sure, a 'testing data set' will likely be just crucial. One way to get such a 'testing data set' is to partition the 'historical data' into two parts, use the first to 'fit' a model and the second to 'test' the fit. Of course, there are still risks: If fit 10,000 models, find 10 that fit well and test each of the 10 with the 'testing data set' and accept the model that fits the testing data the best, then still may have some problems from a 'generalized version of overfitting'! As I recall, there has been some mathematical statistics to address this issue.
Where can get by without a 'testing data set'? Broadly if know more than the meager assumptions common in 'machine learning' or 'curve fitting'.
What more can be known? In principle the variety is large.
Examples? Sure: Broadly just simple, old 'regression analysis', looked at as statistical estimation, makes a long list of quite detailed assumptions. E.g., we assume that there is a model the works and that we know in good detail the form of that model. We assume a lot about the 'historical data' we have, E.g., we assume 'homoscadasticity' and mean zero, independent and identically distributed (i.i.d.) Gaussian for the errors. We make some assumptions about dimensionality (e.g., to get around 'overfitting'). Then the usual derivations give minimum variance, unbiased estimates of the unknown parameters and more, all without any use of 'testing data'. "Look Ma, no testing data required!".
"Yes, son, but as your father kept telling you, a LOT of assumptions are required, and the assumptions are not all easy to verify. Or the regression derivations are a nice logical trip from island A to island B we would like to get to but we don't always know how to get to island A.".
Other examples? Sure: Calculate the trajectory of a space craft doing 'slingshots' in the inner solar system and then reaching, say, Saturn. We start with Newton's second law, his law of gravity, maybe a little about the solar wind, a lot of details about the orbits of the planets, and do some good numerical work with an initial value problem of an ordinary differential equation. We build a 'model' but don't really 'fit for parameters' or use 'historical data' and have no real use for 'testing data'. Why? Because we believe in Newton's laws and our numerical work. A 'model'? Yes. Fitting 'parameters'? No,
Can there be a connection between space craft trajectories and economic models? Sure: Bring more assumptions than just curve fitting. An example is to bring, essentially, accounting. So, then can get a Leontief input/output model. We bring basically just accounting data and not other historical data, do no real 'parameter' estimation, and use no 'testing' data. If the input data is noisy, then, sure, so will be the output and we might do some work with confidence intervals. Still we don't check with 'testing data'.
More examples? Sure: The broad field, with many techniques, of distribution-free statistical hypothesis testing is based on historical data and some assumptions and really needs no testing data. What is obtained is much like a 'model' where can plug in new data and get the intended results. The assumptions are typically that the data is i.i.d.
Net, a lot can be done beyond the common approach of machine learning curve fitting.
Re: Why Economic Models are Always Wrong
#45Macroeconomics resembles a science in exactly two ways: it looks at history, and it makes predictions (or prescribes courses of action; these are equivalent). Greek mythology resembled a science in those same two ways.
Stop tarring microeconomics with the brush you used on macro.
Re: Why Economic Models are Always Wrong
#46Macroeconomics resembles a science in exactly two ways: it looks at history, and it makes predictions (or prescribes courses of action; these are equivalent). Greek mythology resembled a science in those same two ways.
Stop tarring microeconomics with the brush you used on macro.
You're right, updated.
Re: Why Economic Models are Always Wrong
#47I think what the author is describing is simple overfitting. http://en.wikipedia.org/wiki/Overfitting It is quite a newbie mistake for a scientist to be surprised by it. It affects every kind of modelling. I thought maybe this article would talk about why economic models are worst than other kinds of models. There are issues that arise when applying scientific models to the economy caused by the fact that when even g…
Alternatively, it may be simple information theory: A model that takes in 100 bits of specification simply can not correctly describe a process that has 10,000 bit's worth of degrees of freedom. And that's before we talk about iteration over time, and before we get to the final killer you mention, which is when the models are ruined by their own application to the domain. I think radical underspecification is much mo…
You're right, and overfitting cannot be an explanation for this phenomenon -- when there are many equally valid alternative outcomes to a problem, which is what's being described, the solution is underdetermined by definition.
In the (ML) terms I'm used to, it is as an error surface with many local minima. That is, if you start out with a guess for the parameters and try to progressively optimize the cost function to reach a point where the error is lowest (i.e. the tangent of the error is 0), where you end if is extremely dependent on where you start out. When you find a local minimum, you have found a point where there is no nearby point that is better, but there may be some other point (or many) somewhere else in the model that is better. The very best one is the global minimum.
This is a well known problem in ML for non-convex error functions, and there are various methods for trying to avoid local minima and reach a global minimum.
But this case is actually worse than that -- it is an error surface with many global minima. Each is effectively a perfect fit for the data to date, but give different predictions about future data. Since each function is a perfect fit, it is literally impossible to predict the proper parameters. Which is what underspecification is.
Re: Why Economic Models are Always Wrong
#48Macroeconomics resembles a science in exactly two ways: it looks at history, and it makes predictions (or prescribes courses of action; these are equivalent). Greek mythology resembled a science in those same two ways.
Economics ... makes predictions (or prescribes courses of action; these are equivalent). This is absolutely false, except at the micro level. In terms of policy , economics can only inform us of the relative costs of various alternatives. It cannot tell us which alternative is right . Consider the question of free trade. Virtually every economist agrees that free trade improves total efficiency (viz the Law of Compar…
You are right. To prescribe a course of action is to make a prediction (that this action will lead to results in some way superior to alternative courses). The reverse, as you correctly point out, is not necessarily the case.
Re: Why Economic Models are Always Wrong
#49This is known to anyone who's ever monkeyed with any type of machine learning: genetic algorithms, Bayesian filters, anything. I agree with many of the commenters in this article. This should be common knowledge. I also, like many commenters, couldn't help but think of model-based climate predictions.
"All models are wrong. Some models are useful."
- George Box
Re: Why Economic Models are Always Wrong
#50This is known to anyone who's ever monkeyed with any type of machine learning: genetic algorithms, Bayesian filters, anything. I agree with many of the commenters in this article. This should be common knowledge. I also, like many commenters, couldn't help but think of model-based climate predictions.
The problem is that you're comparing statistical methods with process-based methods. The mathematically inclined tend to have a reflex to approach modeling wiht this sort of black box methods. The thing is that for modeling processes like geomorphology, hydrology but also less quantitative processes like quality of life in urban environments, black box methods cannot be verified nor reasoned about - with issues like overfitting etc. becoming a problem.
On the other hand, you can model by building conceptual models, calibrating them by hand (using computer methods for the number crunching only) and reasoning about divergences between model results and observed data rather than computing them away with raw power. This is what modeling should be about - a tool for understanding.
(this topic is dear to my heart - I have had this discussion so often. Models are not crystal balls, they are tools for understanding processes. Which is why I am so desperate when another economist, mathematician or computer scientist stands up and wants to model processes that require understanding with their barbaric brute force statistical methods to not have to study things that are outside of their comfort zone. When all you have is a hammer etc.)