Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Economic Models are Always Wrong

scientificamerican.com

Post reply on HN
Print view View original

31–40 of 61 posts

Re: Why Economic Models are Always Wrong

#31

Post by Radim »

jerf wrote: post #18
BenoitEssiambre wrote: post #13

I think what the author is describing is simple overfitting. http://en.wikipedia.org/wiki/Overfitting It is quite a newbie mistake for a scientist to be surprised by it. It affects every kind of modelling. I thought maybe this article would talk about why economic models are worst than other kinds of models. There are issues that arise when applying scientific models to the economy caused by the fact that when even g…

Alternatively, it may be simple information theory: A model that takes in 100 bits of specification simply can not correctly describe a process that has 10,000 bit's worth of degrees of freedom. And that's before we talk about iteration over time, and before we get to the final killer you mention, which is when the models are ruined by their own application to the domain. I think radical underspecification is much mo…

Re. your eloquent bracket: the "true number of bits", as in "the most compact description of any given state of a system", is in general uncomputable (see Kolmogorov complexity), for any sufficiently powerful language of description.

If you require notions such as "the true minimum number of bits" to be practical, you have to put additional restrictions on the language by which you describe the system -- such as your probability model. The representation does matter.

Re: Why Economic Models are Always Wrong

#32

Post by mdda »

highfreq wrote: post #26

A "scientist" finds by cross-validation that his model is over fitting the data. Luckily it wasn't published by a reputable source of science journalism. http://en.wikipedia.org/wiki/Cross-validation_(statistics) Also who the heck is Wilmott? He just pops up in the last paragraph with no introduction.

He's pretty well know in the Quant community : http://wilmott.com/about.cfm , but I also had to double-check the article to see where he was introduced. I guess there was some heavy-handed editing.

Re: Why Economic Models are Always Wrong

#33

Post by jeffdavis »

BenoitEssiambre wrote: post #13

I think what the author is describing is simple overfitting. http://en.wikipedia.org/wiki/Overfitting It is quite a newbie mistake for a scientist to be surprised by it. It affects every kind of modelling. I thought maybe this article would talk about why economic models are worst than other kinds of models. There are issues that arise when applying scientific models to the economy caused by the fact that when even g…

"I think what the author is describing is simple overfitting."

It doesn't look like overfitting to me. The input data is perfect, and the model is perfect, so it doesn't look like overfitting can occur.

Re: Why Economic Models are Always Wrong

#34

Post by jeffdavis »

amalcon wrote: post #23
jerf wrote: post #18

Earlier quoted context omitted.

Alternatively, it may be simple information theory: A model that takes in 100 bits of specification simply can not correctly describe a process that has 10,000 bit's worth of degrees of freedom. And that's before we talk about iteration over time, and before we get to the final killer you mention, which is when the models are ruined by their own application to the domain. I think radical underspecification is much mo…

Here's the thing: you're both right. It's both radically underspecified and overfitted. The information-theoretic argument demonstrate that a model cannot exactly match the reality unless it's as complex as the reality. This article speaks of the separate problem that economic models are not evaluated in any sort of experiments, and thus are prone to overfitting. This makes them unlikely to even approximate well. Con…

"It's both radically underspecified and overfitted."

He used a perfect model (of a hypothetical world) which had exactly the right parameters, and then he calibrated it using exactly correct data.

So I don't see how this could be underspecified or overfitted. Can you please explain?

"The information-theoretic argument demonstrate that a model cannot exactly match the reality unless it's as complex as the reality."

In this case he defined his model to be reality.

Re: Why Economic Models are Always Wrong

#35

Post by andylei »

danmaz74 wrote: post #29
andylei wrote: post #12

Earlier quoted context omitted.

> So, even if you had a perfect model for the market without you applying your model Actually, most trader's models do take market impact into account. If you had a perfect model for the market, I'm pretty sure that you (as a participant) would be included. In fact, your own actions are the easiest part of the model to get right, because you control them entirely.

Ok, but you would need to take into account the interaction with other traders' models that are put into play all the time...

yup, building good models is hard. doesn't mean they always produce bad results. billions of dollars of quant hedge fund money prove that.

Re: Why Economic Models are Always Wrong

#36

Post by DevX101 »

Great discussion! The author doesn't seem to introduce the concept of training/testing datasets which absolutely critical to obtaining any reasonable model. So I don't buy the author's thesis that economic models are always wrong.

The solution to the hypothetical problem posed in the article is to separate the historical dataset into training and testing groups. The models should be generated while only 'seeing' the training data. You will, as the author mentioned, get many models that appear to fit the data. Most of these models will be garbage.

The fun part is when the testing data is introduced against the many models generated above. Most of the models will completely bomb, but a handful may actually predict the previously 'unseen' testing data with high accuracy. Those few models which pass the testing stage are the ones worth their salt.

Due to the self-aware nature of the markets, successful models probably will not be true indefinitely, but it's very possible they may be true long enough to be profitable. The less known your successful models are, the longer they will be successful predictors of the market. Hence why successful quant funds are notoriously secretive with their approaches. Open source would never work in finance.

Re: Why Economic Models are Always Wrong

#37

Post by amalcon »

jeffdavis wrote: post #34
amalcon wrote: post #23

Earlier quoted context omitted.

Here's the thing: you're both right. It's both radically underspecified and overfitted. The information-theoretic argument demonstrate that a model cannot exactly match the reality unless it's as complex as the reality. This article speaks of the separate problem that economic models are not evaluated in any sort of experiments, and thus are prone to overfitting. This makes them unlikely to even approximate well. Con…

"It's both radically underspecified and overfitted." He used a perfect model (of a hypothetical world) which had exactly the right parameters, and then he calibrated it using exactly correct data. So I don't see how this could be underspecified or overfitted. Can you please explain? "The information-theoretic argument demonstrate that a model cannot exactly match the reality unless it's as complex as the reality." In…

Those particular statements referred to some representative economic model, not the experiment in question. In the experiment in question, the model is fully specified by definition.

As far as overfitting goes, that applies when you have a parameterized general model and need to discover the correct parameters. You probably won't get the exact correct parameters; instead, you'll (hopefully) get parameters that approximate reality well.

More closely matching the training data can actually make it a worse approximation in the general case.

Re: Why Economic Models are Always Wrong

#38

Post by pvarangot »

Every model is "wrong", by definition of it being a "model" and not "reality". It's one of the few mind opening things I've learnt at university.

That's not a problem if you take it as an incentive to improve how much you know about the real world. It's a problem when you put the model before the people, and say that "models got us in trouble because of calibration problems".

An economic crisis is not an unavoidable natural disaster, it's people screwing up other people.

Re: Why Economic Models are Always Wrong

#39

Post by danmaz74 »

andylei wrote: post #35
danmaz74 wrote: post #29

Earlier quoted context omitted.

Ok, but you would need to take into account the interaction with other traders' models that are put into play all the time...

yup, building good models is hard. doesn't mean they always produce bad results. billions of dollars of quant hedge fund money prove that.

"essentially, all models are wrong, but some are useful" George EP Box - one of the greatest truths imho :)
Post reply on HN