Earlier quoted context omitted.
Really smart monkeys. And monkeys which will (hopefully) be self-correcting to converge on the bulls-eye. Most people don't realize that the "markets" are 49% random, 48% sentiment driven, and 3% fundamentals. If you approach the problem with that assumption held true, monkeys throwing darts isn't such a horrible mechanism for investing. See: "Monkeys Are Better Stockpickers Than You'd Think: Why dart-throwing primat…
> Most people don't realize that the "markets" are 49% random [...] If you approach the problem with that assumption held true, monkeys throwing darts isn't such a horrible mechanism for investing. Maybe I haven't understood that part of data science but I never got why throwing more unpredictability on an already unpredictable data source would somehow make it more predictable.
I guess another way of saying it is that your mess is starting to look more like their mess.
Anything that deals with the future is inherently non-predictable. Using chaos theory as a framework, we say it is unpredictable because we are unable (and will always be unable) to model the currently system completely. There will always be data that was not captured hiding between the data that was captured. Follow the arrow-of-time far enough out into the future and that non-captured data will manifest itself in the captured data, thereby (usually) creating a deviation from the modeled future.
To get around this we use statistics and probability. We rely on the law of large numbers and regression to the mean. In other words, we hope that the future won't get too weird and will be similar enough to the past, within some confidence interval.
So, we're not really predicting a specific outcome, we're predicting that the outcome will be some point within some confidence interval.
The reason we can get better at this, is as we capture more data we can better guess the inputs and assumptions we use to create the model. We throw out stuff that didn't happen to be relevant. We discover stuff that we should have considered relevant. If we're lucky the model closely follows the physical laws of our reality and we can apply the frameworks so arduously worked out by chemists, physicists, biologists, etc. If we're not so lucky we're dealing with sentiment, conjecture, or any of the other human inputs of the financial markets, and we are forced to make up formulas that work until they fail spectacularly.