Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google Search Terms Can Predict the Stock Market

blogs.smithsonianmag.com

Post reply on HN
Print view View original

41–50 of 57 posts

Re: Google Search Terms Can Predict the Stock Market

#41

Post by PaulHoule »

someben wrote: post #29
PaulHoule wrote: post #9

Earlier quoted context omitted.

Lots of people use the Dow precisely because it is a bad indicator. If they don't like the conclusion they get with something better, like the W5000 or the S&P500, they can try it again with the Dow for a "second opinion." The whole methodology is a joke anyway. If you evaluate a huge number of search terms, some of them are going do better than others. So terms that might mean something ("debt") get mixed up with te…

Logistic regression and other parametric, non-regularized linear learners tend to do poorly with NLP forecasting -type modeling. (They usually overfit.)

You're right.

That's why you build another threshold-type learner and then apply logistic regression to convert the score produced by learner A into a probability score.

Then you can tune at the exact point of the precision-recall curve that maximizes business value.

Re: Google Search Terms Can Predict the Stock Market

#42

Post by someben »

fennecfoxen wrote: post #36
someben wrote: post #27

Earlier quoted context omitted.

What is your source for the ole' "derivatives reduce volatility in the spot market" assumption?

Theory. If you expect that the price of bitcoins is going to be somewhere in the next month, but it's wildly deviated from that on the spot market, there is a monetary incentive to buy/sell bitcoin until the prices are in line with long-term expectations -- discounted for risk, of course. Derivatives like futures options reduce that risk, because you can lock in that future price right now. If anyone's offering, that…

When people use derivatives to hedge their exposure to "bad things happening" and the market is not very liquid (i.e. BTC), arbitrage opportunities can persist. For example, if there is no buyer at an out-of-the-money option strike price I know is overpriced, then the market cannot correct itself even though I am "right." When these mis-pricings stick around for a while, the eventual corrections are often much more extreme. Derivatives reduce overall volatility when the spot market is mature and liquid, which is a far cry from MtGox right now.

Re: Google Search Terms Can Predict the Stock Market

#43

Post by someben »

mikeyouse wrote: post #32
Symmetry wrote: post #20

Earlier quoted context omitted.

I'm (and economists) aren't saying that there are never $10 bills on the sidewalk, just that they don't stay there for long after someone notices them. Stock prices went down on the weekends for decades before someone noticed. But once it was noticed it stopped pretty quickly.

I'm more interested in the persistent market inefficiencies; growth vs. value, January effect, low P/E, all interesting examples that haven't been arbitraged away.

Check out the behavioral finance literature, like the Shleifer book: http://www.amazon.com/Inefficient-Markets-Introduction-Behav...

Re: Google Search Terms Can Predict the Stock Market

#44

Post by mattparcher »

Somewhat related, on a more micro level: some have suggested a trend in which news about Anne Hathaway drives up the price Berkeshire Hathaway’s stock.

Six examples from 2008-2010: http://www.huffingtonpost.com/dan-mirvish/the-hathaway-effec...

A computer scientist who works with hedge funds: "We come across all sorts of strange things in our line of business, strange correlations" http://www.theatlantic.com/technology/print/2011/03/does-ann...

As the intelligence of our technology grows, I find it amusing to consider these less-logical patterns in the algorithms (indirectly) responsible for so much of our economy.

Re: Google Search Terms Can Predict the Stock Market

#45

Post by math »

rm999 wrote: post #3

Cool paper, but the fact that a silly keyword like 'color' is so significant without an author explanation makes me question their results. I really wish they dug deeper and explained unintuitive results like that. There may be a bug in their experiment. Also, why would they use the Dow Jones Industrial Average? It is a ridiculously bad average of stock market performance for many reasons. This planet money podcast g…

For what it's worth, I came across this yesterday and tried replicating their results. I didn't use exactly the same process - I used the S&P 500 and the trading strategy was not exactly the same (because i'm using a framework that makes something else easier to try). But my approach was close enough that I would expect to see similar results if their results were robust enough to be of any use. I didn't get similar results, in fact 'debt' went the other way, and I calculated an associated p value of about 0.8 (i.e. no significance whatsoever). Of course I could have made a mistake, and having come this far going to spend some time double checking everything today, but it'd also surprise me greatly if there was money lying around for the taking this easily..

Re: Google Search Terms Can Predict the Stock Market

#46

Post by bjterry »

mynegation wrote: post #38
fennecfoxen wrote: post #34

Earlier quoted context omitted.

Why would it be illegal? :( I mean, subject to a variety of risk and regulatory issues, perhaps, and not a core competency, and a variety of other things, but... the fundamental reason for insider-trading laws is to make sure the insiders don't abuse their positions and act against the interests of the company's owners (shareholders) by trading in stock tips instead of building shareholder value. If you get knowledge…

The legality of information you obtain is not about whether you obtained it independently, but based on whether it is public knowledge (or can be derived from it) or not. Let's say you are not connected to company X, but you have a friend that works there. One day your friend tells you material fact. Now, your friend probably broke a couple of rules, but _you_ cannot control what people tell you and you cannot be hel…

I believe this is not a fully accurate picture, although the example you give in your second paragraph is definitely accurate.

At a high level you aren't allowed to use "material, non-public information" for investment purposes, but information isn't material just because you can make money off of it in some way, otherwise "channel checks" would be illegal. Material non-public information has to come from insiders of the company, so the only argument that could be made that it was illegal for Google to make investments is based on material information that was being provided to them, via search terms, by corporate insiders. If they are merely using the sentiment exposed by the public to them through search terms that is probably legal. Similarly it's legal for hedge funds to fly planes over department stores and count the cars in their parking lots to gauge the level of business they are seeing at Christmas time, even though this isn't public information.

Re: Google Search Terms Can Predict the Stock Market

#49

Post by asperous »

rubinelli wrote: post #33
ISL wrote: post #21

Earlier quoted context omitted.

Relevant technical terms: 'look-elsewhere effect' and 'false alarm rate' http://en.wikipedia.org/wiki/Look-elsewhere_effect http://en.wikipedia.org/wiki/False_positive_rate There are more similar considerations. Statisticians have been very busy for the last century or so.

As my statistics professor liked to say, "Numbers are meek creatures. If you torture them long enough, they will say whatever you want to hear."

This is the key to the accounting profession

Re: Google Search Terms Can Predict the Stock Market

#50

Post by Iterated »

DoubleMalt wrote: post #7

Google search terms could also predict a completely random time series. You just have to pick the right search terms. If you look at enough words the chances, that some of them have predictive value approaches 100%.

I used to backtest trading indicators a lot and the majority of time the indicator would work on a stock like Apple but wouldn't work on a stock like Tyson Foods. Or any other stock. I was just fitting the curve.
Post reply on HN