Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Numerai – A hedge fund built by a global community of anonymous data scientists

numer.ai

Post reply on HN
Print view View original

61–70 of 120 posts

Re: Numerai – A hedge fund built by a global community of anonymous data scientists

#61

Post by solaarphunk »

lordnacho wrote: post #16

Hedge fund guy here. - You need to know something about the domain in order to make sensible predictions. Is the data daily? Is it per second? Is it ticks? You can't build a sensible model if you don't know that, even if you have good predictions. Relative cost will vary a lot between timescales. - It matters what the features are. Maybe there's some clever reason why it doesn't, but until I hear why I'm going to tak…

Although numerai could control for some of the factors that you mention above, I'm pretty sure they are falling into the data-mining fallacy that most people do when they approach quantitative trading from the outside world. I completely agree with your skepticism of their approach.

While prediction based on data may be valuable in some cases, it isn't robust enough to scale up in any meaningful way. Context matters, like you state above, and most quantitative traders start by taking their contextual knowledge of the markets, and then collecting data on features, and THEN they fit a model to it.

Skipping these steps is only going to lead to a bunch of blowups. I doubt they have any meaningful sharpe that they could scale up or publicly defend with the approach they have taken so far. I'd guess they are paying people VC dollars, not actual market profits right now.

I do think its cool that they have been able to use homomorphic encryption to solve the problem of wanting anonymize data, but I'm not sure it actually helps in this case.

Re: Numerai – A hedge fund built by a global community of anonymous data scientists

#62

Post by evanpw »

dooglius wrote: post #60
jamez1 wrote: post #59

Earlier quoted context omitted.

The data is homomorphically encrypted, meaning you can do operations (such as add and subtract) on the ciphered message and it will also perform them on the underlying data.

Yes, I realize that. The issue is that the result of any operation is also encrypted, which means that there should be no way to connect the target of the training data (encrypted or not) to the output of a function of encrypted data. Suppose the unencrypted data is (a,b,c) where a+b=c, and (x,y,z)=encrypt((a,b,c)). We have an addition function plus on encrypted data such that decrypt(plus(x,y))=a+b=c=decrypt(z), but…

I read through the blog posts, and it seems like the encryption is order-preserving. It's designed to leak enough information to be useful in prediction, but not enough for users to trade on it independently.

Re: Numerai – A hedge fund built by a global community of anonymous data scientists

#63

Post by blahblah3 »

asdfologist wrote: post #39

I don't see how Numerai can avoid the multiple comparisons problem [0]. If people submit thousands of random models, then some subset of them will do a fantastic job in predicting prices in historical simulations but do poorly under real market conditions. As long as the models are black boxes, there's likely no good way to distinguish them from noise. [0] https://en.wikipedia.org/wiki/Multiple_comparisons_problem

This can be mitigated by evaluating all the models on a hold-out test set (similar to what kaggle does and what was done in the netflix prize). The multiple comparisons problem is also mitigated by the fact that the models wont be completely random, there will likely be some positive correlation between them.

edit: Also, by hoeffding's inequality the number of training examples needed for a given level of confidence is only logarithmic in the number of models (even assuming they are independent). See page 6 here: http://cs229.stanford.edu/notes/cs229-notes4.pdf

Re: Numerai – A hedge fund built by a global community of anonymous data scientists

#64

Post by mortehu »

jamez1 wrote: post #56
bigdubs wrote: post #6

Earlier quoted context omitted.

Monkeys throwing darts at the board. This brings to mind the Buffett hedge fund wager, where he invested in a vanguard s&p 500 tracking fund (VFIAX) and a hedge fund actively managed an equal amount, and Mr. Buffett ended up winning handily.

Buffet's wager is a trick, which works on simple people who don't understand statistics. When you take a group of hedge funds, they end up being a very good proxy for the market. At about 20, they will be almost indiscernible. So - a passive fund, buying the market, with lower fees, should outperform.

The Protégé basket of funds contains five hedge funds, not twenty.

The returns are far from indiscernible from S&P 500. At the end of 2015, the S&P 500 portfolio had seen 3x the profits of the Protégé portfolio.

Re: Numerai – A hedge fund built by a global community of anonymous data scientists

#65

Post by gravypod »

Houshalter wrote: post #48
gravypod wrote: post #11

Couldn't I make thousands of fake accounts and submit thousands of slightly different models. Then after a while I could push one or two of my stocks higher in my models artificially. If you represented a large enough % of the "data scientists" in this hedge fund you could make it look like your stock is "definitely" work investment. After they invest in your company, you could take off to the hills. Hell, you don't…

This won't work. As far as I can tell, you don't get to "pick stocks". You just submit predictions for unlabelled variables that could represent anything. It's no different than a standard kaggle competition.

Then could the operators not do this? Just claim that they sent it out to N people and they all suggested the stock. How would you prove they didn't.

Re: Numerai – A hedge fund built by a global community of anonymous data scientists

#66

Post by cjlars »

asdfologist wrote: post #39

I don't see how Numerai can avoid the multiple comparisons problem [0]. If people submit thousands of random models, then some subset of them will do a fantastic job in predicting prices in historical simulations but do poorly under real market conditions. As long as the models are black boxes, there's likely no good way to distinguish them from noise. [0] https://en.wikipedia.org/wiki/Multiple_comparisons_problem

They would need a model built on top of the submitted models to weight or select which specific trading signals to act on. A plausible model might be walkforward testing on out of sample data, or as simple as trailing performance in live trading, or both. Still, with a large set of models, you won't be able to get around multiple comparisons problem entirely.

That doesn't matter though because you only a need small amount of good signal to be profitable in the money management biz. The larger issue is whether the aggregate signal quality is high enough to be able to pay for development and trading costs.

Re: Numerai – A hedge fund built by a global community of anonymous data scientists

#67

Post by jamez1 »

mortehu wrote: post #64
jamez1 wrote: post #56

Earlier quoted context omitted.

Buffet's wager is a trick, which works on simple people who don't understand statistics. When you take a group of hedge funds, they end up being a very good proxy for the market. At about 20, they will be almost indiscernible. So - a passive fund, buying the market, with lower fees, should outperform.

The Protégé basket of funds contains five hedge funds, not twenty. The returns are far from indiscernible from S&P 500. At the end of 2015, the S&P 500 portfolio had seen 3x the profits of the Protégé portfolio.

In his case, these were 5 funds of hedge funds. Which underneath could be much more than 20 funds.

You're also forgetting how much the compounding of the fees can cost, and you have two layers of them. This could easily make the S&P at least 3x higher in returns after 10 years.

Re: Numerai – A hedge fund built by a global community of anonymous data scientists

#68

Post by jamez1 »

dooglius wrote: post #60
jamez1 wrote: post #59

Earlier quoted context omitted.

The data is homomorphically encrypted, meaning you can do operations (such as add and subtract) on the ciphered message and it will also perform them on the underlying data.

Yes, I realize that. The issue is that the result of any operation is also encrypted, which means that there should be no way to connect the target of the training data (encrypted or not) to the output of a function of encrypted data. Suppose the unencrypted data is (a,b,c) where a+b=c, and (x,y,z)=encrypt((a,b,c)). We have an addition function plus on encrypted data such that decrypt(plus(x,y))=a+b=c=decrypt(z), but…

They could just be normalizing every data point to be between 0 and 1 by dividing by the range. That's a homomorphic encryption.. it passes your weird assumptions.

I don't know why you're harping on about sound encryption, the point of this is to keep the statistical information intact in the cipher, without giving away the underlying market data.

Re: Numerai – A hedge fund built by a global community of anonymous data scientists

#69

Post by numbas »

dsacco wrote: post #10
bigdubs wrote: post #6

Earlier quoted context omitted.

Monkeys throwing darts at the board. This brings to mind the Buffett hedge fund wager, where he invested in a vanguard s&p 500 tracking fund (VFIAX) and a hedge fund actively managed an equal amount, and Mr. Buffett ended up winning handily.

It's easy to be dismissive of this, especially by working from the popular Warren-Buffet-index-funds-beat-hedge-funds story that makes the rounds. It's true that most hedge funds and active traders lose money (or at least, underperform the market). But as I am fond of pointing out, there are a non-negligible number of funds and traders who consistently and demonstrably earn significantly stronger returns than the mar…

Check their board of directors -- all very smart and established people.

A former tournament winner did well on both the public and private leaderboard. It is very difficult to do this by luck. He was also a student from Bangladesh who got the opportunity to play with hedge fund data with just the cost of an internet connection and zero risk for messing it up. Should he start working for a hedge fund now, without any finance experience? Could be a good bet. Numerai would still beat him, because they can aggregate all the top models into an ensemble. It is hard to beat 50 individuals, but near impossible to beat a team of 50 competitors. Compare the variance of a single decision tree with a Random Forest.

Re: Numerai – A hedge fund built by a global community of anonymous data scientists

#70

Post by wyager »

delegate wrote: post #26

"It's just a pure math problem. It's like a math competition. You don't need to know anything about finance, you don't know anything about hedge funds... you don't even have to speak English..." This is not a pure math problem. Eventually the outcome of all these models and predictions affects the stock prices and - if it becomes as successful as you hope - the economy as a whole. And the physical world: people, anim…

> than this data juggling deep learning bullshit which results in profits being paid out to a bunch of intelligent, greedy and unwise people.

Holy anti-intellectualism, Batman!

If you describe machine learning as "data juggling bullshit", this is very strong evidence that you simply don't understand what it is. This is an indictment of you, not of machine learning. Machine learning would more accurately be called "applied computational statistics" in 99% of cases.

What makes you think that the people using applied statistics to make money are "unwise"? Based on your tone, I would guess that it's because what they're doing doesn't agree with your folk definition of "an honest day's work" or something like that. This isn't really a good criticism; it just means you don't see the utility of what they're doing, which requires some degree of abstract thinking about the market.

Post reply on HN