Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Numerai – A hedge fund built by a global community of anonymous data scientists

numer.ai

Post reply on HN
Print view View original

101–110 of 120 posts

Re: Numerai – A hedge fund built by a global community of anonymous data scientists

#101

Post by dooglius »

jamez1 wrote: post #68
dooglius wrote: post #60

Earlier quoted context omitted.

Yes, I realize that. The issue is that the result of any operation is also encrypted, which means that there should be no way to connect the target of the training data (encrypted or not) to the output of a function of encrypted data. Suppose the unencrypted data is (a,b,c) where a+b=c, and (x,y,z)=encrypt((a,b,c)). We have an addition function plus on encrypted data such that decrypt(plus(x,y))=a+b=c=decrypt(z), but…

They could just be normalizing every data point to be between 0 and 1 by dividing by the range. That's a homomorphic encryption.. it passes your weird assumptions. I don't know why you're harping on about sound encryption, the point of this is to keep the statistical information intact in the cipher, without giving away the underlying market data.

What I'm talking about is indistinguishability [0], [1] which shows up in definitions of homomorphic encryption (e.g. [2], [3]). If the data is only normalized, or even encrypted in an order-preserving way, it seems possible to figure out information about the underlying data (e.g. if the target is whether a symbol moves up or down, and if you can figure out what even one of the features refer to, there's enough information to turn your model on the data into predictions you can just trade on).

[0] https://en.wikipedia.org/wiki/Computational_indistinguishabi... [1] https://en.wikipedia.org/wiki/Ciphertext_indistinguishabilit... [2] http://cs.au.dk/~stm/local-cache/gentry-thesis.pdf [3] https://arxiv.org/ftp/arxiv/papers/1305/1305.5886.pdf

Re: Numerai – A hedge fund built by a global community of anonymous data scientists

#102

Post by hal9000xp »

lordnacho wrote: post #16

Hedge fund guy here. - You need to know something about the domain in order to make sensible predictions. Is the data daily? Is it per second? Is it ticks? You can't build a sensible model if you don't know that, even if you have good predictions. Relative cost will vary a lot between timescales. - It matters what the features are. Maybe there's some clever reason why it doesn't, but until I hear why I'm going to tak…

I think hedge fund's founders motivation behind this idea could be like this:

Let's give many people some [encrypted] data to play with. May be someone will find a curious and unexpected pattern we didn't looked at before. May be this idea won't be workable straight away but we [hedge fund] are going to investigate this idea further and probably use this in our trading strategy.

In short, they are probably doing ideas mining using power of crowd.

Re: Numerai – A hedge fund built by a global community of anonymous data scientists

#103

Post by muton »

hal9000xp wrote: post #102
lordnacho wrote: post #16

Hedge fund guy here. - You need to know something about the domain in order to make sensible predictions. Is the data daily? Is it per second? Is it ticks? You can't build a sensible model if you don't know that, even if you have good predictions. Relative cost will vary a lot between timescales. - It matters what the features are. Maybe there's some clever reason why it doesn't, but until I hear why I'm going to tak…

I think hedge fund's founders motivation behind this idea could be like this: Let's give many people some [encrypted] data to play with. May be someone will find a curious and unexpected pattern we didn't looked at before. May be this idea won't be workable straight away but we [hedge fund] are going to investigate this idea further and probably use this in our trading strategy. In short, they are probably doing idea…

From the competition rules: "You retain all intellectual property rights to your model. You never have to tell anyone how you built it and you never have to tell us who you are. You only upload your predictions."

Re: Numerai – A hedge fund built by a global community of anonymous data scientists

#105

Post by kriro »

delegate wrote: post #87
wyager wrote: post #70

Earlier quoted context omitted.

> than this data juggling deep learning bullshit which results in profits being paid out to a bunch of intelligent, greedy and unwise people. Holy anti-intellectualism, Batman! If you describe machine learning as "data juggling bullshit", this is very strong evidence that you simply don't understand what it is. This is an indictment of you, not of machine learning. Machine learning would more accurately be called "ap…

> What makes you think that the people using applied statistics to make money are "unwise"? Because it is an incredible waste of talent. I know this is in contradiction with all the 'values' that have been drilled into our minds since we were born, but it's about time we wake up and reorder our priorities. >Based on your tone, I would guess that it's because what they're doing doesn't agree with your folk definition…

The curse of Plato strikes again. You (or other eiltes) who know better than the actual people doing what they do that they waste their time and could use it in more meaningful/fruitful ways.

The fact that these people chose to spend their time doing numerai probably means that they think that's the best use of their time.

Re: Numerai – A hedge fund built by a global community of anonymous data scientists

#106

Post by gravypod »

Houshalter wrote: post #76
gravypod wrote: post #65

Earlier quoted context omitted.

Then could the operators not do this? Just claim that they sent it out to N people and they all suggested the stock. How would you prove they didn't.

Well the hedge fund manager can invest in whatever stocks he wants already, so I don't see the point of this scheme.

The manager could take bribes to invest in a few failing companies and cover his ass while doing so by saying "anonymous statisticians" told him to.

Re: Numerai – A hedge fund built by a global community of anonymous data scientists

#108

Post by wyager »

Houshalter wrote: post #77
wyager wrote: post #70

Earlier quoted context omitted.

> than this data juggling deep learning bullshit which results in profits being paid out to a bunch of intelligent, greedy and unwise people. Holy anti-intellectualism, Batman! If you describe machine learning as "data juggling bullshit", this is very strong evidence that you simply don't understand what it is. This is an indictment of you, not of machine learning. Machine learning would more accurately be called "ap…

I read it mainly as a criticism of this site and this application of deep learning. Predicting stock prices slightly better doesn't really make the world a better place or improve anyone's lives. But it does consume the time and resources of very intelligent people who might be able to do incredible things elsewhere. Particularly the problem of this website, is it's huge focus on anonymity. You don't need to expose y…

> Predicting stock prices slightly better doesn't really make the world a better place or improve anyone's lives.

Actually, it does. Increasing liquidity, communicating price information, more accurately accounting for predictable future changes in price, and absorbing risk allow for increased agricultural and industrial production. Like I said, you have to think through a few layers of abstraction.

A nice rule of thumb is that if someone is getting paid to do something you personally think is useless, it's either a government job or you just don't get what they're doing, but the market does.

Re: Numerai – A hedge fund built by a global community of anonymous data scientists

#109

Post by wyager »

pyrophane wrote: post #75
wyager wrote: post #70

Earlier quoted context omitted.

> than this data juggling deep learning bullshit which results in profits being paid out to a bunch of intelligent, greedy and unwise people. Holy anti-intellectualism, Batman! If you describe machine learning as "data juggling bullshit", this is very strong evidence that you simply don't understand what it is. This is an indictment of you, not of machine learning. Machine learning would more accurately be called "ap…

I didn't read it as a criticism of machine learning or intellectuals, but rather of the many bright minds that are trying to make money off of the stock market by writing software to make better trades. This kind of activity doesn't create value or even, to my knowledge, liquidity. I believe he's suggesting that there are a lot of other areas where their knowledge and skills could be applied which would create some s…

> This kind of activity doesn't create value or even, to my knowledge, liquidity.

It does both, the latter by more efficiently communicating price information and incorporating future events into asset prices. More accurate AI traders absorb risk, which is good for the production economy.

Re: Numerai – A hedge fund built by a global community of anonymous data scientists

#110

Post by wyager »

delegate wrote: post #87
wyager wrote: post #70

Earlier quoted context omitted.

> than this data juggling deep learning bullshit which results in profits being paid out to a bunch of intelligent, greedy and unwise people. Holy anti-intellectualism, Batman! If you describe machine learning as "data juggling bullshit", this is very strong evidence that you simply don't understand what it is. This is an indictment of you, not of machine learning. Machine learning would more accurately be called "ap…

> What makes you think that the people using applied statistics to make money are "unwise"? Because it is an incredible waste of talent. I know this is in contradiction with all the 'values' that have been drilled into our minds since we were born, but it's about time we wake up and reorder our priorities. >Based on your tone, I would guess that it's because what they're doing doesn't agree with your folk definition…

> in fact it is unclear if we're going to make it to the next century as a species.

The only way we're going to wipe ourselves out is with nukes, nanotech, or engineered viruses. Global warming is not an insurmountable concern. It will be expensive for coastal cities, though.

There's no reason we can't focus on multiple things. Global warming isn't so pressing that we need to dedicate all intellectual output to it. Every 10 or 20 years, the environmental alarmists come up with something new to obsess over (I think it was landfills previously), and they're doing it now with global warming.

Post reply on HN