Live data from Hacker News

Numerai – A hedge fund built by a global community of anonymous data scientists

numer.ai

111–120 of 120 posts

Re: Numerai – A hedge fund built by a global community of anonymous data scientists

#111
post #91

Earlier quoted context omitted.

Really smart monkeys. And monkeys which will (hopefully) be self-correcting to converge on the bulls-eye. Most people don't realize that the "markets" are 49% random, 48% sentiment driven, and 3% fundamentals. If you approach the problem with that assumption held true, monkeys throwing darts isn't such a horrible mechanism for investing. See: "Monkeys Are Better Stockpickers Than You'd Think: Why dart-throwing primat…

> Most people don't realize that the "markets" are 49% random [...] If you approach the problem with that assumption held true, monkeys throwing darts isn't such a horrible mechanism for investing. Maybe I haven't understood that part of data science but I never got why throwing more unpredictability on an already unpredictable data source would somehow make it more predictable.

It doesn't make it "more predictable". You're simple achieving better fit to the sample data distribution.

I guess another way of saying it is that your mess is starting to look more like their mess.

Anything that deals with the future is inherently non-predictable. Using chaos theory as a framework, we say it is unpredictable because we are unable (and will always be unable) to model the currently system completely. There will always be data that was not captured hiding between the data that was captured. Follow the arrow-of-time far enough out into the future and that non-captured data will manifest itself in the captured data, thereby (usually) creating a deviation from the modeled future.

To get around this we use statistics and probability. We rely on the law of large numbers and regression to the mean. In other words, we hope that the future won't get too weird and will be similar enough to the past, within some confidence interval.

So, we're not really predicting a specific outcome, we're predicting that the outcome will be some point within some confidence interval.

The reason we can get better at this, is as we capture more data we can better guess the inputs and assumptions we use to create the model. We throw out stuff that didn't happen to be relevant. We discover stuff that we should have considered relevant. If we're lucky the model closely follows the physical laws of our reality and we can apply the frameworks so arduously worked out by chemists, physicists, biologists, etc. If we're not so lucky we're dealing with sentiment, conjecture, or any of the other human inputs of the financial markets, and we are forced to make up formulas that work until they fail spectacularly.

Re: Numerai – A hedge fund built by a global community of anonymous data scientists

#112
post #108

Earlier quoted context omitted.

I read it mainly as a criticism of this site and this application of deep learning. Predicting stock prices slightly better doesn't really make the world a better place or improve anyone's lives. But it does consume the time and resources of very intelligent people who might be able to do incredible things elsewhere. Particularly the problem of this website, is it's huge focus on anonymity. You don't need to expose y…

> Predicting stock prices slightly better doesn't really make the world a better place or improve anyone's lives. Actually, it does. Increasing liquidity, communicating price information, more accurately accounting for predictable future changes in price, and absorbing risk allow for increased agricultural and industrial production. Like I said, you have to think through a few layers of abstraction. A nice rule of th…

good grief you're right, I never realised, the market is always right about everything!

Re: Numerai – A hedge fund built by a global community of anonymous data scientists

#113
post #37

Their dataset reeks of startup hustling. I just downloaded the training set [1] and plotted some of its descriptive statistics [2]. It looks that all features are uniform distributions and the response variable is Bernoulli coin-flipping. In layman's terms, you can't really come up with a good predictive model with this training set. I give them the benefit of the doubt that they wanted to have something in place to…

>to push the website live __ The website has been live since like December 2015. There's a new tournament weekly. (I think it used to be every second week)... New tournament, new data -- always encrypted. Just look at the leaderboard; look at some of the participants' submission history

>any serious data scientist __ - I mean, any human - data scientist or not - would see at least that the website wasn't just pushed live; or at least have noticed the film is from like August... ^are you a serious data scientist? If you are (which I assume you are - just also an eager beaver to download the datasets without learning about the company sufficiently) -- then you'd do well to join the tournament and have a real go before you [attempt] to knock it as you have here -- and then come back and give some real feedback -- I'm sure everyone would appreciate to hear the 'after' report from you -- since you're so kind as already to dish out benefits of doubt :) Looking forward to catching you above controlling capital!

Re: Numerai – A hedge fund built by a global community of anonymous data scientists

#114
post #108

Earlier quoted context omitted.

I read it mainly as a criticism of this site and this application of deep learning. Predicting stock prices slightly better doesn't really make the world a better place or improve anyone's lives. But it does consume the time and resources of very intelligent people who might be able to do incredible things elsewhere. Particularly the problem of this website, is it's huge focus on anonymity. You don't need to expose y…

> Predicting stock prices slightly better doesn't really make the world a better place or improve anyone's lives. Actually, it does. Increasing liquidity, communicating price information, more accurately accounting for predictable future changes in price, and absorbing risk allow for increased agricultural and industrial production. Like I said, you have to think through a few layers of abstraction. A nice rule of th…

I don't believe that increasing liquidity produces much, if any, value for the world. The stock market itself is pretty disconnected from any real value, and at best any benefits just benefit rich investors buying stocks. It certainly doesn't increase production.

But the opportunity cost is what I'm getting at. So many very intelligent people could be producing immense value in other areas, but instead drained away doing this garbage. And they are incentivized to keep anything useful they invent a secret.

Re: Numerai – A hedge fund built by a global community of anonymous data scientists

#115
post #108

Earlier quoted context omitted.

> Predicting stock prices slightly better doesn't really make the world a better place or improve anyone's lives. Actually, it does. Increasing liquidity, communicating price information, more accurately accounting for predictable future changes in price, and absorbing risk allow for increased agricultural and industrial production. Like I said, you have to think through a few layers of abstraction. A nice rule of th…

I don't believe that increasing liquidity produces much, if any, value for the world. The stock market itself is pretty disconnected from any real value, and at best any benefits just benefit rich investors buying stocks. It certainly doesn't increase production. But the opportunity cost is what I'm getting at. So many very intelligent people could be producing immense value in other areas, but instead drained away d…

Wrong, wrong, wrong. Avoid speculating on such things when you clearly have no background in them. You will accomplish nothing but spreading misinformation.

http://www.investmentreview.com/files/2010/07/The-value-of-l...

https://fp7.portals.mbs.ac.uk/Portals/59/docs/Finger,%20Mark...

http://people.stern.nyu.edu/adamodar/pdfiles/papers/liquidit...

https://www.macquarie.com.au/dafiles/Internet/mgl/au/mfg/mim...

https://en.m.wikipedia.org/wiki/Liquidity_crisis

The stock market is absolutely connected to "real value" (which doesn't make sense as a concept in the way you're trying to use it). Besides allowing companies to raise capital for future endeavors, it also allows the market to communicate price information, which is critical for reducing the risk associated with a given transaction.

> at best any benefits just benefit rich investors buying stocks.

It benefits everyone; investors risk their capital by selling it to companies in exchange for partial ownership, which allows the company to grow, and if the investment pays off the person who risked their money is rewarded. Without the stock market, companies would have to get loans for growth, which is risky and too expensive to be practical.

> It certainly doesn't increase production.

Yes, it very much does, for the reasons I outlined above. This is like Econ 101; why are you commenting so strongly when you clearly do not have any background here?

> but instead drained away doing this garbage.

A small efficiency increase expressed over 100 trillion dollars of trades every year is a huge productivity increase for society.

Re: Numerai – A hedge fund built by a global community of anonymous data scientists

#116
post #108

Earlier quoted context omitted.

> Predicting stock prices slightly better doesn't really make the world a better place or improve anyone's lives. Actually, it does. Increasing liquidity, communicating price information, more accurately accounting for predictable future changes in price, and absorbing risk allow for increased agricultural and industrial production. Like I said, you have to think through a few layers of abstraction. A nice rule of th…

good grief you're right, I never realised, the market is always right about everything!

Nice straw man, but the market doesn't have to be right about everything; it just has to not be completely wrong about a market worth tens of trillions.

The stock market is one of the mechanisms that allows the market to be right by communicating price information.

Re: Numerai – A hedge fund built by a global community of anonymous data scientists

#118
post #115

Earlier quoted context omitted.

I don't believe that increasing liquidity produces much, if any, value for the world. The stock market itself is pretty disconnected from any real value, and at best any benefits just benefit rich investors buying stocks. It certainly doesn't increase production. But the opportunity cost is what I'm getting at. So many very intelligent people could be producing immense value in other areas, but instead drained away d…

Wrong, wrong, wrong. Avoid speculating on such things when you clearly have no background in them. You will accomplish nothing but spreading misinformation. http://www.investmentreview.com/files/2010/07/The-value-of-l... https://fp7.portals.mbs.ac.uk/Portals/59/docs/Finger,%20Mark... http://people.stern.nyu.edu/adamodar/pdfiles/papers/liquidit... https://www.macquarie.com.au/dafiles/Internet/mgl/au/mfg/mim... https:/…

I think you are the one spreading misinformation. I'm well aware of the arguments and find them extremely unconvincing.

Look, for instance, at high frequency trading. Traders spend millions of dollars trying to get signals across the earth a millisecond faster than the competitors. Does it actually benefit anyone that information travels a millisecond faster? Who benefits from the high frequency traders at all? Yet they eat up millions, maybe even billions, of the economy's resources doing this nonsense.

Look at these trader spending millions of dollars to fly drones over oil tankers or parking lots, to get slightly more accurate information faster than anyone else. Look at these hedge funds spending tons of money on bribes to people in companies to get insider information before the public does. These things clearly provide zero benefit to the world, it's just a massive waste of resources.

This hedge fund is a bit different, in that they seem to be actually doing fundamentals and longer term bets. That's not so bad, but it's still pretty disconnected from any real world benefits. At best they predict a company will increase in value and buy it's stock. How does that benefit anyone? It's almost a zero sum game. The people that buy the stock that week lose, because the price is higher than it otherwise would have been. The people that sold the stock lose, because it was really worth more than they actually sold it for. Any money the hedge fund makes necessarily comes from someone else losing that money.

The stock market itself is not really connected to the real world. Sure sometimes companies sell stock to raise capital. But most stocks being sold and bought are not by the companies themselves. It's a huge indirect chain of traders and investors and speculators, trading these stocks many, many years after the company sold them. And the company itself would still be able to sell stocks if there were fewer traders. At worst the prices would be slightly less accurate.

Price information of stocks is also a horribly inefficient way of getting this information. These companies are often involved in many separate projects. So even if someone has a model that can exactly predict the success of products, it's very difficult to use that information. You also need to determine how much the company is worth given all of it's projects and businesses and property it owns. It doesn't directly give that information to the company itself - they see their stock went down, but they may have no idea why. Not to mention Keynesian beauty contest problems, where you have to predict not what the company is worth, but what other people think it will be worth, and how much they know currently, and what the interest rate is, etc.

But you didn't address my main concern, which is the bad incentives of secrecy, and the opportunity cost of employing smart people doing this, instead of doing something else. Even if it does produce some value, it doesn't mean it's worth the cost.

Re: Numerai – A hedge fund built by a global community of anonymous data scientists

#119

Earlier quoted context omitted.

I read it mainly as a criticism of this site and this application of deep learning. Predicting stock prices slightly better doesn't really make the world a better place or improve anyone's lives. But it does consume the time and resources of very intelligent people who might be able to do incredible things elsewhere. Particularly the problem of this website, is it's huge focus on anonymity. You don't need to expose y…

Don't you think that is says something (not terribly good) about society that these very intelligent people must spend their time this way if they want to be rewarded? Why spend 80 hours a week chasing grant money when you can spend 60 moving numbers around?

No. First off, "reward" is not just about the money[0]. The job presented on this site, on the other hand, due to the encryption, is motivated purely and entirely by monetary reward. Everything else is shielded.

In fact I'd say this job is quite uniquely, extremely, only about the money. Even when compared to other kinds sweet/lucrative, perhaps "stupid lucky" jobs you may find, that may require 60 hours mindlessly pushing papers, yet paid extremely well. I'd be hard-pressed to come up with any hypothetical kind of job could be more sharply, singularly focussed on the "do extremely-intelligent-monkey-dance, receive ample bio-survival-tickets" to the exclusion of any other meaning or fulfilment than this Encrypted Machine Lottery/Learning/Sudoku.

Second, I could be wrong (or too optimistic) but I'd hope that most "very intelligent" people demand more nourishing reward than just money to regularly put in more than 40h/wk. You can do this for research and advancement of science, or maybe because one considers the work itself a net positive to society, or maybe you chose to sacrifice those hours of your life to support loved ones, maybe there's some deep personal reward in it, or maybe it's temporary and you're saving for something rewarding. But to pretend just money is enough to waste your life on, hey it's a choice, but I'm going to have to see you turn in your "very intelligent"-card.

On the other hand, there will be exceptions. Some people don't care, just want to maximize $$$ for the least amount of effort/hours of one's life, regardless of side-effects. This website does not fill a niche for these people. If you are extremely intelligent, just want money and don't care about moral aspects, there's always been plenty "business opportunities" to fulfil these particular needs.

Finally, if one would argue, but this job isn't quite explicitly "badwrong" like those others. I'd suggest to think it through: On the one hand, a (possibly) well-paying job that is designed from the bottom up to have no way of determining whether it has a net positive or negative external effect besides the money it pays you. On the other hand, literally anything else you could do with your time.

I can see people trying this for a short time, as a funny puzzle, at most.

[0] I don't mean this one, but because I still think it's funny, I'll point out the way-too-easy retort here: this says something (not terribly good) about you. (j/k)

Re: Numerai – A hedge fund built by a global community of anonymous data scientists

#120
post #115

Earlier quoted context omitted.

Wrong, wrong, wrong. Avoid speculating on such things when you clearly have no background in them. You will accomplish nothing but spreading misinformation. http://www.investmentreview.com/files/2010/07/The-value-of-l... https://fp7.portals.mbs.ac.uk/Portals/59/docs/Finger,%20Mark... http://people.stern.nyu.edu/adamodar/pdfiles/papers/liquidit... https://www.macquarie.com.au/dafiles/Internet/mgl/au/mfg/mim... https:/…

I think you are the one spreading misinformation. I'm well aware of the arguments and find them extremely unconvincing. Look, for instance, at high frequency trading. Traders spend millions of dollars trying to get signals across the earth a millisecond faster than the competitors. Does it actually benefit anyone that information travels a millisecond faster? Who benefits from the high frequency traders at all? Yet t…

It's also plausible that they are intelligent and hard working because of their willingness to earn money.
Post reply on HN