Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I might be a complete failure (after 8 years of work)

pathdependent.com

Post reply on HN
Print view View original

21–30 of 118 posts

Re: I might be a complete failure (after 8 years of work)

#21

Post by HSO »

jey wrote: post #14

Seems like it'd be a better bet to apply all your skills to creating value (wealth) instead of just trying to push it around a smidge more efficiently. In other words: find a real problem and solve it, increasing global wealth in the process, and collect the economic reward.

Hm, I accidentally upvoted this so I may as well leave a comment. The dichotomy between "creating" wealth and "pushing it around" is false, in my opinion. Lots of people make it and it's intuitive, I know, but figuring out the proper value of things, trying to make forecasts about the future, even profiting from differences that shouldn't be there, all this is part of the capitalist meta-system called financial markets.

One should not confuse the agency issues, borderline-legal information advantages, or even outright fraud in the markets with the basic activity that sustains them. Think about the alternative: Would you like to have to sit with a dimwit credit officer (= bureaucrat) from a big bank who can then effectively veto your plans? Or would you rather have a lively market that decides on such things? Do you want to do your IPO in a public equity market or would you prefer to try to break into an insider club of old rich people who may never come to understand your funny newfangled business model ("what's internet?!").

German sociologist Georg Simmel wrote an interesting book about money, one of the ideas I liked is that its fungibility dissolves the need for relationships, making us free to choose them, and thus effectively increasing personal freedom. The same, I'd venture, is true for capital markets. By spreading transactions around, they increase your options and freedom as entrepreneur. And for markets to work, you need people actually _in_ the market, like it or not.

Figuring out value (as opposed to the going price), identifiying and exploiting inefficiencies, trying to make more intelligent forecasts of likely developments, all these _are_ "real problems" in a market economy. Having intelligent, hard-working people working on them makes the system as a whole more intelligent.

Re: I might be a complete failure (after 8 years of work)

#22

Post by alextp »

drewcrawford wrote: post #3

You can keep adding data points to your spline/polynomial interpolation, but that doesn't mean the model will be any better. In fact, for the purposes of predictive power, it will probably be worse. I'm not extremely well educated on the subject, so somebody correct me, but these predictive models have always struck me as just complicated splines. Add a thousand data points, and the "predictions" for old datapoints l…

Yes, that is the main pitfall with machine learning in general. Usually you do two things to mitigate it (but they don't fix it entirely): regularization and a test set.

Regularization is some means of complexity control. You switch form trying to find the model that has minimal error on your data to finding the model that minimizes "error(model, data) + C*complexity(model)". There are many ways to measure the complexity, and usually the value of C is pretty unimportante (it is empirically found that it only affects the solutions when you change it in orders of magnitude). Keeping a test set is always testing your predictions on held-out data that you didn't use to train the model and that is hopefully independent from your training data.

Still, if you come up with a huge number of models and test them on the same test data you're statistically bound to find a few that seem to work very well, until you try it in different data.

My feeling with most of these market-beating schemes is that just the price history of stocks shouldn't be that predictive of future prices (but aggregating real-world information should). After all, usually what creates spikes and valleys in the prices are some relevant news or decisions.

Re: I might be a complete failure (after 8 years of work)

#23

Post by narrator »

I tried to beat the market using all kinds of AI stuff off and on for a number of years. Learned a lot about AI techniques, and became a better programmer, which helped in my day job, but never got anywhere. It's a fool's game.

Besides, most of these HFT systems hold positions for only a few minutes or less. There's no way someone without a huge account, to cover commissions from trading so much, and access to a lot of expensive real-time data streams could possibly make any headway at it.

Not to say there aren't people who do make money at trading, but they are swing traders and/or people who've figured something out about the market that the market doesn't know yet, and thus has not revealed at all in the price movements of the stock. Since price movements are the primary input into a lot of trading algorithms, most trading algorithms are blind to these developments.

Re: I might be a complete failure (after 8 years of work)

#24

Post by aspiringsensei »

edw519 wrote: post #19

After eight years, I have nothing concrete to show for my efforts. So sad, so true. This is what happens when you play a zero-sum game. Even if you win, someone has to lose. Here's an idea. For the next 8 years, why don't you do something that helps others by making the pie bigger for everyone instead of just trying to game a bigger piece of a smaller pie for yourself. When you do for others, you always have somethin…

This is a very valid way to put this, and it helped me. Thank you.

Re: I might be a complete failure (after 8 years of work)

#25

Post by lars512 »

cjy wrote: post #12
rortian wrote: post #2

An interesting post. I'd be interested hear what kind of background you've got. I think your remark on finance professors is quite wrong. It takes quite a lot of dedication to do a Phd. I'm not great respecter of academic prestige, but its a bit insulting to imply that people fail their way into a professorship. I'd be more than happy to share some pointers on financial papers I've read and such. Just talk about your…

I don't think he was being insulting. He was just explaining that the incentives discourage anyone from publicly disproving the efficient market hypothesis.

This is where it gets fun though. An inefficiency being found amounts to you knowing something about pricing that others don't. You exploit it for profit, and the process of exploiting it removes or reduces the inefficiency. The only way the market is truly screwed is if consistent arbitrage opportunities become available, but the kinds of inefficiencies people find seem more transient than consistent.

On another note, betting markets are (or were) apparently far less efficient than financial markets, in that the true likelihood of a team/horse/camel winning in a competition may be different to the money which punters are willing to bet on it. I know of a guy who made his fortune after many gruelling years by finally coming up with prediction models which worked in these kind of gambling scenarios.

Re: I might be a complete failure (after 8 years of work)

#26

Post by rortian »

chasingsparks wrote: post #7
rortian wrote: post #2

An interesting post. I'd be interested hear what kind of background you've got. I think your remark on finance professors is quite wrong. It takes quite a lot of dedication to do a Phd. I'm not great respecter of academic prestige, but its a bit insulting to imply that people fail their way into a professorship. I'd be more than happy to share some pointers on financial papers I've read and such. Just talk about your…

I didn't mean it in an insulting way. I'm actually starting a Masters/Ph.D. program in computational social sciences in the fall, hoping to fully immerse myself in an environment that also finds markets fascinating. I have derived Black-Scholes; I study market micro-structure; I read as much as I can on anything related to social systems from volatility surfaces in options pricing to more generic machine learning met…

Cool, definitely get back to me. Kinda random but I am very interested in complex networks as well as some finance stuff.

Re: I might be a complete failure (after 8 years of work)

#27

Post by HSO »

kiba wrote: post #10
drewcrawford wrote: post #3

You can keep adding data points to your spline/polynomial interpolation, but that doesn't mean the model will be any better. In fact, for the purposes of predictive power, it will probably be worse. I'm not extremely well educated on the subject, so somebody correct me, but these predictive models have always struck me as just complicated splines. Add a thousand data points, and the "predictions" for old datapoints l…

Taleb's empiricism implies resisting generalization from data and limiting the derivation of general rules from particular observations as one can be missing hidden properties. Thus he believes that scientists, economists, historians, policy makers, businessmen, and financiers are victims of an illusion of pattern. They overestimate the value of rational explanations of past data, and underestimate the prevalence of…

http://en.wikipedia.org/wiki/Apophenia

Re: I might be a complete failure (after 8 years of work)

#28

Post by mbateman »

A lot of people are leaving comments critical of his goal of beating the market as playing a zero-sum game, just pushing wealth around, having no value for society, etc.

This strikes me as dubious and high-handed, to say the least.

First, as to the value of the endeavor. He is trying to, in his words, "come up with an algorithm that finds pockets of profitability in a cloud of probable randomness." If he could really do this, it would create capital flow, the ability to borrow, and so on, for people that could turn it into wealth that would not otherwise have access to that wealth. Financial markets serve a real function, and beating the efficient market hypothesis would improve their ability to fulfill that function.

Second of all, he's extremely interested in the problem. He's obsessed with modeling complex systems and sees this as a sort of holy grail. Surely the fact that he's interested in it and values it counts for a lot, especially if you grant the above point, that there is a value to what he's doing. He has his reasons for being interested in it, and it's a productive endeavor.

I think it's false that his lack of progress, or the reasons that he's frustrated at his lack of progress, have to do with these aspects of his goal. It's more that he's working on an extremely hard (arguably insoluble) problem where it's hard to see incremental gains. This is a problem that ambitious people in many different fields face.

Re: I might be a complete failure (after 8 years of work)

#29

Post by rortian »

fernando wrote: post #15
rortian wrote: post #2

An interesting post. I'd be interested hear what kind of background you've got. I think your remark on finance professors is quite wrong. It takes quite a lot of dedication to do a Phd. I'm not great respecter of academic prestige, but its a bit insulting to imply that people fail their way into a professorship. I'd be more than happy to share some pointers on financial papers I've read and such. Just talk about your…

Could you elaborate on this? In which ways is "a random" walk incorrect? In what is high frequency trading based?

On the last question give me a little time to look up a paper. I'll post it. Essential it shows how the strategy of most long-short hedge funds work. I'd have to look again, but one of the major high frequency strategies uses a version of this.

I'd love to read it, but the easiest way to answer your second question is to point you to 'A non random walk down wall street.'

One huge assumption that Black-Scholes makes is that returns are normal distributed. This is not done for empirical reasons rather mathematical ones, namely it is so convenient.

Edit: Pdf http://www.newyorkfed.org/research/conference/2007/liquidity...

Some meta data and such if you don't feel like going with a pdf:

http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1015987

Re: I might be a complete failure (after 8 years of work)

#30

Post by aspiringsensei »

My mother was down on the floor of the NYSE the other month, and she got the opportunity to corner a trader who she noticed was relying heavily on algorithms.

She asked "What's the difference between a good trader and a good algorithm?" The trader said "nothing.If you're buying 1000 shares, it's probably better to simply put in a market order. A large hedge fund seeking to buy many hundreds of thousands - where the difference between paying 8.30 and 8.31 a share matters - algorithms can actually be helpful."

The issue here is the definition of "trader." For this guy, trader is a person who executes large trades for an investment manager. A trading decision is not the same as an investment decision, and in that vein many trader's motivations are simply to buy a desired quantity of a financial instrument as cheaply as possible.

If you're buying 1000 shares, it's probably better to simply put in a market order. A large hedge fund seeking to buy many hundreds of thousands - where the difference between paying 8.30 and 8.31 a share matters - algorithms can actually be helpful. If you're buying 1000 shares, it's probably better to simply put in a market order. A large hedge fund seeking to buy many hundreds of thousands - where the difference between paying 8.30 and 8.31 a share matters - algorithms can actually be helpful.

Humans are probably better computers for figuring out which stocks to buy and sell to my mind, but there's certainly a place for algorithms if they are not too high-minded.

Post reply on HN