I might be a complete failure (after 8 years of work) • Hacker News Board

Re: I might be a complete failure (after 8 years of work)

#101

Post by eru » Mon, Jun 28, 2010, 11:31 AM UTC

Natsu wrote: post #100

VictorHo wrote: post #87

Earlier quoted context omitted.

Calling trading a zero-sum game shows a fundamental lack of understanding of finance. We may have very negative views on the industry, but it exists because it serves many fundamental (and necessary) functions, including: * Accuracy in pricing shares (and it is important to know the actual value of a company for many reasons) * Enable market participation. In many illiquid OTC markets, it is all but impossible for an…

> * Accuracy in pricing shares (and it is important to know the actual value of a company for many reasons) Define "accuracy"? Preferably in such a way that helps me reconcile it with events like the flash crash. One of my biggest problems in understanding the financial markets is how you separate the signal (events reflecting actual change in company value) from the noise (speculation, flash crashes, etc.). While I…

It's more like, everybody who has a view on the value of a security can put some money where his mouth is. The price will move accordingly, until nobody who has a different opinion has any money left to put behind it.

If smaller differences between current price and opinions can lead to action--because of lower transaction costs--the market is better.

Re: I might be a complete failure (after 8 years of work)

#102

Post by retube » Mon, Jun 28, 2010, 12:54 PM UTC

His approach to this - the goal of beating the market - has been totally wrong. It sounds as if he's been at this completely solo - he mentions in the comments that he worked alone as he didn't want to share his algorithms.

He would have benefitted enormously from working in an institution that specialises in this kind of work (hedge funds etc). Being largely self-taught and having worked alone since he was 17 pretty much guarantees he doesn't know anything institutional trading firms won't know or have studied - employing as they do advanced mathematicians, physicists etc to work at the cutting edge of algo research and trading.

Not only has he missed out academically, he's missed out operationally. "Playing-the-game" is as big a part - if not bigger part - of trading than pricing/valuation. With the resources of a large institution you can learn to play the market in a way you can't possibly by yourself. This is in fact largely how they make money - consistently, year on year.

Thirdly - even if he did discover a pricing inconsistency or whatever, it's doubtful he'd be able to leverage it in as profitable a way that he would be able to as a firm.

Re: I might be a complete failure (after 8 years of work)

#103

Post by chegra » Mon, Jun 28, 2010, 12:56 PM UTC

"He[Dan Zanger] holds the unofficial record in trading stocks by turning $11,000 into more than $18 million in 18 months in 1999-2000. He grew that to an incredible $42 million in less than two years and has the tax receipts to prove it." http://www.investopedia.com/articles/trading/04/082504.asp

"Williams won the 1987 World Cup Championship of Futures Trading from the Robbins Trading Company were he turned $10,000 to over $1,100,000 (11,376%) in a 12 month competition with real money. Ten Years later his daughter Michelle won the same contest" http://en.wikipedia.org/wiki/Larry_R._Williams

Somebody has some sort of algorithm.

Re: I might be a complete failure (after 8 years of work)

#104

Post by khafra » Mon, Jun 28, 2010, 3:16 PM UTC

chasingsparks wrote: post #58

gvb wrote: post #47

Earlier quoted context omitted.

Alchemy and transmutation of lead to gold is a pretty apt analogy for John Nelson's quest to turn his model of the market into gold. a) The alchemical belief in transmutation was based on a thoroughly wrong understanding of the underlying processes. John's (and other's) lack of success indicates a faulty model. b) Ironically, it transpired that, under true nuclear transmutation, it is far easier to turn gold into lea…

If you look at the first sentence of the article, it is linked to the subheading of a blog post that explained my intellectual evolution -- the subtitle of which was "The Contemporary Philosopher’s Stone" which is how I refer to this problem. ( http://en.wikipedia.org/wiki/Philosophers_stone ) You're not the only one drawing the comparison ;)

Sounds like in that frame, most of the consoling messages here are along the lines of "our gold is not the common gold."

Re: I might be a complete failure (after 8 years of work)

#105

Post by pradocchia » Mon, Jun 28, 2010, 4:22 PM UTC

10ren wrote: post #69

Question: has anyone managed to do this? (eg. do hedge funds do this, or do they combine algorithms with their own ideas and quasi-insider knowledge, plus massive amounts of timely data - inaccessible to an individual). My old supervisor (ACM fellow) comforted my failure to create AI by saying that lots of other clever people hadn't done it either. I have a feeling that you are attempting something more difficult tha…

Simply put, if anyone has managed to do it, you aren't likely to hear about it. That said, I know of two:

First was a math prof that would play the markets every couple years when he needed a little extra cash.

Second was someone who had isolated patterns of market intervention and figured out how to profit from them. But that's more like insider trading than beating the market algorithmically.

Re: I might be a complete failure (after 8 years of work)

#106

Post by alextp » Mon, Jun 28, 2010, 4:23 PM UTC

Robin_Message wrote: post #98

alextp wrote: post #22

Earlier quoted context omitted.

Yes, that is the main pitfall with machine learning in general. Usually you do two things to mitigate it (but they don't fix it entirely): regularization and a test set. Regularization is some means of complexity control. You switch form trying to find the model that has minimal error on your data to finding the model that minimizes "error(model, data) + C*complexity(model)". There are many ways to measure the comple…

Isn't it better to adopt a Baynesian approach and directly model the complexity of the model rather than some arbitrary C complexity? To wit, P(model|data)=P(data|model)P(model) / P(data) , which is to say that you calculate the probability a model is true given the data you see based on the probability you see the data you see given a certain model, multiplied by the probability of the model you are using. I guess t…

Yes, but the formulation I showed you is equivalent to a bayesian prior. For example, if you want to learn a weight vector w that gives high likelihood to the data and has a gaussian prior with 0 mean and Cidentity covariance, the MAP answer is "minimize -log(likelihood) + C||w||", where ||w|| is the square norm of w. Equivalently, if the prior is a laplacian you just change the norm from the l2 to the l1 norm.

Being bayesian gives you an extra capability that is model averaging, and this does usually improve the behavior at a high computational cost.

I really like bayesian models, and right now I'm experimenting with one that should do unsupervised sentiment analysis without a priori knowledge of word polarity or things like that (yes, I'm a phd student in machine learning).

Re: I might be a complete failure (after 8 years of work)

#107

Post by Robin_Message » Mon, Jun 28, 2010, 4:45 PM UTC

alextp wrote: post #106

Robin_Message wrote: post #98

Earlier quoted context omitted.

Isn't it better to adopt a Baynesian approach and directly model the complexity of the model rather than some arbitrary C complexity? To wit, P(model|data)=P(data|model)P(model) / P(data) , which is to say that you calculate the probability a model is true given the data you see based on the probability you see the data you see given a certain model, multiplied by the probability of the model you are using. I guess t…

Yes, but the formulation I showed you is equivalent to a bayesian prior. For example, if you want to learn a weight vector w that gives high likelihood to the data and has a gaussian prior with 0 mean and C identity covariance, the MAP answer is "minimize -log(likelihood) + C ||w||", where ||w|| is the square norm of w. Equivalently, if the prior is a laplacian you just change the norm from the l2 to the l1 norm. Bei…

I do remember reading that the Bayesian approach leads to previous empirical formulas falling out. Is that the case here, or was that formula derived using Bayes? I'm a PhD student in something else, and I'm trying to do some machine learning. So, what should I read to make what you said make sense :)?

Re: I might be a complete failure (after 8 years of work)

#108

Post by dunk010 » Mon, Jun 28, 2010, 4:57 PM UTC

Well, he might just not be smart enough. Large financial institutions and many smaller hedge funds reap countless millions from the markets and much of this comes from proprietary trading (well, all of it in the case of hedge funds.) There's a massive brain-drain into financial institutions of many of our best and brightest, all in search of making a fortune by the time they're 30. So on the one hand you have top students from top institutions applying their considerable intellect and training into applying advanced mathematical methods against an extremely complex problem. On the other you have a guy who tinkers around himself hoping after hope that the amateur (for that is by definition what he is) attempts that he makes at beating the market are going to some day work out and make him his fortune.

Unfortunately the odds are stacked against you, and those are compounded by fact that you have little or no opportunity to employ any serious leverage.

"I would have probably, or at least possibly, been wealthy by other means by now". At _best_ "possibly", I think. At 25 that is an incredibly arrogant statement to make.

Re: I might be a complete failure (after 8 years of work)

#109

Post by chc » Mon, Jun 28, 2010, 5:06 PM UTC

wooster wrote: post #91

chc wrote: post #56

Earlier quoted context omitted.

I think you don't disagree. The grocer is not in a seller in the same ecosystem as the farmer. The grocer is one of the external people who benefits from the zero-sum game the farmers are playing.

Farming is pretty much the opposite of a zero sum game.

The growing part isn't, but the selling part can be seen that way in the right conditions. I agree, though, it isn't the example I would have chosen.

Re: I might be a complete failure (after 8 years of work)

#110

Post by rbanffy » Mon, Jun 28, 2010, 5:18 PM UTC

kiba wrote: post #10

drewcrawford wrote: post #3

You can keep adding data points to your spline/polynomial interpolation, but that doesn't mean the model will be any better. In fact, for the purposes of predictive power, it will probably be worse. I'm not extremely well educated on the subject, so somebody correct me, but these predictive models have always struck me as just complicated splines. Add a thousand data points, and the "predictions" for old datapoints l…

Taleb's empiricism implies resisting generalization from data and limiting the derivation of general rules from particular observations as one can be missing hidden properties. Thus he believes that scientists, economists, historians, policy makers, businessmen, and financiers are victims of an illusion of pattern. They overestimate the value of rational explanations of past data, and underestimate the prevalence of…

> People project patterns onto data when in fact, they don't exists.

So, the patterns we see in the Ulam spiral can be completely random?

We observe patterns. Human brains are really good in finding them. The problem is finding patterns in past data may or may not provide useful insights into future data.

With the resources being dedicated to this problem, I find it highly unlikely I could be able to offer any helpful advice that would prove more helpful than what already exists.

It would be hard enough to predict the market if the only factors influencing it were signals coming out of the companies we trade. Predicting what other algos may me doing in the space we are trying to examine and the influence their trades exert in its future is... Well... I won't try.