Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How I made $500k with machine learning and high frequency trading

jspauld.com

Post reply on HN
Print view View original

231–240 of 320 posts

Re: How I made $500k with machine learning and high frequency trading

#231

Post by zygomega »

jspaulding wrote: post #81
yajoe wrote: post #43

Earlier quoted context omitted.

This. I love crazy projects and Show HN's until the cows come home, but this one is dangerous that I must repeat the warning to others. I cannot emphasize how important it is to understand that people who trade using price action ( http://en.wikipedia.org/wiki/Price_action_trading ) are just speculating based on where they expect the price to move. It's no different than people who play Texas Hold'em online and specu…

Hey, I didn't actually intend this to be a course. I do not make any money in the market right now so am certainly not qualified to teach a course on it. And of course, if I was making money in the market I wouldn't have posted this at all. So please everyone remember that. These comments have made me realize it's probably for the best if I do not post the source code. Basically you are competing against armies of PH…

I think you undersell yourself - kudos to your success. I'd back a hacker with a plan (and a cash flow crises perhaps) over an army of PhDs any day! Maybe the course you should think about teaching is how to how to orgainse such a high-quality hack as you've described in the article :)

I'm currently building a semi-high frequency trading solution and the problem I run into is the sheer breadth of expertise you need to get it all happening. Modern chip design, low-latency, lock-free concurrent messaging, fault-tolerant system design, adaptive learning algorithms, k-means clustering and broker APIs are just a smattering of the ideas I'm trying to get across to make progress. For me, algorithm creation comes more easily than reading about and implementing a broker interface.

There is certainly armies of PhDs out there backed by big money but they exist behind heavily guarded intellectual property walls. An open source HFT/Algo/Automated trading platform that brings a hacker sensibility to this problem domain would be seriously competitive.

Re: How I made $500k with machine learning and high frequency trading

#232

Post by gubatron »

of course no firm would respond to his noob low yield model.

The level of the coders doing HFT is beyond the comprehension of this guy, added to the team of Mathematicians, Physicists and computer scientists at your average HFT firm, they probably laugh when they read this.

Good try though, it was awesome that eventually he tuned it to profitability, but there's no way in hell they'd buy that amateur software/algorithm.

Kudos though for taking on the task of learning how to code and making money with ML.

Re: How I made $500k with machine learning and high frequency trading

#234

Post by mcarvin »

jspaulding wrote: post #121
bjourne wrote: post #108

Earlier quoted context omitted.

Guess you are familiar with http://en.wikipedia.org/wiki/Survivorship_bias ? In 2009, there were probably tonnes of people trying to exploit the market using similar low-tech methods as you. Even if all of them were at best break-even, some of them likely made a lot of money on their unprofitable algorithms by pure chance thanks to the size of the cohort. Those few blogged about it and those who lost money didn't. :)…

It's a great point and seems like a very smart thing to keep in mind. I think in my case, based on the statistics involved, the odds that my success was luck just seems astronomically small. But, guess I'm biased in my own way :)

Please help me understand this better?

With a deep understanding of markets and trading I fail to see why you see 'luck' as an explanatory variable is inversely correlated with the frequency of your trades (notwithstanding the effect of trading expenses)?

From what I have gleaned the following seems to be true: 1. Your algorithms worked (made money) 2. Then your algorithms did not work, but you could not figure out why

If you do not know why something stopped working it seems unlikely that you had a full understanding of why it was working in the first place. Without understanding the nature of the predictive value of the algorithm while it was working, its success seems to be good fortune.

Your algorithm could have shown a systematic correlation to any number of factors that could have created strong performance over several months. Performance would then be attributed to accidentally 'timing' a favorable market.

I know you feel differently, what am I missing?

And either way - kudos on the $500k.

Re: How I made $500k with machine learning and high frequency trading

#236

Post by ScottBurson »

tarr11 wrote: post #23

This is just glorified gambling. I am not sure what special insight or advantage he had, other than his own model. Every trader has a model. It could have easily been called "how i lost 500k with machine learning". Like gambling, it's easy to manipulate statistics to show that you did well in some period of time. I worked for a large investment bank about 10 years ago, writing trading programs for quant traders who w…

There's a sentence in this article that is critical and yet very easy to overlook: the author had 2 years experience daytrading manually. That already gave him a lot of knowledge of how the markets work and where an edge might be found.

I think that if someone is a good programmer and has some mathematical chops and has that kind of experience daytrading, taking a shot at automated trading is probably a reasonable thing for them to do. Without all of that background, you're right, they're almost certain to lose money.

Re: How I made $500k with machine learning and high frequency trading

#237

Post by halcyondaze »

martinced wrote: post #60
tarr11 wrote: post #23

This is just glorified gambling. I am not sure what special insight or advantage he had, other than his own model. Every trader has a model. It could have easily been called "how i lost 500k with machine learning". Like gambling, it's easy to manipulate statistics to show that you did well in some period of time. I worked for a large investment bank about 10 years ago, writing trading programs for quant traders who w…

It could have easily been called "how i lost 500k with machine learning". If you've really worked in that field than it's very surprising you've never heard about what professional poker players call bankroll management (and they "stole" the concept from professional traders). The whole point is that you can --either if you gain an edge or get lucky-- win big. Very big. But you're never exposing a large part of your…

Yes this is the point I was going to write myself. I played online poker for four years and won over $100,000. I wasn't very good compared to the top 5% of players at my stakes, but I was much better at bankroll management, tilt control, and all of the other soft skills. The way I structured my bankroll made it actually impossible to go broke as well.

Re: How I made $500k with machine learning and high frequency trading

#238

Post by dmmalam »

beagle3 wrote: post #219
dmmalam wrote: post #206

Being pedantic, 4000 trades a day isn't HFT. This is stil algo trading, of which HFT is a subset. I consider HFT to be any strategy where speed itself is the what gives the edge. Colocation is usually a prerequisite, though not sufficient. It's a shame HFT gets all the attention, when it's really a tiny portion of trading activity. Algo-trading in general is 70%+ of market activity in the US. Also limiting trades isn…

In the US, HFT is mostly synonymous with "all out tech war, flooding the order queue so your less-equipped peers get lags". (Nanex publishes analysis on these events, which are not occuring several times a month and keep accelerating). In Europe, HFT is mostly what OP describes, because they have reasonable control (e.g., you have to have one execution per 10 orders or pay a fine; in US exchanges, you can sometime fi…

When I say limiting trades, I mean naively saying 'I will have at most x positions outstanding'. Each trade has an associated risk (variance), that interacts in complicated ways in a portfolio, which I'm sure you know.

silly example, 100 small positions could be less risky than 1 large position or, 1 long, and 1 short trade will cancel each other out and create a riskless portfolio (with 0 return).

You need to have a risk budget, account for each trade, and work out the risk for the composite portfolio. Obviously this is not fool proof, but it's a way better approximation of the real world.

Re: How I made $500k with machine learning and high frequency trading

#239

Post by nashequilibrium »

Pls do not follow the advice of the OP. I started a hedgefund in 2004 doing HF platform arbitrage and ran it for 5yrs and i can honestly tell you that this is just survivorship bias. This is a very complex field and being off slightly, having a slight bias, a fraction of a point off your execution pricing and a slightly flawed money management system is recipe for disaster.

The biggest issue is the confusion that you can apply machine learning to HF trading. HF trading sub 15min mark is more about playing the deal flow, and only the institutions have an edge on this. This is why goldman had to separate the buy and sell sides in the early 2000's. Above 15mins you are able to find an edge using time series analyses since the market is scaling invariant according to Benoit Mandelbrot and this does not apply to dealflow. Also having access to dealflow allows you to predict volatilty seconds ahead which allows you decrease your risk and increase you reward as well as handle your costs since the volatility will impact your transaction costs even if transaction costs themselves stay the same. There is just so much stuff to cover that a comment will not do justice in explaining what is wrong with this guys logic.

Re: How I made $500k with machine learning and high frequency trading

#240

Post by ishaanc90 »

tarr11 wrote: post #23

This is just glorified gambling. I am not sure what special insight or advantage he had, other than his own model. Every trader has a model. It could have easily been called "how i lost 500k with machine learning". Like gambling, it's easy to manipulate statistics to show that you did well in some period of time. I worked for a large investment bank about 10 years ago, writing trading programs for quant traders who w…

I don't get you haters. sure the title is a bit misleading because he never really discusses what his alphas were. But its a pretty good high level description of the architecture of a hft system. I was a quant at GS and these are not the retail investors you pick off. You have your own set of alphas and most of them are meant to pick on mom and pops clicking away at home. This guy didn't reveal his strategy but nevertheless the graph shows his strategy had a significant edge. The lifetime of a strategy also looks like that. It is another thing that his title for the post is kind of off.
Post reply on HN