Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Lessons learned building an ML trading system

tradientblog.com

Post reply on HN
Print view View original

111–120 of 160 posts

Re: Lessons learned building an ML trading system

#111

Post by __d »

traK6Dcm wrote: post #107
__d wrote: post #97

Earlier quoted context omitted.

In a little more detail ... To be competitive in US equities HFT, you need an FGPA with 40GbE ports hosted in a server (which needs to power and cool the FPGA, and deal with the less latency-sensitive bits of your system). You'll need some storage as well. That server needs to be co-located with your target exchange(s) matching engines, and connected via 40GbE. You might additionally want remote market data via mm-wa…

Thanks a lot for the detail. I've always wondered what the actual costs look like.

Thanks for posting Nick's article!

I chased up some actual details from NASDAQ as an example:

http://www.nasdaqtrader.com/Trader.aspx?id=PriceListTrading2

Re: Lessons learned building an ML trading system

#112

Post by alexcnwy »

nickreese wrote: post #55
alexcnwy wrote: post #52

Earlier quoted context omitted.

Fascinating story, thanks for sharing! "we realized that there will always been a better resourced or more dedicated team willing to fight you for your alpha" - that's part of what makes financial markets such a fun and interesting challenge but agreed, intermediary risk at the timescales you were operating at in this kind of unregulated market is real and not fun I'd love to hear what problem you moved to that you b…

My partner and I had built broadbandnow.com and were looking for a new interesting problem to solve to get us fire up again about fun tech problems. This just scratched the itch. The team was running day to day of the other business while we worked on the HFT bot. Shortly after we realized we either should exit or hire an exec team to run the business. After a few months of executive search we found an offer we liked…

Very interesting. Your nursing home project sounds like you could definitely add loads of value - best of luck for it!

Re: Lessons learned building an ML trading system

#113

Post by alexcnwy »

pinouchon wrote: post #84

I spent the last year working fulltime on a system similar to the one described here. I trade the top ~20 cryptos on binance. I use deep learning models (combination of temporal, causal convnets and RNNs) with heavy data augmentation. I built my own tooling for data collection, training, backtesting and live deployment. Having a data engineer background coming into this was hugely helpful: most of my time was spent m…

I'd love to hear more about what kind of data augmentation you're doing. A friend of mine recently got a GAN to work for timeseries which is really interesting.

I've done a lot of work in the space and would love to chat - just emailed you :)

Re: Lessons learned building an ML trading system

#114

Post by kortilla »

akra wrote: post #80
traK6Dcm wrote: post #68

Earlier quoted context omitted.

Many HFT systems provide liquidity. If there is no liquidity, retail investors like you or me cannot buy or sell. Just imagine you want to exchange a currency because you go traveling and the exchange tells you "Sorry, nothing available right now, gotta come back in a few weeks". That's what would happen if there is no liquidity.

I always had questions about the liquidity on offer though seeing participants flee the market in black swan events. HFT liquidity could be illusionary - its only there when its not quite required similar to how the bank "only offers you a loan when you don't need one". Of course this is exactly the point when liquidity is required; normally there's sufficient liquidity in normal times from market participants. Its e…

> normally there's sufficient liquidity in normal times from market participants.

Not really. The spreads were terrible before HFT market making.

Re: Lessons learned building an ML trading system

#115

Post by brobinson »

perl4ever wrote: post #88
nickles wrote: post #50

Earlier quoted context omitted.

> But at some point he was confused why many of his buy/sell orders weren't being executed despite being open for several minutes (hours?) his algo would make bids far away from the current spread, anticipating movement * Did his levels actually get crossed? It could be that he never had the opportunity to get filled at the price he hoped for. * What was his queue position? Stock exchanges tend to be price time order…

It's a fact that Joe Blow will get sub-penny price improvement at random. I have no idea how often or how long it's been a thing, but it happens.

I've been getting fills out to four decimal places for a few years now. (Schwab and Interactive Brokers' smart routing)

Re: Lessons learned building an ML trading system

#116

Post by ghgr »

alexcnwy wrote: post #113
pinouchon wrote: post #84

I spent the last year working fulltime on a system similar to the one described here. I trade the top ~20 cryptos on binance. I use deep learning models (combination of temporal, causal convnets and RNNs) with heavy data augmentation. I built my own tooling for data collection, training, backtesting and live deployment. Having a data engineer background coming into this was hugely helpful: most of my time was spent m…

I'd love to hear more about what kind of data augmentation you're doing. A friend of mine recently got a GAN to work for timeseries which is really interesting. I've done a lot of work in the space and would love to chat - just emailed you :)

Indeed GANs are showing very promising results. There's a series of blog posts from Fernando de Meer which discuss this topic in a very approachable way. https://quantdare.com/generating-financial-series-with-gener...

Re: Lessons learned building an ML trading system

#117

Post by ACow_Adonis »

traK6Dcm wrote: post #95
fny wrote: post #78

Can someone comment on how taxes are handled when automated trades are made like this? It's something that seems wholly absent from the cost calculations.

Author here. Personally, I just don't. I tried doing it but it was too complicated. So I end up just hiring a tax accountant specializing in crypto, send them all the data I have, and pay a few $k. In case something goes wrong, it's their fault and they take the risk.

Not sure what country you're in, but that's not the way tax accountants work in mine. You're still liable for mistakes/problems they make here :(

Re: Lessons learned building an ML trading system

#118

Post by roenxi »

Akababa wrote: post #105
roenxi wrote: post #81

Earlier quoted context omitted.

Hang on, that is conflating separate ideas. The substance of your post is describing price discovery - price discovery isn't the issue here. The issue is that the price discovery is being done in milliseconds. The substance of the complaint is the speed of the auction that discovers the price. The faster price discovery is being done, the fewer traders can participate. If a few milliseconds can change the value of a…

> If a few milliseconds can change the value of a financial instrument then we have to accept that waiting a second or two will allow a lot more traders to reevaluate and offer a fair price. I'm not quite getting this point. If I have a sell order at price x and it's filled by a HFT 1 second before someone with a slower algorithm, how does that result in a more "fair" price for me? If the HFT instead posts a buy orde…

That would be totally boring trading and unremarkable, but that isn't what HFT firms are doing. If that were the strategy then exchanges would presumably just sell a skip-the-queue service when two traders offer the same price and there wouldn't be much point worrying about how far machines are from the exchange.

HFT trading isn't about executing the same trade as someone else but a tiny margin faster. That wouldn't have any special impact on market spreads or liquidity, for example.

There aren't any complaints against the T in HFT; as traders they are helpful. The value questions are about the HF and whether it is a desirable part of the market or an unhelpful arbitrage opportunity created only by implementation details of the exchange.

Re: Lessons learned building an ML trading system

#119

Post by alexcnwy »

ghgr wrote: post #116
alexcnwy wrote: post #113

Earlier quoted context omitted.

I'd love to hear more about what kind of data augmentation you're doing. A friend of mine recently got a GAN to work for timeseries which is really interesting. I've done a lot of work in the space and would love to chat - just emailed you :)

Indeed GANs are showing very promising results. There's a series of blog posts from Fernando de Meer which discuss this topic in a very approachable way. https://quantdare.com/generating-financial-series-with-gener...

Very cool, thanks for the link!

Re: Lessons learned building an ML trading system

#120

Post by alexcnwy »

thundergolfer wrote: post #76

This post is an exemplar of the crucial relationship between domain-specific knowledge and ML competency in the ML space. The bulk of the post is detailing the tricky ins and outs of trading, and overall the author gives the impression that they're broadly knowledgeable about stock markets. Contrast this post with those you see with ML hobbyists who delve into medicine or fake-news and produce useless results testame…

Totally agree.

Funnily enough I think the ML hobbyist problem is most pervasive in the "predict the stock market" domain. There was a post on HN a few days ago [1] that was overfitting the validation set and hand-waving away fees and spreads. The author concluded that "there was no subtle underlying pattern" because they failed to find one.

[1] https://news.ycombinator.com/item?id=21624907

Post reply on HN