Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: I'm building an open-source, high-frequency trading system

scarcecapital.com

Post reply on HN
Print view View original

131–140 of 198 posts

Re: Show HN: I'm building an open-source, high-frequency trading system

#131

Post by fnordfnordfnord »

white_devil wrote: post #2

Are you trying to get hired by the financial industry, or is there some other reason for doing this? As you may well be aware, HFT is a scourge on the world's economy, and it's a game only the biggest and best-connected players benefit from.

>HFT is a scourge on the world's economy

I'd point out that a lot of the things that are going on in HFT, are rehashes of old trading scams, and either are or would be illegal if there was any adult supervision. I was tempted to say something about the SEC being left behind the technology, and doesn't understand it. But for that to be true, and have this crap go on for so long, they either need to be complete fools, or they benefit from the status quo.

>and it's a game only the biggest and best-connected players benefit from.

That's definitely true in the case of latency arbitrage. But here again. It's the exchanges themselves that have decided that some players get an advantage over others. As an example, latency arbitrage, and a whole host of other problems could be sorted out by putting traders on exchange-hosted virtual machines. http://www.dailyfinance.com/2010/06/05/rigged-market-latency...

If not virtual machines, there are other ways to fix these problems, but first you need to get the players to acknowledge that there is a problem.

http://www.zerohedge.com/article/do-it-yourself-latency-arbi...

Nanex has done a lot of good analysis of HFT, some of which is published here:

nanex.net/FlashCrash/OngoingResearch.html

Re: Show HN: I'm building an open-source, high-frequency trading system

#132

Post by kasey_junk »

zygomega wrote: post #128
kasey_junk wrote: post #124

Earlier quoted context omitted.

As your complexity curve increases, your latency expectations must go down. That is what I meant when I said there is no single trading system that can be the "best". If what you are interested in is complex decision making then it may make sense to use a different sort of messaging technology than LMAX, but you won't be getting into anything remotely "low latency". Nothing wrong with that, just needs to be a known e…

Yep, I've poured over the technology and concepts there and its awesome. So awesome I think you can increase complexity without too much of a latency penalty. Just behind the fast guys but way ahead of the straight algo guys is where I think the opportunities lie (I'm kind of answering your previous question about where the dials sit in my mind). I might be wrong but, and would like to test out all dial positionings…

I have no opinion about the choice of haskell. That said, you are already making decisions about the dials if you go in with an architecture that relies on massive concurrency, functional languages, etc. You cannot for instance get the latency dial very low with that central architecture decision.

Again there is nothing wrong with that. It's just better to state it up front as an expectation and/or a goal than to assume you are going to be able to pivot easily once you have an architecture in place.

Re: Show HN: I'm building an open-source, high-frequency trading system

#133

Post by fnordfnordfnord »

zygomega wrote: post #72
acjohnson55 wrote: post #64

Earlier quoted context omitted.

Others are two lazy? It's my understanding that the average Joe has no access to the hardware and low latency network connections to be able to do this on his own.

There is a rather large economic rent attached to HFT at the moment. If my project or others like it can eat into that rent a touch, it might not stay that way.

Exactly, if you take the position that gains from HFT are basically "stolen" profits from otherwise more productive market participants, then any that can be "stolen back" is a win.

Re: Show HN: I'm building an open-source, high-frequency trading system

#134

Post by spitfire »

sseveran wrote: post #118
carterschonwald wrote: post #105

In another month or two the analytical tooling for doing stats and numerics in Haskell land are going to have a huge leap forward in capabilities. Might be worth considering going full Haskell then :-)

Whats happening then?

From my talks with Carter previously he's working on a haskell based platform for analytic tooling in Haskell. Basically the core primitives for making large scale data analysis apps in Haskell.

Or that's what he was up to in august. Hopefully he can chime in here and update everyone on his progress. I honestly hope he succeeds in his plans.

Re: Show HN: I'm building an open-source, high-frequency trading system

#135

Post by zygomega »

carterschonwald wrote: post #126
sseveran wrote: post #118

Earlier quoted context omitted.

Whats happening then?

you know exactly what i'm up to slowly with wellposed :) (building numerics/ data analysis tools that dont' fill me with rage and ire over terrible engineering and usability. A matrix / linear algebra kernel of tools are on track to be ready for hackage release + paid pro versions in 1-2 months)

Classical landing page had me hooked anyways :-)

Re: Show HN: I'm building an open-source, high-frequency trading system

#136

Post by chicagobob »

boothead wrote: post #38

Have you considered python for the research/algo side of things? I work in a hedge fund in London at the moment and there's a massive shift away from R towards python in the algorithmic shops that I know about here. There's also the great work that quantopian are doing on their backtesting framework zipline [1] [1] https://github.com/quantopian/zipline

Are you aware of zipline ports (or similar libraries) in Ruby or ObjectiveC (or plain C)?

Re: Show HN: I'm building an open-source, high-frequency trading system

#137

Post by zygomega »

bdunbar wrote: post #110

> using haskell,R and emacs org-mode. That last is so very, very, cool. I'll have to dive in, take a look, see if I can find out _how_ you're using it. Been wanting to do a project with org-mode for a while, now.

It is - I can't imagine working without it. But I'm not sure I'm the poster child for how to use org-mode properly. Every piece of code in the repo actually resides in the org file which is a big monolithic journal of where I'm at.

o-blog is a great emacs package that lets you publish sites straight from org.

Re: Show HN: I'm building an open-source, high-frequency trading system

#138

Post by fnordfnordfnord »

zygomega wrote: post #129
fnordfnordfnord wrote: post #123

FYI: This isn't an HFT bot but is a decent, recent Python + ncurses trading client framework for Bitcoin on MtGox. http://prof7bit.github.io/goxtool/ Even if you aren't interested in Bitcoin, it might be useful as a real-ish and cheap place to test with low barriers to entry. I've found that backtesting on historical data is usually not realistic enough since most people fail to consider liquidity. I'm interested bec…

Isn't MtGox over as a bitcoin trading platform? I agree more generally that bitcoin could be a useful market to test out. And your story fits exactly with a hands-off automated approach to trading. We wont ever be charting a price series in the production environment because computers don't get charts.

>Isn't MtGox over as a bitcoin trading platform?

I wish. I keep waiting for some more interesting news from Coinlab [1][2], but so far, Mtgox are still the largest exchange by volume. Here is a chart showing volume by (selected) exchanges. http://bitcoincharts.com/markets/currency/USD.html

The recent events seem to only have got them more customers (though it remains to be seen how many of them will stick). In any case, some of the other exchanges have API's that are similar to mtgox.

[1] https://mtgox.com/press_release_20130228.html

[2] http://www.forbes.com/sites/jonmatonis/2012/04/24/coinlab-at...

Re: Show HN: I'm building an open-source, high-frequency trading system

#139

Post by mangrish »

Cool! Very interesting.

I think given the languages you have selected you are coming more from a quant background? These languages are great for heuristics and analysis but you would really want all 'static' components such as connectivity built in assembly/C/C++. For 'algo' components I like Java as you can still pull microsecond order latencies when crunching numbers but more importantly it gives you a huge time to market advantage than the C/C++ for almost the same speeds. I'm also not clear on if you are connecting directly to the market for market data or using aggregation (like reuters). The latter would be too slow. I'm also not clear on what middleware you are using which is probably the biggest decision you will have to make. Most either use inhouse tech or 29west LBM (everyone still calls it that even though they were bought out).

An overlooked part of HFT in my experience OS optimisation and even things like TCP bypass (for some components) which can lead to huge speed advantages and end to end latency reductions. I agree with those about FPGA.. in my experience they really don't come into the equation except for components that rarely change.

Anyway a few guys including Martin Thompson have felt similarly to you and initiated the lodestone project (http://lodestonefoundation.wordpress.com/). If you are keen to learn more their architectures for low latency, distributed and componentisation then I feel that you could join forces and contribute to that initiative. FYI: Martin built most of the technology behind the disruptor (http://lmax-exchange.github.io/disruptor/).

Great to see interest in making this knowledge more widely available :)

Re: Show HN: I'm building an open-source, high-frequency trading system

#140

Post by zygomega »

alexforster wrote: post #97

I have no experience in the financial industry, but HFT has always fascinated me as a potential source for very high rates of "events". Could you share your general insight about the sheer volume of data that commonly gets pushed through an HFT system? I'd also be terribly interested in a multi-megabyte/gigabyte "recording" of HFT trade data. http://www.nyxdata.com/capacity

So, yes, on my toy wannabe hft feed, I'm currently clocking about 10 million ticks per day on about 8 futures tickers I'm tracking. The raw feed is between 1 and 1.5 gig per day. I can imagine the petabyteness of the bigger guys.
Post reply on HN