Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: I'm building an open-source, high-frequency trading system

scarcecapital.com

Post reply on HN
Print view View original

71–80 of 198 posts

Re: Show HN: I'm building an open-source, high-frequency trading system

#71

Post by zygomega »

boothead wrote: post #69
boothead wrote: post #59

Earlier quoted context omitted.

Are you planning to run R in production, or is it purely for research that then gets translated to haskell?

Replying here as I can't add on another reply below The way we do our monitoring/UI is with zeromq/protocol buffers as an external surface to the trading system (basically a PUB socket) and a very thin bridging and translation layer to WebSockets and JSON in python. That way you get to use d3 or anything else on the front end. We use backbone, knockback and d3 with coffeescript.

Sounds great, but way beyond either my paygrade or budget.

Re: Show HN: I'm building an open-source, high-frequency trading system

#72

Post by zygomega »

acjohnson55 wrote: post #64
zygomega wrote: post #43

Earlier quoted context omitted.

Good points. To be more precise, I think that an open source approach to system research and design can be hyper-competitive versus closed-door secret-squirrel development run by a committee looking for short-term wins. This project benefits me firstly because I get feedback on my initial scratchings and maybe even collaboration on areas outside my comfort zone. And it could well enable me and others to avoid having…

Others are two lazy? It's my understanding that the average Joe has no access to the hardware and low latency network connections to be able to do this on his own.

There is a rather large economic rent attached to HFT at the moment. If my project or others like it can eat into that rent a touch, it might not stay that way.

Re: Show HN: I'm building an open-source, high-frequency trading system

#73

Post by zygomega »

tolitius wrote: post #58

1. Timestamps in your seed data might benefit from nanoseconds if you really talking about "high" frequency. 2. I agree with you comment that it is easier to think about concurrency in Haskell than in something like C++, however you can't really compete with C/C++ in Haskell. Not even with cgo (Go packages that call C code), not with OCaml or any other higher level beasts that promise the speed. Fortran would be the…

I wish I could get down to nano units. iqfeed (a good value feed) just got millisecs in so will settle for that.

I'm preparing some speed tests between C++ and haskell on an identical block of processing so stay tuned! You might be surprised - haskell is way ahead of clojure on compiler smarts.

The split you suggest is exactly what I think is wrong with the way things get done right now. I'd like to integrate the quant inside the read and stream - now that's potentially a large speed up that might compensate a tight budget.

Re: Show HN: I'm building an open-source, high-frequency trading system

#74

Post by boothead »

zygomega wrote: post #66
boothead wrote: post #59

Earlier quoted context omitted.

Are you planning to run R in production, or is it purely for research that then gets translated to haskell?

I see the research as production. There's an autonomous system processing the event stream and deciding on trades that's hands free. I see that as a haskell codebase with C speedups for critical stuff, with no R anywhere to be seen. But then I see a wrapper piece that reports to humans about what's going on within the system. I see R as a great visualization tool for that. I've had good success so far with using R as…

I've had a quick read through the .org file now and it appears that you're using R for things like moving averages over the stream of prices. Have you considered using one of Haskell's FRP (or pipes/conduit) libraries for this side of things?

Re: Show HN: I'm building an open-source, high-frequency trading system

#75

Post by fawce »

zygomega wrote: post #46
boothead wrote: post #38

Have you considered python for the research/algo side of things? I work in a hedge fund in London at the moment and there's a massive shift away from R towards python in the algorithmic shops that I know about here. There's also the great work that quantopian are doing on their backtesting framework zipline [1] [1] https://github.com/quantopian/zipline

Yes, I am aware of that trend and considering a switch. I'm personally more used to R is all, but if I found collaborators I'd jump to where the code base goes. The only other factor is I find R pretty aligned with haskell having a somewhat functional pedigree, so that code translates pretty nicely between a rapid hack at the problem to the more robust approach.

zygomega, I'm one of the zipline maintainers and we'd love to collaborate with you. There are a few people doing HFT research with zipline, and there's a lot of work to do. At quantopian (my day job), we focus on longer hold periods, so there is room in the zipline ecosystem for you to do HFT.

The main benefit we've found with python as the algo language is that it allows for stat programming with pandas, but also OO or functional programming for the algo logic. This smoothes the transition from research to production, just as you're describing with R -> haskell, but you can stay in one language.

I think one of the biggest potential wins with parallelization is if you can assume all positions are closed overnight, most often true for HFT. That way, you can simulate all the trading days in a test range in parallel. This is quite similar to the parallel processing we do to handle the large number of concurrent backtests running at quantopian. We did all of that with python, but I'd be fascinated to see it done with haskell.

Re: Show HN: I'm building an open-source, high-frequency trading system

#76

Post by mseebach »

white_devil wrote: post #27
zygomega wrote: post #9

Earlier quoted context omitted.

No, I'm already in the finance industry. There are two main reasons I'm doing this: - I think the finance industry is very closed when it comes to intellectual property development, and an open source approach can be seriously competitive. An open approach may well be the future when it comes to being 'connected' - HFT is an interesting multi-disciplinary problem and the shear breadth of expertise required - modern c…

> an open source approach can be seriously competitive Do you really think any Joe Schmoe off the street could just grab your open source HFT and start making money with it? If not, how does your project benefit anyone? Are you working in the industry, but not in HFT? Maybe this is project is just practise for getting a job in HFT? I imagine that's where programmers get the fattest paychecks in the world. Only a frac…

> Good thing you're not at all biased.

> As you may well be aware, HFT is a scourge on the world's economy, and it's a game only the biggest and best-connected players benefit from.

Good thing you're not stating opinion as fact.

Re: Show HN: I'm building an open-source, high-frequency trading system

#77

Post by nazka »

olalonde wrote: post #21
nazka wrote: post #17

I am curious about this project because I love this subject. However, it is impossible to build an open source HFT system especially because it will be open source. The other sharks will know how you trade and they will only need to play against you by sending opposite orders every time. That is what happens when the source code of a big US bank were stolen. So it should be more appropriate to say it will be a fast t…

Couldn't the infrastructure be open source while the specific trading algorithms kept closed source?

Yes you send your code on the servers of a company and you use their API to hit the market with their servers, or you rent your own racks. It will cost you a lot but in the HFT world it is the only solution. Any nano second counts. You are the fastest or you are not.

Re: Show HN: I'm building an open-source, high-frequency trading system

#78

Post by zygomega »

danielhughes wrote: post #68
zygomega wrote: post #32

Earlier quoted context omitted.

If you need smart algorithms operating autonomously versus motivated enemies your go to must be open source. The security guys proved that a long time ago. You're right in that the HF bit doesn't really fit but 'fast trading' doesn't have the same ring to it.

I don't think your security analogy applies to trading algorithms. Open source is good for security because it aids in the discovery of vulnerabilities. However, good trading algorithms are usually reliant on some form of information asymmetry. If you make a discovery and develop a trading algorithm to exploit it, I would think it would be advantageous to keep it a secret for as long as possible. This is not my field…

Thanks Dan, the analogy is a stretch maybe, but most algorithms fail due to being narrow models of reality, and criticised by too few eyes. If I discover something, it's extremely unlikely that others will believe it anyway (unless I prove it by becoming filthy rich as a result in which case I'm just fine).

You're pretty right about good trading algorithms but that doesn't necessarily apply to good traders or good trading systems. A good system - hft or otherwise - needs to be flexible and evolve to stay in touch with an evolving ecology. And to do this, it needs to keep track of many different trading ideas and algorithms, each of which may have its day in the sun.

Re: Show HN: I'm building an open-source, high-frequency trading system

#79

Post by VLM »

olalonde wrote: post #21
nazka wrote: post #17

I am curious about this project because I love this subject. However, it is impossible to build an open source HFT system especially because it will be open source. The other sharks will know how you trade and they will only need to play against you by sending opposite orders every time. That is what happens when the source code of a big US bank were stolen. So it should be more appropriate to say it will be a fast t…

Couldn't the infrastructure be open source while the specific trading algorithms kept closed source?

This could be a lot more fun than classic codewars.

For 99% of the "system" there's no particular reason you'd have to connect to the NYSE and trade real stocks for real money. Looks like they're feeding from iqfeed, but I think it would be huge fun to create an imaginary competition exchange, convince about 100 algo writers to compete, and shove them up against each other purely for the fun of it, see who's a better algo writer.

With modern virtualization, it should be pretty easy to distribute a pack of images, including practice data, to replicate an entire financial system on a small scale in your basement, then once you think you have a decent algo, upload your image to the competition league. If your hft codewars league wants limits for storage or cpu cycles, virtualization is a pretty easy way to implement it.

Superficially the most obvious thing to do is totally make everything up, from top to bottom, but it would probably work just as well to use real live market data and overlay on top of it. Just as air traffic control simulation has very slowly moved over the decades from purely random to sorta realistic to actual airport data.

This could be quite a bit of fun.

Re: Show HN: I'm building an open-source, high-frequency trading system

#80

Post by zygomega »

qompiler wrote: post #47

Sorry to burst your bubble but high-frequency trading is only viable and necessary for market makers. The end result of any type of algorithm you think of is always a curve-fitting function of existing data.

No need to apologise - curve fitting is a big issue. But I would think that everything that happens on kaggle would be curve fitting by your definition right? And how is the stuff that happens in brains something other than curve fitting? It's how you connect the data dots and what you choose to focus on that makes the difference.
Post reply on HN