Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

High Frequency Trading Development Kit

stoneridgetechnology.com

Post reply on HN
Print view View original

31–40 of 83 posts

Re: High Frequency Trading Development Kit

#31

Post by dtby »

bwhite wrote: post #19
roel_v wrote: post #16

Earlier quoted context omitted.

Set it up, trade 'virtually' (i.e., just simulate), prove your algos, get bankrolled by someone who is convinced of the algos worth. Quite straightforward. I'm highly skeptical of a hobbyist beating the competition though (maybe in micro niches, where one would add domain specific knowledge to get an edge - that's my pet idea on this topic :) )

It is straightforward, but impractical. Getting good historical market data is very, very expensive. Even recording current market data is not cheap and not without significant difficulty.

Why would one even consider entering these markets without the ability to demonstrate alpha over historical results?

Yes, your alpha needs to include the ability to beat transaction costs... so if you can't pay for the [relatively exceptionally cheap] cost of several years of your chosen market's historical data, maybe you're not really serious.

It's fun to play at being a financier, but let's not pretend that access to market data is the significant barrier to entry which is preventing disruption in the world financial markets.

Re: High Frequency Trading Development Kit

#32

Post by bwhite »

keithwarren wrote: post #22
bwhite wrote: post #10

Earlier quoted context omitted.

There are plenty of prop shops that would let you trade with a very modest (low 5-figure or even 4-figure) dollar amount. Some will leverage your money, some will flat-out stake you in addition to your input. These places have the benefit of offering you relatively low costs -- minuscule compared to the cost of a retail trade at your average discount broker -- but they do not necessarily have the sort of investment i…

If there are plenty, could you please point one out?

Start at: http://www.traderslog.com/proprietarytradingfirms/

Some of these are purely manual trading but many have software APIs.

Re: High Frequency Trading Development Kit

#33

Post by [deleted] »

brownegg wrote: post #12
tghw wrote: post #5

Earlier quoted context omitted.

A month or two ago I spoke to someone working in these shops. Apparently, they're using the FPGAs to run many of the high-bandwidth algorithms, because going from NIC to motherboard to CPU and back again added too much latency. So they program the algorithms into the FPGA (which has its own Ethernet port) so it can receive, process, and send the data without ever hitting circuitry not dedicated to the task.

This is true, but FPGAs can still only handle relatively simple algorithms. FPGAs make certain kinds of tasks nearly trivial (think truly parallel tasks such as decoding market data for different instruments), but there's just not enough gates (in most cases) to do interesting stuff in the application / logic layer. FPGAs canonical use case is still as very fast codecs. There are exceptions, but they're really only t…

[deleted]

Re: High Frequency Trading Development Kit

#34

Post by Nrsolis »

brownegg wrote: post #12
tghw wrote: post #5

Earlier quoted context omitted.

A month or two ago I spoke to someone working in these shops. Apparently, they're using the FPGAs to run many of the high-bandwidth algorithms, because going from NIC to motherboard to CPU and back again added too much latency. So they program the algorithms into the FPGA (which has its own Ethernet port) so it can receive, process, and send the data without ever hitting circuitry not dedicated to the task.

This is true, but FPGAs can still only handle relatively simple algorithms. FPGAs make certain kinds of tasks nearly trivial (think truly parallel tasks such as decoding market data for different instruments), but there's just not enough gates (in most cases) to do interesting stuff in the application / logic layer. FPGAs canonical use case is still as very fast codecs. There are exceptions, but they're really only t…

Wow. This is almost entirely wrong.

Any modern FPGA (hint: the Virtex-5 they are using in these machines) can implement a relatively complex general purpose CPU. It wont be a Pentium 4, but it can be complex enough to run programs you'd need to design your own compiler for.

Of course, they only ever use the IP cores that they need to implement the algorithm they want. That usually means modeling the limit order book.

Monte Carlo, anyone?

Re: High Frequency Trading Development Kit

#35

Post by JonnieCache »

shn21 wrote: post #29
profquail wrote: post #14

Earlier quoted context omitted.

It looks like you got a few downvotes because people assumed you're being snarky (maybe you are?) -- but I think that most people (even hackers) who work outside of the financial sector are going to wonder why this is on the front page. In short though, it's a 1U blade server with a full-size FPGA in it; implementing your algorithms in hardware makes them extremely fast (thus the mention of the C-to-HDL compiler), wh…

How one donwvotes? I do not see a downvote link?

You need 500 karma to downvote.

Re: High Frequency Trading Development Kit

#36

Post by larsberg »

cdavid wrote: post #7
tghw wrote: post #5

Earlier quoted context omitted.

A month or two ago I spoke to someone working in these shops. Apparently, they're using the FPGAs to run many of the high-bandwidth algorithms, because going from NIC to motherboard to CPU and back again added too much latency. So they program the algorithms into the FPGA (which has its own Ethernet port) so it can receive, process, and send the data without ever hitting circuitry not dedicated to the task.

This seems a bit surprising: what is the latency coming from the NIC to the motherboard ? I somehow doubt the latency due to distance within one piece of hardware matters (current CPU cannot do much more than a few 1000s cycles for the electricity to go around a few times in a motherboard, which is nothing to do any kind of advanced processing).

It's pretty terrible (in HFT terms), particularly if you're trying to use a junky on-board NIC.

Folks not using FPGAs use separate ethernet cards and enable kernel bypass and can do significantly better with that.

Re: High Frequency Trading Development Kit

#37

Post by swah »

brownegg wrote: post #12
tghw wrote: post #5

Earlier quoted context omitted.

A month or two ago I spoke to someone working in these shops. Apparently, they're using the FPGAs to run many of the high-bandwidth algorithms, because going from NIC to motherboard to CPU and back again added too much latency. So they program the algorithms into the FPGA (which has its own Ethernet port) so it can receive, process, and send the data without ever hitting circuitry not dedicated to the task.

This is true, but FPGAs can still only handle relatively simple algorithms. FPGAs make certain kinds of tasks nearly trivial (think truly parallel tasks such as decoding market data for different instruments), but there's just not enough gates (in most cases) to do interesting stuff in the application / logic layer. FPGAs canonical use case is still as very fast codecs. There are exceptions, but they're really only t…

But a Virtex 5 could hold several simple CPUs designed for this task ;)

I don't know if the algorithms they use can be expressed easily as state machines though.

Re: High Frequency Trading Development Kit

#38

Post by dman »

chollida1 wrote: post #24
ortatherox wrote: post #17

Earlier quoted context omitted.

Thanks, I came to ask that question also. So is there any reason that this can't be done using OpenCL instead?

We've tried using OpenCL for pricing, the biggest bottle neck is getting data into and out of video card memory. It's just too slow to move it from main to video memory and back. Also OpenCL works best when working on large pieces of data at once, with High frequency trading often you want to work on very small pieces, 50 bytes, and process them as fast as possible. Batching up the data and working on it in a SIMD ty…

APU's should fix some of that.

Re: High Frequency Trading Development Kit

#40

Post by bwhite »

dtby wrote: post #31
bwhite wrote: post #19

Earlier quoted context omitted.

It is straightforward, but impractical. Getting good historical market data is very, very expensive. Even recording current market data is not cheap and not without significant difficulty.

Why would one even consider entering these markets without the ability to demonstrate alpha over historical results? Yes, your alpha needs to include the ability to beat transaction costs... so if you can't pay for the [relatively exceptionally cheap] cost of several years of your chosen market's historical data, maybe you're not really serious. It's fun to play at being a financier, but let's not pretend that access…

The cost of market data and a system equipped to process it are not inconsiderable.

Historical tick data (quotes and trades) for US equities will run you $20k/year, minimum. US equities runs about 30-40GB/day so you are looking at almost 10TB/year. You need to be able to access the data quickly so you probably don't want to compress it and probably do want to duplicate the data, slicing and dicing it in different ways. Call it 20TB, minimum, per year of data.

Futures and options data will run another $20k/year, each, and futures has about as much data as equities and options has an order of magnitude more. That is another 100TB/year, minimum, per year, probably closer to 200TB.

How are going to to study these data? Even if you were able to operate at disk speed (which seems unlikely unless you have lots of disks and have duplicated and distributed your data amongst them) it would take you five minutes to run against one day's worth of data, or close to a day to run a year's simulation. If you consider more than a few cores in parallel, reading from disk is going to be seriously hampered.

None of this even considers the cost of the software necessary to support this effort: contending with corporate actions, tracking halts, etc.

This all adds up to a significant financial hurdle.

Post reply on HN