Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LMAX Disruptor – High Performance Inter-Thread Messaging Library

lmax-exchange.github.io

Post reply on HN
Print view View original

21–30 of 89 posts

Re: LMAX Disruptor – High Performance Inter-Thread Messaging Library

#21

Post by onetimeuse92304 »

defrost wrote: post #19
vivzkestrel wrote: post #18

Earlier quoted context omitted.

what language do you think would be the backbone for such a system? C/C++/Golang or something high level like node.js/Java

Each to their own but if you read and understand the comment above they're describing a dedicated OS for the task .. so think about what you'd choose to write a small task dedicated OS with. Simple C is most likely, ASM is possible, a language such as OCaml generating C to hook into the low level buffers would be intriguing ... the list is long and largely determined by the experience preference of whoever tackles it…

You are surprisingly accurate.

I used combination of Common Lisp (SBCL), ANSI C and assembly (not much assembly, though, only very small pieces that I had trouble emitting other ways).

The main application was in Lisp, it would start up and set up the environment (using some low level code written in C/assembly).

But everything on the path of the market signal to order would be highly optimised native machine code, but some of that code would be written in C and some of it would be compiled at runtime with Lisp.

Parsing the incoming multicast feed from the exchange was implemented with Lisp emitting super efficient machine code (almost no function calls) based on a bunch of XML files describing the messages. I shamelessly stole the idea from the book Practical Common Lisp (really good if you want to get into Lisp).

Things like business rules would be compiled into decision trees using current market situation and decision trees would be reorganised, optimised and compiled into machine code and inserted into the processing path. Most of the time a very large decision tree with hundreds of complex decisions (for example, taking into account market volatility) could be distilled to just couple branching instructions. This decision tree compilation would happen after every market signal, up to 10 thousand times a second (the exchange had a basic tick of 1/10000 of a second giving me guaranteed 100us before the next market signal).

Same with actual algorithms -- I wrote a small DSL for the traders and this DSL would be compiled with Lisp to machine an inserted into the processing path.

Some parts of the framework would be built with Lisp, too. It was easier for me to write a DSL and then compile it to machine than to write it in C.

If you want to understand one principle from all this is to look at all instructions and especially branches that you have between receiving the signal and emitting the order and try to figure out if you can eliminate it and if you can't, try to find ways to do it ahead of time even if doing it ahead of time requires a lot more effort.

Re: LMAX Disruptor – High Performance Inter-Thread Messaging Library

#22

Post by defrost »

onetimeuse92304 wrote: post #21
defrost wrote: post #19

Earlier quoted context omitted.

Each to their own but if you read and understand the comment above they're describing a dedicated OS for the task .. so think about what you'd choose to write a small task dedicated OS with. Simple C is most likely, ASM is possible, a language such as OCaml generating C to hook into the low level buffers would be intriguing ... the list is long and largely determined by the experience preference of whoever tackles it…

You are surprisingly accurate. I used combination of Common Lisp (SBCL), ANSI C and assembly (not much assembly, though, only very small pieces that I had trouble emitting other ways). The main application was in Lisp, it would start up and set up the environment (using some low level code written in C/assembly). But everything on the path of the market signal to order would be highly optimised native machine code, b…

Not my first rodeo :-)

I too made a real time hardware level trading system back in the day, on the back of building a multi channel seismic aquisition system with a custom RealtimeOS talking to a bunch of DSP cards that each sampled a trailing flotation cable that each had multiple microphones with the entire grid going toward building up a profile of the seafloor and layers underneath along with bobbing boat|cable cancelation, etc.

Very similar architectures in many ways - with rolling DSP filters in place of trading response algorithms .. etc.

Lisp -> ASM or Lisp -> C either way code generation from a higher language was a good way to get the heavy lifting done.

Re: LMAX Disruptor – High Performance Inter-Thread Messaging Library

#23

Post by pclmulqdq »

Every time a new generation plays with the LMAX disruptor, it's time to remind them that the modes with multiple producers/consumers can have really bad tail latency if your application's threading is not designed in the intended way.

Disruptor and most other data structures that come from trading are designed to run with thread-per-core systems. This means systems where there will be no preemption during a critical section. They can get away with a lot of shenanigans on the concurrency model due to this. If you are using these data structures and have a thread-per-request model, you're probably going to have a bad time.

Re: LMAX Disruptor – High Performance Inter-Thread Messaging Library

#24

Post by xoranth »

pclmulqdq wrote: post #23

Every time a new generation plays with the LMAX disruptor, it's time to remind them that the modes with multiple producers/consumers can have really bad tail latency if your application's threading is not designed in the intended way. Disruptor and most other data structures that come from trading are designed to run with thread-per-core systems. This means systems where there will be no preemption during a critical…

In general, what are the advantage of a thread-per-request model? Better load balancing between cores?

Re: LMAX Disruptor – High Performance Inter-Thread Messaging Library

#25

Post by sakras »

colanderman wrote: post #6

I had implemented more-or-less this same concurrency scheme for an IPS/DDoS prevention box ~10 years ago, running on Tilera architecture. It was fast (batching + separating read & write heads really does help a ton)... but not as fast as Tilera's built-in intercore fabric. It had some limitations but was basically a register store/load to access and only like 1 or 2 cycles intercore latency. (Aside, generic atomic op…

> batch your updates locally to issue fewer of them at least

I don’t know why I never thought of this, brilliant!

Re: LMAX Disruptor – High Performance Inter-Thread Messaging Library

#26

Post by onetimeuse92304 »

defrost wrote: post #22
onetimeuse92304 wrote: post #21

Earlier quoted context omitted.

You are surprisingly accurate. I used combination of Common Lisp (SBCL), ANSI C and assembly (not much assembly, though, only very small pieces that I had trouble emitting other ways). The main application was in Lisp, it would start up and set up the environment (using some low level code written in C/assembly). But everything on the path of the market signal to order would be highly optimised native machine code, b…

Not my first rodeo :-) I too made a real time hardware level trading system back in the day, on the back of building a multi channel seismic aquisition system with a custom RealtimeOS talking to a bunch of DSP cards that each sampled a trailing flotation cable that each had multiple microphones with the entire grid going toward building up a profile of the seafloor and layers underneath along with bobbing boat|cable…

Cool project!

I guess when different people try to achieve extreme low latency or efficiency, the solutions start to converge into small number of ideas.

Re: LMAX Disruptor – High Performance Inter-Thread Messaging Library

#27

Post by jiggawatts »

onetimeuse92304 wrote: post #15
vivzkestrel wrote: post #10

stupid question: how to build a trading system? anyone got a starter guide, resources?

I built a PoC of a 5us trading system (guaranteed 5us response in every situation) for a brokerage house a long time ago, around the time of LMAX Disruptor. It was one man job and I had to start with nothing (they had no knowledge at all). Fun project and I learned a lot. * full kernel bypass (I even implemented driver for the networking hardware) * everything that could disrupt the application disabled (like SME int…

> the main insight was that rather than wait for market signals to then decide what to do, you can precalculate your responses up to and including the actual message to be sent to the exchange.

I saw a talk about this dialed up to eleven: the entire processing occurred in a "smart NIC" instead of the CPU. The response would start getting sent even as the inbound packet was still being received. The go/no-go decision was effectively just sending the final CRC bytes correctly or deliberately incorrectly, thus invalidating the outbound packet that was already 99% sent.

Before that talk I couldn't figure out why there was a market for NICs with embedded FPGAs, CPUs and memory.

Day traders basically subsidised these things, and now they do efficient packet switching for large cloud providers.

Reminds me of how crypto-mining subsidised a lot of GPU development, and now we have 4K ray tracing and AIs thanks to that.

Re: LMAX Disruptor – High Performance Inter-Thread Messaging Library

#28

Post by onetimeuse92304 »

xoranth wrote: post #24
pclmulqdq wrote: post #23

Every time a new generation plays with the LMAX disruptor, it's time to remind them that the modes with multiple producers/consumers can have really bad tail latency if your application's threading is not designed in the intended way. Disruptor and most other data structures that come from trading are designed to run with thread-per-core systems. This means systems where there will be no preemption during a critical…

In general, what are the advantage of a thread-per-request model? Better load balancing between cores?

Thread per request can never have better load balancing between cores than a well designed, custom solution.

You are essentially asking the operating system to do the scheduling for you. But the OS will never be able to do it perfectly as it has no knowledge of what your application is doing.

The main advantage of OS scheduling is that you get pretty good results without having to think about it at all. Pretty good, but never perfect.

Re: LMAX Disruptor – High Performance Inter-Thread Messaging Library

#29

Post by vivzkestrel »

defrost wrote: post #22
onetimeuse92304 wrote: post #21

Earlier quoted context omitted.

You are surprisingly accurate. I used combination of Common Lisp (SBCL), ANSI C and assembly (not much assembly, though, only very small pieces that I had trouble emitting other ways). The main application was in Lisp, it would start up and set up the environment (using some low level code written in C/assembly). But everything on the path of the market signal to order would be highly optimised native machine code, b…

Not my first rodeo :-) I too made a real time hardware level trading system back in the day, on the back of building a multi channel seismic aquisition system with a custom RealtimeOS talking to a bunch of DSP cards that each sampled a trailing flotation cable that each had multiple microphones with the entire grid going toward building up a profile of the seafloor and layers underneath along with bobbing boat|cable…

what would you do if you wanna run all this on a web server instead? sorry if thats the wrong question. how does your system interact with say android or ios clients or a webapp with ui

Re: LMAX Disruptor – High Performance Inter-Thread Messaging Library

#30

Post by onetimeuse92304 »

jiggawatts wrote: post #27
onetimeuse92304 wrote: post #15

Earlier quoted context omitted.

I built a PoC of a 5us trading system (guaranteed 5us response in every situation) for a brokerage house a long time ago, around the time of LMAX Disruptor. It was one man job and I had to start with nothing (they had no knowledge at all). Fun project and I learned a lot. * full kernel bypass (I even implemented driver for the networking hardware) * everything that could disrupt the application disabled (like SME int…

> the main insight was that rather than wait for market signals to then decide what to do, you can precalculate your responses up to and including the actual message to be sent to the exchange. I saw a talk about this dialed up to eleven: the entire processing occurred in a "smart NIC" instead of the CPU. The response would start getting sent even as the inbound packet was still being received. The go/no-go decision…

Cool! I actually wasn't aware about NICs with FPGAs on them. You learn something new every day on HN.

My solution wasn't as fast and it could never do what you describe (start sending bytes before the packet was fully received). The market signal messages were actually batched together (usually one to 5), compressed with zlib and sent as a single multicast packet.

Post reply on HN