Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LMAX Disruptor – High Performance Inter-Thread Messaging Library

lmax-exchange.github.io

Post reply on HN
Print view View original

31–40 of 89 posts

Re: LMAX Disruptor – High Performance Inter-Thread Messaging Library

#31

Post by waynesonfire »

onetimeuse92304 wrote: post #21
defrost wrote: post #19

Earlier quoted context omitted.

Each to their own but if you read and understand the comment above they're describing a dedicated OS for the task .. so think about what you'd choose to write a small task dedicated OS with. Simple C is most likely, ASM is possible, a language such as OCaml generating C to hook into the low level buffers would be intriguing ... the list is long and largely determined by the experience preference of whoever tackles it…

You are surprisingly accurate. I used combination of Common Lisp (SBCL), ANSI C and assembly (not much assembly, though, only very small pieces that I had trouble emitting other ways). The main application was in Lisp, it would start up and set up the environment (using some low level code written in C/assembly). But everything on the path of the market signal to order would be highly optimised native machine code, b…

> implemented with Lisp emitting super efficient machine code

I'm really intrigued by your usage of Lisp. It's on my bucket list to learn and your post is very inspiring.

When you say Lisp was emitting machine code; are you referring to the machine code the Lisp compiler emitted for your Lisp application or was your Lisp application acting as a sort of JIT compiler and actually emitting machine code?

Maybe you can reference the section in Practical Common Lisp that introduces the idea?

Re: LMAX Disruptor – High Performance Inter-Thread Messaging Library

#32

Post by onetimeuse92304 »

vivzkestrel wrote: post #29
defrost wrote: post #22

Earlier quoted context omitted.

Not my first rodeo :-) I too made a real time hardware level trading system back in the day, on the back of building a multi channel seismic aquisition system with a custom RealtimeOS talking to a bunch of DSP cards that each sampled a trailing flotation cable that each had multiple microphones with the entire grid going toward building up a profile of the seafloor and layers underneath along with bobbing boat|cable…

what would you do if you wanna run all this on a web server instead? sorry if thats the wrong question. how does your system interact with say android or ios clients or a webapp with ui

If you have a project that you want to see done, you will be more likely to succeed using some more traditional architecture.

What I described is chasing latency at all costs. The costs are hardware costs, maintainability costs, development costs, inefficiency (yes, a lot more CPU is used than what is needed just to run the critical path fast). This is very extreme situation and it would be very unlikely to be a good tradeoff for your application.

If you have a lot of clients connecting there are different tradeoffs to think about and different possible architectures to evaluate, but I can't help you not knowing what your problem is.

Re: LMAX Disruptor – High Performance Inter-Thread Messaging Library

#33

Post by onetimeuse92304 »

waynesonfire wrote: post #31
onetimeuse92304 wrote: post #21

Earlier quoted context omitted.

You are surprisingly accurate. I used combination of Common Lisp (SBCL), ANSI C and assembly (not much assembly, though, only very small pieces that I had trouble emitting other ways). The main application was in Lisp, it would start up and set up the environment (using some low level code written in C/assembly). But everything on the path of the market signal to order would be highly optimised native machine code, b…

> implemented with Lisp emitting super efficient machine code I'm really intrigued by your usage of Lisp. It's on my bucket list to learn and your post is very inspiring. When you say Lisp was emitting machine code; are you referring to the machine code the Lisp compiler emitted for your Lisp application or was your Lisp application acting as a sort of JIT compiler and actually emitting machine code? Maybe you can re…

Machine code isn't some kind of magical idea. It is just bytes and you can craft the bytes any way you want.

In my case I wrote my own compiler that generated machine code bytes based on my needs.

The compiler wasn't at all complicated. It was not general compiler that would have to be able to deal with everything you could throw at it. It only supported one model of CPU. It did not require any optimisation mechanisms because it was assumed the instructions it received were already optimised by high level logic that generated them. It would only support very small number of instructions or mechanisms, it was mostly manipulating registers and doing branches and jumps. It produced small pieces of code to be inserted into larger C program mostly, so that this C program did not have to run decisions on what to do based on something else.

You can also use SBCL to emit any vops you want to execute as part of your Lisp program. So if you want to have a piece of crafted code to run as part of your Lisp program (for example to do some complicated operation that would be inefficient in lisp) you can also do it. I actually used vops to do some low level stuff but that was another matter, most of the time I was just outputting a buffer with crafted native machine code which would then be sent to the receiving thread (core) to insert and execute it for the next market signal.

Re: LMAX Disruptor – High Performance Inter-Thread Messaging Library

#34

Post by citrin_ru »

xoranth wrote: post #24
pclmulqdq wrote: post #23

Every time a new generation plays with the LMAX disruptor, it's time to remind them that the modes with multiple producers/consumers can have really bad tail latency if your application's threading is not designed in the intended way. Disruptor and most other data structures that come from trading are designed to run with thread-per-core systems. This means systems where there will be no preemption during a critical…

In general, what are the advantage of a thread-per-request model? Better load balancing between cores?

Easy to implement.

Re: LMAX Disruptor – High Performance Inter-Thread Messaging Library

#35

Post by nine_k »

citrin_ru wrote: post #34
xoranth wrote: post #24

Earlier quoted context omitted.

In general, what are the advantage of a thread-per-request model? Better load balancing between cores?

Easy to implement.

But does anyone run an OS thread per request unironically? I thought that nearly every request-response server implementation would use a thread pool. The best, like Erlang, can give you the feeling of arbitrarily many extremely cheap threads, while also running on a thread pool.

Re: LMAX Disruptor – High Performance Inter-Thread Messaging Library

#36

Post by xoranth »

onetimeuse92304 wrote: post #28
xoranth wrote: post #24

Earlier quoted context omitted.

In general, what are the advantage of a thread-per-request model? Better load balancing between cores?

Thread per request can never have better load balancing between cores than a well designed, custom solution. You are essentially asking the operating system to do the scheduling for you. But the OS will never be able to do it perfectly as it has no knowledge of what your application is doing. The main advantage of OS scheduling is that you get pretty good results without having to think about it at all. Pretty good,…

> You are essentially asking the operating system to do the scheduling for you. But the OS will never be able to do it perfectly as it has no knowledge of what your application is doing.

From LMAX presentations, it looks like they want you to split your application into tasks [1], define a graph of task dependencies, have each core process a particular kind of task and have task processors communicate their producers via a ring buffer.

In particular, the allocation of tasks is static. The use of a ring buffer means that there is very little contention, and task processing is very efficient, but some cores might end up underutilized.

On the other hand, if you have a thread per request, and allow them to migrate between cores, idle cores can steal tasks from busy ones. So in theory you could get better utilization, but task processing is less efficient since you need to share more data between cores.

("threads" don't need to be OS threads, they can be green threads)

That said, I am not sure if GP meant this by thread-per-request, or "legacy" applications that use a thread pool, or something else.

[1]: https://www.slideshare.net/trishagee/introduction-to-the-dis...

Re: LMAX Disruptor – High Performance Inter-Thread Messaging Library

#37

Post by bnastic »

onetimeuse92304 wrote: post #15
vivzkestrel wrote: post #10

stupid question: how to build a trading system? anyone got a starter guide, resources?

I built a PoC of a 5us trading system (guaranteed 5us response in every situation) for a brokerage house a long time ago, around the time of LMAX Disruptor. It was one man job and I had to start with nothing (they had no knowledge at all). Fun project and I learned a lot. * full kernel bypass (I even implemented driver for the networking hardware) * everything that could disrupt the application disabled (like SME int…

> the main insight was that rather than wait for market signals to then decide what to do, you can precalculate your responses up to and including the actual message to be sent to the exchange

Ah those nasty market opens in the morning and trying to get a good spot in the queue

Re: LMAX Disruptor – High Performance Inter-Thread Messaging Library

#38

Post by bitcharmer »

onetimeuse92304 wrote: post #15
vivzkestrel wrote: post #10

stupid question: how to build a trading system? anyone got a starter guide, resources?

I built a PoC of a 5us trading system (guaranteed 5us response in every situation) for a brokerage house a long time ago, around the time of LMAX Disruptor. It was one man job and I had to start with nothing (they had no knowledge at all). Fun project and I learned a lot. * full kernel bypass (I even implemented driver for the networking hardware) * everything that could disrupt the application disabled (like SME int…

Appreciate you describing this as a PoC because in reality it's impossible to do In fact, you can't guarantee 5us for anything, at least not on common operating systems. You would have to run your code with the interrupt flag cleared to prevent any IPIs or hrticks getting in the way. But that would be opening a scary can of worms.

Re: LMAX Disruptor – High Performance Inter-Thread Messaging Library

#39

Post by dajt »

onetimeuse92304 wrote: post #33
waynesonfire wrote: post #31

Earlier quoted context omitted.

> implemented with Lisp emitting super efficient machine code I'm really intrigued by your usage of Lisp. It's on my bucket list to learn and your post is very inspiring. When you say Lisp was emitting machine code; are you referring to the machine code the Lisp compiler emitted for your Lisp application or was your Lisp application acting as a sort of JIT compiler and actually emitting machine code? Maybe you can re…

Machine code isn't some kind of magical idea. It is just bytes and you can craft the bytes any way you want. In my case I wrote my own compiler that generated machine code bytes based on my needs. The compiler wasn't at all complicated. It was not general compiler that would have to be able to deal with everything you could throw at it. It only supported one model of CPU. It did not require any optimisation mechanism…

How did you inject the machine code into a running program on anything near a modern architecture? Unless you had your own OS, wouldn't code segments be RO and data segments not executable?

I don't know enough about any processor created in the last 30 years to know if running bare metal without a commercial OS would allow you to not have those constraints.

Post reply on HN