Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LMAX Disruptor – High Performance Inter-Thread Messaging Library

lmax-exchange.github.io

Post reply on HN
Print view View original

51–60 of 89 posts

Re: LMAX Disruptor – High Performance Inter-Thread Messaging Library

#51

Post by vbezhenar »

onetimeuse92304 wrote: post #15
vivzkestrel wrote: post #10

stupid question: how to build a trading system? anyone got a starter guide, resources?

I built a PoC of a 5us trading system (guaranteed 5us response in every situation) for a brokerage house a long time ago, around the time of LMAX Disruptor. It was one man job and I had to start with nothing (they had no knowledge at all). Fun project and I learned a lot. * full kernel bypass (I even implemented driver for the networking hardware) * everything that could disrupt the application disabled (like SME int…

Why do you need OS for that kind of project? If you're implementing network drivers and avoiding scheduler, you could just run your code on bare metal.

Re: LMAX Disruptor – High Performance Inter-Thread Messaging Library

#52

Post by xoranth »

onetimeuse92304 wrote: post #42
xoranth wrote: post #36

Earlier quoted context omitted.

> You are essentially asking the operating system to do the scheduling for you. But the OS will never be able to do it perfectly as it has no knowledge of what your application is doing. From LMAX presentations, it looks like they want you to split your application into tasks [1], define a graph of task dependencies, have each core process a particular kind of task and have task processors communicate their producers…

I know all about LMAX architecture, at least all that has been published (see my other comments for this submission). Static allocation is a special case of scheduling. You decide which parts of the process run on which core -- the scheduling in this case is done at design or configuration time. > On the other hand, if you have a thread per request, and allow them to migrate between cores, idle cores can steal tasks…

> For example, in a typical event driven architecture where you have worker threads running each on separate core, there would be something to decide where the task is queued and usually the logic will take into account how busy particular worker thread is.

Wouldn't that work well only if the time taken by each task is predictable? I.e. you mention working on a trading system. But in a trading system you want to run the same branchless code path regardless of the kind of incoming event and whether you are sending an order or not after running the trading logic. So the individual "task" is very predictable.

On the other hand think of a task like "return all the comments for a certain page". The time taken by an individual task is unpredictable, proportional to the number of comments. So you'll regularly get one core getting enqueued a bunch of tasks with no comments, finishing quickly and then staying idle.

With work stealing, after finishing, that core would get a chance at "stealing" tasks from other threads' queues.

(of course, the architecture I am describing would be awful for a trading system)

> The operating system does nothing in this case, what it sees is a number of threads each running on its separate core that don't need to be preempted (hopefully).

Btw, I agree that pinning OS threads to each core and then layering something of your own on top of it is going to be faster. It is just that you can layer on top a green thread system (like Go), and get something thread-for-request -like.

Re: LMAX Disruptor – High Performance Inter-Thread Messaging Library

#53

Post by twic »

defrost wrote: post #19
vivzkestrel wrote: post #18

Earlier quoted context omitted.

what language do you think would be the backbone for such a system? C/C++/Golang or something high level like node.js/Java

Each to their own but if you read and understand the comment above they're describing a dedicated OS for the task .. so think about what you'd choose to write a small task dedicated OS with. Simple C is most likely, ASM is possible, a language such as OCaml generating C to hook into the low level buffers would be intriguing ... the list is long and largely determined by the experience preference of whoever tackles it…

My team did it in Rust. Others in our company have done it in C++, making heavy use of available wizardry. It's widespread in the industry to do it in Java, albeit a strange subset of Java where you avoid most of the features.

I'd be amazed if anyone did it entirely in C. The productivity is just too low.

Re: LMAX Disruptor – High Performance Inter-Thread Messaging Library

#54

Post by twic »

vivzkestrel wrote: post #29
defrost wrote: post #22

Earlier quoted context omitted.

Not my first rodeo :-) I too made a real time hardware level trading system back in the day, on the back of building a multi channel seismic aquisition system with a custom RealtimeOS talking to a bunch of DSP cards that each sampled a trailing flotation cable that each had multiple microphones with the entire grid going toward building up a profile of the seafloor and layers underneath along with bobbing boat|cable…

what would you do if you wanna run all this on a web server instead? sorry if thats the wrong question. how does your system interact with say android or ios clients or a webapp with ui

The low-latency core communicates with a more normal app via messages, over a socket or through shared memory or whatever. So you have normal UI code where a user can enter an instruction, then instead of writing to a database or making a HTTP request, it sends a message to the low latency process. That should respond to acknowledge it, and it will send more messages later if it makes any trades etc.

Re: LMAX Disruptor – High Performance Inter-Thread Messaging Library

#55

Post by twic »

bitcharmer wrote: post #38
onetimeuse92304 wrote: post #15

Earlier quoted context omitted.

I built a PoC of a 5us trading system (guaranteed 5us response in every situation) for a brokerage house a long time ago, around the time of LMAX Disruptor. It was one man job and I had to start with nothing (they had no knowledge at all). Fun project and I learned a lot. * full kernel bypass (I even implemented driver for the networking hardware) * everything that could disrupt the application disabled (like SME int…

Appreciate you describing this as a PoC because in reality it's impossible to do In fact, you can't guarantee 5us for anything, at least not on common operating systems. You would have to run your code with the interrupt flag cleared to prevent any IPIs or hrticks getting in the way. But that would be opening a scary can of worms.

You don't do risk, deal booking, etc in between the tick and the trade. You do them afterwards.

You do drop copy completely separately, that's the whole point of it.

Re: LMAX Disruptor – High Performance Inter-Thread Messaging Library

#56

Post by gmfawcett »

nine_k wrote: post #35
citrin_ru wrote: post #34

Earlier quoted context omitted.

Easy to implement.

But does anyone run an OS thread per request unironically? I thought that nearly every request-response server implementation would use a thread pool. The best, like Erlang, can give you the feeling of arbitrarily many extremely cheap threads, while also running on a thread pool.

> does anyone run an OS thread per request unironically?

Of course they do. There are loads of appropriate applications. Heck, people still run CGI programs unironically.

Re: LMAX Disruptor – High Performance Inter-Thread Messaging Library

#57

Post by vkaku »

I've actually seen this particular library used (and misused and abused). People tried to offload I/O and data heavy tasks on it and was a spectacular fail, with multiple threads getting blocked and people having to frequently adjust it's buffer size and batch size.

One of those things to remember is that Java I/O layering (stuff like JPA) is really terrible. And people in my known Java world tend to prefer the abstractions while the people in the trading world try to use GC-less code (unboxed primitives and byte arrays).

Unless you have verified your E2E I/O to be really fast (possible off heap), you're just pushing a few bytes here and there, your latencies are all in check - this library is not for you. Do all that work first, then use this library.

Re: LMAX Disruptor – High Performance Inter-Thread Messaging Library

#58

Post by SkyArrow »

reisse wrote: post #47
jiggawatts wrote: post #27

Earlier quoted context omitted.

> the main insight was that rather than wait for market signals to then decide what to do, you can precalculate your responses up to and including the actual message to be sent to the exchange. I saw a talk about this dialed up to eleven: the entire processing occurred in a "smart NIC" instead of the CPU. The response would start getting sent even as the inbound packet was still being received. The go/no-go decision…

> The go/no-go decision was effectively just sending the final CRC bytes correctly or deliberately incorrectly, thus invalidating the outbound packet that was already 99% sent. This trick will get you banned on some of exchanges now :) Another one, which is public knowledge for years, and also often penalized, is to send TCP fragment with header of the message well in advance, "booking" place in the queue. Then send…

This bears the question of how does an exchange efficiently detect, log and take action against these kinds of behaviours without increasing its own latency too much and (perhaps?) affecting the market?

Does it even matter if a centralised exchange increases its own latency when all market participants have to go through it? I can only think of the case when a security is listed on multiple exchanges, where the latency could mean a small arbitrage opportunity.

Re: LMAX Disruptor – High Performance Inter-Thread Messaging Library

#59

Post by pjc50 »

colanderman wrote: post #6

I had implemented more-or-less this same concurrency scheme for an IPS/DDoS prevention box ~10 years ago, running on Tilera architecture. It was fast (batching + separating read & write heads really does help a ton)... but not as fast as Tilera's built-in intercore fabric. It had some limitations but was basically a register store/load to access and only like 1 or 2 cycles intercore latency. (Aside, generic atomic op…

Tilera! There's a name I'd not heard in years. They leaned very heavily on providing a very large number of cores with small local storage and having to do all the intercore work yourself.

Re: LMAX Disruptor – High Performance Inter-Thread Messaging Library

#60

Post by kasey_junk »

SkyArrow wrote: post #58
reisse wrote: post #47

Earlier quoted context omitted.

> The go/no-go decision was effectively just sending the final CRC bytes correctly or deliberately incorrectly, thus invalidating the outbound packet that was already 99% sent. This trick will get you banned on some of exchanges now :) Another one, which is public knowledge for years, and also often penalized, is to send TCP fragment with header of the message well in advance, "booking" place in the queue. Then send…

This bears the question of how does an exchange efficiently detect, log and take action against these kinds of behaviours without increasing its own latency too much and (perhaps?) affecting the market? Does it even matter if a centralised exchange increases its own latency when all market participants have to go through it? I can only think of the case when a security is listed on multiple exchanges, where the laten…

Most exchanges have switched to order gateways that are either fpga or asic based.

Also every packet you send to an exchange is trivially attributed. They just kick you off if your shenanigans cause a problem. And then they tell all the other exchanges about you.

Post reply on HN