Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LMAX Disruptor – High Performance Inter-Thread Messaging Library

lmax-exchange.github.io

Post reply on HN
Print view View original

71–80 of 89 posts

Re: LMAX Disruptor – High Performance Inter-Thread Messaging Library

#71

Post by lowbloodsugar »

xoranth wrote: post #52
onetimeuse92304 wrote: post #42

Earlier quoted context omitted.

I know all about LMAX architecture, at least all that has been published (see my other comments for this submission). Static allocation is a special case of scheduling. You decide which parts of the process run on which core -- the scheduling in this case is done at design or configuration time. > On the other hand, if you have a thread per request, and allow them to migrate between cores, idle cores can steal tasks…

> For example, in a typical event driven architecture where you have worker threads running each on separate core, there would be something to decide where the task is queued and usually the logic will take into account how busy particular worker thread is. Wouldn't that work well only if the time taken by each task is predictable? I.e. you mention working on a trading system. But in a trading system you want to run…

I feel like you are stuck on the idea of “this should be simple for me” which, of course, favors the OS threads solution. If your point is that LMAX inst just a drop in substitute for OS threads then we agree. If your point is that OS threads produce better results than a thought out LMAX solution then we do not. Most people and organizations probably don’t have the need or the skill for LMAX anyway.

Re: LMAX Disruptor – High Performance Inter-Thread Messaging Library

#72

Post by mgaunard »

eatonphil wrote: post #69
mgaunard wrote: post #63

I built trading systems for LMAX exchanges. Their technology seems quite far from the state of the art to me. I didn't know they even claimed to attempt being the fastest exchange in the world. They're very far from being so and it's quite clear that there are architectural decisions in that platform that would prevent that.

Where should one look/read about for state of the art?

In terms of trading exchanges, I'd say the ones with the best deterministic low-latency appear to be the Deutsche Boerse T7 ones, in particular Eurex.

This has led participants wishing to compete there on speed to use some pretty advanced custom ASICs.

Re: LMAX Disruptor – High Performance Inter-Thread Messaging Library

#74

Post by pixelmonkey »

Martin Fowler has a lovely deep-dive blog post on this architecture:

https://martinfowler.com/articles/lmax.html

It includes lots of diagrams and citations.

One term I always loved re: LMAX is “mechanical sympathy.” Covered in this section:

https://martinfowler.com/articles/lmax.html#QueuesAndTheirLa...

Re: LMAX Disruptor – High Performance Inter-Thread Messaging Library

#75

Post by gpderetta »

mgaunard wrote: post #70
quadrature wrote: post #64

Earlier quoted context omitted.

What kinds of issues did you see?. Do you think there are better alternatives to the disruptor ?

I don't particularly know anything about this disruptor, and it being in Java kinda biases me towards dismissing it out of hand (no one does serious systems programming in Java). From a quick reading it's just a standard spmc queue with some mpmc capabilities. Queues (preferably lock-free and bounded) are a basic component of any low-latency distributed software system. They seem to understand the basics right, nothi…

It is a broadcast queue. Consumers all receive the same data, so they don't contend to pop elements. And producers never wait for consumers. Slow consumers have to deal with missing packets.

At the end of the day, it is a specialised ring buffer that happens to be useful for many use cases.

Re: LMAX Disruptor – High Performance Inter-Thread Messaging Library

#76

Post by pjmlp »

onetimeuse92304 wrote: post #65
twic wrote: post #53

Earlier quoted context omitted.

My team did it in Rust. Others in our company have done it in C++, making heavy use of available wizardry. It's widespread in the industry to do it in Java, albeit a strange subset of Java where you avoid most of the features. I'd be amazed if anyone did it entirely in C. The productivity is just too low.

The question really is "why?" Most systems have relatively few hotspots or can be reorganised to have relatively few hotspots. You don't need to write everything in a low level language to just have those few things run fast, unless you are also facing other constraints like low memory. I have better uses of my time than write megabytes of pointer manipulation in C. That's why I prefer high level languages that can e…

So are the wonders of polyglot programming, too many devs are too focused on the one language for everything.

Since 2006 that the projects I work on follow a similar approach, Java, C# and nowadays node as well, coupled with C++ when required.

Re: LMAX Disruptor – High Performance Inter-Thread Messaging Library

#77

Post by pjmlp »

marginalia_nu wrote: post #50
vivzkestrel wrote: post #18

Earlier quoted context omitted.

what language do you think would be the backbone for such a system? C/C++/Golang or something high level like node.js/Java

I wouldn't bundle C with Golang and Java with Node. Golang and Java are roughly a tier on their own. Java is definitely a thing in this space[1], although it's a fairly unidiomatic style of Java that reduces allocations and puts a lot of emphasis on consistent low latency. [1] e.g. https://marketswiki.com/wiki/TRADExpress ; though they've been absorbed into Nasdaq now and my visibility into CFT ended with that, no id…

Hopefully Panama will help to make it better, and there is still hope for Valhala.

Re: LMAX Disruptor – High Performance Inter-Thread Messaging Library

#78

Post by gpderetta »

vbezhenar wrote: post #51
onetimeuse92304 wrote: post #15

Earlier quoted context omitted.

I built a PoC of a 5us trading system (guaranteed 5us response in every situation) for a brokerage house a long time ago, around the time of LMAX Disruptor. It was one man job and I had to start with nothing (they had no knowledge at all). Fun project and I learned a lot. * full kernel bypass (I even implemented driver for the networking hardware) * everything that could disrupt the application disabled (like SME int…

Why do you need OS for that kind of project? If you're implementing network drivers and avoiding scheduler, you could just run your code on bare metal.

Because it is convenient being able to SSH to your trading machine and run standard Unix applications for deployments, normal start of day/end of day processing, diagnostics, profiling, etc. The overhead, during normal operations can be significantly reduced.

Re: LMAX Disruptor – High Performance Inter-Thread Messaging Library

#79

Post by gpderetta »

onetimeuse92304 wrote: post #65
twic wrote: post #53

Earlier quoted context omitted.

My team did it in Rust. Others in our company have done it in C++, making heavy use of available wizardry. It's widespread in the industry to do it in Java, albeit a strange subset of Java where you avoid most of the features. I'd be amazed if anyone did it entirely in C. The productivity is just too low.

The question really is "why?" Most systems have relatively few hotspots or can be reorganised to have relatively few hotspots. You don't need to write everything in a low level language to just have those few things run fast, unless you are also facing other constraints like low memory. I have better uses of my time than write megabytes of pointer manipulation in C. That's why I prefer high level languages that can e…

A trading system doesn't have a single hot spot that can be easily optimized. The critic paths can be hundreds of thousands of line of code.

There a lot of separate processes and subsystem that do not need to be absolutely latency optimal (or even at all), that are definitely amenable to be written on other languages. Of course when you already have a bunch of C++ programmes, things tend to end up being written in C++ even if it is not strictly necessary.

Re: LMAX Disruptor – High Performance Inter-Thread Messaging Library

#80

Post by mgaunard »

gpderetta wrote: post #75
mgaunard wrote: post #70

Earlier quoted context omitted.

I don't particularly know anything about this disruptor, and it being in Java kinda biases me towards dismissing it out of hand (no one does serious systems programming in Java). From a quick reading it's just a standard spmc queue with some mpmc capabilities. Queues (preferably lock-free and bounded) are a basic component of any low-latency distributed software system. They seem to understand the basics right, nothi…

It is a broadcast queue. Consumers all receive the same data, so they don't contend to pop elements. And producers never wait for consumers. Slow consumers have to deal with missing packets. At the end of the day, it is a specialised ring buffer that happens to be useful for many use cases.

That just sounds like a normal spmc queue as I said above.

Of course they receive the same elements, or it wouldn't be multi-consumer.

Of course producers don't wait for consumers, they don't even need to be aware of how many there are or where they are. But even in a system where you'd know (e.g. publishing data to a bunch a TCP connections), it would he a very bad idea to stall production -- handling back pressure should be application-specific.

You have the same elements in UDP multicast which is the network equivalent, and incidentally the preferred technology for communication in the trading industry, particularly for market data disseminated by exchanges.

Post reply on HN