I like the self awareness here but I kind of cringed at some of the points made by the author. LMAX is definitely a counter to the one size/approach/design/trend fits all thinking in software. I guess its true when they say execution is everything. In the end, a great example of a well executed innovative design.
The LMAX Architecture - 100K TPS at Less than 1ms Latency
41–50 of 57 posts
Re: The LMAX Architecture - 100K TPS at Less than 1ms Latency
#42This is great stuff, thanks very much for posting this.
This must be the only place on the web where saying thank you gets you pissed on. I'm going back to lurking, can't figure this place out. Please downvote my "karma" to zero & adios.
Re: The LMAX Architecture - 100K TPS at Less than 1ms Latency
#43This is great stuff, thanks very much for posting this.
This must be the only place on the web where saying thank you gets you pissed on. I'm going back to lurking, can't figure this place out. Please downvote my "karma" to zero & adios.
If everyone said "thank you" to every post, those would swamp any meaningful discussion. Think about it. On this article alone there would be over 100 such comments.
Generally, you can give a +1 by upvoting. If you have something to add to the discussion, you can add a thank you in that comment, but in isolation a "thank you" really adds no value.
Re: The LMAX Architecture - 100K TPS at Less than 1ms Latency
#44This is great stuff, thanks very much for posting this.
This must be the only place on the web where saying thank you gets you pissed on. I'm going back to lurking, can't figure this place out. Please downvote my "karma" to zero & adios.
You say thank you by voting up the URL or comment. Literally saying "thank you", "I agree", or stuff like "+1" is frowned upon because it doesn't add to the discussion (see http://ycombinator.com/newsguidelines.html).
Re: The LMAX Architecture - 100K TPS at Less than 1ms Latency
#45Re: The LMAX Architecture - 100K TPS at Less than 1ms Latency
#46Earlier quoted context omitted.
He states pretty explicitly that he doesn't feel he has an intuitive understanding of hardware's performance quirks right in that section. Why should you then be surprised?
Because this is a guy who's supposed to be an authority on software engineering & reading something like that makes me wonder how on earth he ever came to be an authority. It's a bit like someone convincing you they're a racing driver then admitting they're not really sure what the clutch does. (Yes that's hyperbole, but still...)
To abuse your analogy there are many fast drivers that have little to know mechanical knowledge.
You seem to be under the impression that to be an authority one must have comprehensive and detailed knowledge, particularly in areas you care about. I believe this is rarely the case. It's far more important to understand the essential. The key to being a good leader is not to know everything better than everyone, but to understand how to help everyone contribute their best, then get out of the way.
Re: The LMAX Architecture - 100K TPS at Less than 1ms Latency
#47Earlier quoted context omitted.
Because this is a guy who's supposed to be an authority on software engineering & reading something like that makes me wonder how on earth he ever came to be an authority. It's a bit like someone convincing you they're a racing driver then admitting they're not really sure what the clutch does. (Yes that's hyperbole, but still...)
To abuse your analogy there are many fast drivers that have little to know mechanical knowledge. You seem to be under the impression that to be an authority one must have comprehensive and detailed knowledge, particularly in areas you care about. I believe this is rarely the case. It's far more important to understand the essential. The key to being a good leader is not to know everything better than everyone, but to…
> You seem to be under the impression that to be an authority one must have comprehensive and detailed knowledge, particularly in areas you care about.
Detailed and comprehensive knowledge is exactly what I expect from an authority on a subject. If they don't have it, then on what basis are they considered an authority?
It sounds to me - and I apologise if I've misunderstood - like you are arguing that a software architect doesn't need to understand the performance impact of their designs to be considered an authority on their subject? If so, then I respectfully disagree. It's possible to be a (bad) software architect without understanding that stuff, but I expect better from people who are supposed to be experts in the field.
> The key to being a good leader is not to know everything better than everyone, but to understand how to help everyone contribute their best, then get out of the way.
That's all very well but we're not talking about being a leader, we're talking about being an expert.
Re: The LMAX Architecture - 100K TPS at Less than 1ms Latency
#48The article starts by saying that low latency is the main requirement for financial trading, but then quotes all performance numbers in throughput ("6 million orders per second"). I couldn't find any mention of average or worst-case latency, and throughput numbers alone tell you nothing about latency. In particular, since each input event has to be journaled and replicated (which both involve I/O) before it can be pr…
The LMAX system is more about latency than throughput. When designing for very low-latency the result can be a system that achieves very great throughput if the appropriate techniques are employed. Single threaded applications are very suitable for low-latency because of the avoidance of lock contention and predictability they bring.
Some means of reliable delivery of messages, to and from this single thread, is necessary to make a useful application. These messages must be delivered in the event of system failures. To address this need the Disruptor is employed to pipeline and run in parallel, the replication, journalling and business logic for these messages. The whole system is asynchronous and non-blocking.
In our architecture we have multiple gateway nodes that handle border security and protocol translation to and from our internal IP multi-cast binary protocol for delivery to highly redundant nodes. Lots of external connections can be multiplexed down from the outside world this way.
The 6 million TPS refers to our matching engine business logic event processor. We have other business logic processors for things like risk management and account functions. These can all communicate via our guaranteed message delivery system that can survive node failures and restarts, even across data centres.
Modern financial exchanges can process over 100K TPS and have to respond with latency in the 100s of microseconds firewall to firewall, thus including the entire internal infrastructure. For those tracking the latest developments will see it is possible to have single digit microsecond network hops with IP multicast with user space network stacks and RDMA over 10GigE. Even a well tuned 1GigE stack can achieve sub 40us for a network hop. For reference single digit microseconds is in the same space as a context switch on a lock with the kernel arbitrating. Most financial exchanges rely on having data on multiple nodes before the transaction is secure. A number of these nodes can be asynchronously journalling the data down to disk. At LMAX we tend to have data in 3 or more nodes at any given time.
In my experience of profiling many business applications the vast majority of the time is either spent in protocol translation such as XML or JSON to business objects, or within the JDBC driver doing buffer copying and waiting on the database to respond, when the application domain is well modelled.
Often applications are not well modelled for their domain. This can result in algorithms that, rather than be O(1) for most transactions, have horrible scale up characteristics because of inappropriate collections representing relationships. If you have the luxury of developing an in-memory application requiring high performance it quickly becomes apparent the cost of a CPU cache miss is the biggest limitation to latency and throughput. For this one needs to employ data structures that exhibit good mechanical sympathy for CPU and memory subsystem. At LMAX we have replaced most of the JDK collections with our own that are cache friendly and garbage free.
So far we have had no issue processing all transactions for a given purpose on a single thread, or holding all the live state in memory for a single node. If we ever cannot process all the transactions necessary on a single thread then we simply shard the model across threads/execution contexts. We only hold the live mutating data in-memory and archive out to database completed transactions as they are then read only.
Martin (LMAX CTO)
Re: The LMAX Architecture - 100K TPS at Less than 1ms Latency
#49As usual, it depends on what you define as a "transaction". If you don't need ACID, I can probably jerry-rig something in C that'll give you hundreds of thousands of "transactions" per second. It may have problems with retrieving the data, but hey, look at that sucker go! OK. Time to be more reasonable. 1. Relaxed Guarantees There's no actual durability in the conventional spinning-rust sense. Instead you're hoping t…
All transactions processed by the LMAX system are ACID. The application is modelled such that each input event represents one transaction. Where business transaction span multiple nodes, rather than handle the complexity of retaining ACID properties across multiple messages, our reliable messaging system ensures delivery to a remote system and queuing a message is included in the same transaction context as the business logic. This is true of services that use a high speed journal (event-sourced) and those that perform more traditional database operations. Long running transactions become steps in a state model.
Re: The LMAX Architecture - 100K TPS at Less than 1ms Latency
#50it seems to me that you need something else that i didn't see described in the article (perhaps i missed it?): the output disruptor has to be able to recognise and discard duplicate requests for output. otherwise, when you replay the input to a new logic processor (after the old one died) you're going to get repeat outputs. is that right or have i misunderstood something else? it seems like it places additional const…
The business logic simply outputs state changes that other systems subscribe to. If the subscribing system misses the update our reliable message delivery will ensure the message gets to the subscribing system and processed. Our system guarantees message processing and not just delivery.
Queries against the business domain model are just input events. The business logic will serialise the requested part of the model and publish it out. This type of query can easily be handled by any number of replica nodes.
All messages in the system carry a sequence number so duplicate messages can be detected and ignored.