Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The LMAX Architecture - 100K TPS at Less than 1ms Latency

martinfowler.com

Post reply on HN
Print view View original

31–40 of 57 posts

Re: The LMAX Architecture - 100K TPS at Less than 1ms Latency

#31

Post by rickmode »

haberman wrote: post #24

The article starts by saying that low latency is the main requirement for financial trading, but then quotes all performance numbers in throughput ("6 million orders per second"). I couldn't find any mention of average or worst-case latency, and throughput numbers alone tell you nothing about latency. In particular, since each input event has to be journaled and replicated (which both involve I/O) before it can be pr…

You're saying this approach doesn't fit problem domains that are CPU-intensive. That's a valid point.

The article doesn't directly address this but does compare LMAX to a typical database backed business application. The computation here is not usually CPU-bound.

Re: The LMAX Architecture - 100K TPS at Less than 1ms Latency

#32

Post by jasonwatkinspdx »

vilya wrote: post #15

It's clear from Fowler's discussion of "mechanical sympathy" that he didn't understand the memory hierarchy. I found it pretty shocking that someone in his position could not know this stuff. He really needs to read Ulrich Drepper's "What every programmer should know about memory" ( http://www.akkadia.org/drepper/cpumemory.pdf ). It's such a fundamental topic that everyone ought to at least know the basics of it. Peo…

He states pretty explicitly that he doesn't feel he has an intuitive understanding of hardware's performance quirks right in that section. Why should you then be surprised?

Re: The LMAX Architecture - 100K TPS at Less than 1ms Latency

#33

Post by redcircle »

haberman wrote: post #24

The article starts by saying that low latency is the main requirement for financial trading, but then quotes all performance numbers in throughput ("6 million orders per second"). I couldn't find any mention of average or worst-case latency, and throughput numbers alone tell you nothing about latency. In particular, since each input event has to be journaled and replicated (which both involve I/O) before it can be pr…

They don't call the system low latency --- it is the processing that occurs in the single-threaded main loop that has to be low latency. They of course want the end-to-end system to be low latency, but the term here describes the inner work loop, because their approach doesn't work if you have higher latency work items in the main loop.

If you apply "low latency" to the inner loop only, then you can resolve your second critique too: they won't be supporting anything that is CPU-intensive (since that isn't low latency). Also, all state has to fit in a single node, to keep things low latency.

Re: The LMAX Architecture - 100K TPS at Less than 1ms Latency

#34

Post by mbucc »

I have been using event-based persistence, and while performance is definitely a bonus, the win for me is the ability to replay the transaction log.

For example, let's say you log each page view and include the referer in that transaction. Initially, the only state you maintain (in memory) is the count of page views.

Three months after your system is on-line, you decide you want to take a closer look at who is sending you traffic. So, you update the business logic that processes the pageview_transaction to count page views by referer, perhaps a count of domains by day (for up to 365 days). Replay the log and you will now have that state for the entire history of your app.

I expect there will be benefits for debugging as well, since I can replay the transaction log with a local, annotated version of the app to figure out specific sequence of real-world transactions that surfaced the bug.

And you can use SQL if you like. You just have to keep your transaction processing fast (e.g., by using an in-memory db).

If you are interested, here's a Python implementation: https://github.com/PetrGlad/python-prevayler.

Re: The LMAX Architecture - 100K TPS at Less than 1ms Latency

#38

Post by jbellis »

bodski wrote: post #14

Cliff Click makes a good post [1] discussing the LMAX/Disruptor architecture and how it can give large performance gains in specific circumstances (such as those generally encountered by LMAX). In particular, Disruptor works best when there is a one-to-one ratio of disruptor threads and cpu cores. Great to see Cliff rolling up his sleeves and looking into this stuff from LMAX. Additionally Cliff points out that there…

Too bad there's been essentially zero activity on MRI for over a year.

Re: The LMAX Architecture - 100K TPS at Less than 1ms Latency

#39

Post by vilya »

jasonwatkinspdx wrote: post #32
vilya wrote: post #15

It's clear from Fowler's discussion of "mechanical sympathy" that he didn't understand the memory hierarchy. I found it pretty shocking that someone in his position could not know this stuff. He really needs to read Ulrich Drepper's "What every programmer should know about memory" ( http://www.akkadia.org/drepper/cpumemory.pdf ). It's such a fundamental topic that everyone ought to at least know the basics of it. Peo…

He states pretty explicitly that he doesn't feel he has an intuitive understanding of hardware's performance quirks right in that section. Why should you then be surprised?

Because this is a guy who's supposed to be an authority on software engineering & reading something like that makes me wonder how on earth he ever came to be an authority. It's a bit like someone convincing you they're a racing driver then admitting they're not really sure what the clutch does. (Yes that's hyperbole, but still...)

Re: The LMAX Architecture - 100K TPS at Less than 1ms Latency

#40

Post by vilya »

koevet wrote: post #17
vilya wrote: post #15

It's clear from Fowler's discussion of "mechanical sympathy" that he didn't understand the memory hierarchy. I found it pretty shocking that someone in his position could not know this stuff. He really needs to read Ulrich Drepper's "What every programmer should know about memory" ( http://www.akkadia.org/drepper/cpumemory.pdf ). It's such a fundamental topic that everyone ought to at least know the basics of it. Peo…

Can you be more specific?

About the performance issues?

Well I work on software that does image processing at the moment & we've chosen to represent images as separate channels (i.e, all the red values, then all the green values, then all the blue values, etc). If we had interleaved the channels instead, many common operations that only need to look at one channel - or just one at a time - would be a lot slower.

When data is loaded into the cache, it's loaded in chunks called cache lines. These are usually 64 or 128 bytes long, depending on your CPU. So assuming a 64 byte cache line, if you ask for the value at address 4 then it'll load in all the values in addresses 0-63. If you then ask for the value at address 8, it'll already be cached & therefore quick to access; but if you ask for the value at address 64, it'll have to fill another cache line first - a cache miss.

So back to the images, say we're just looking at the alpha channel of an RGBA image. With separate channels we get 16 alpha values in each cache line (each channel is a float, so 4 bytes). If the channels were interleaved then we'd only get 4 alpha values in each cache line, so the CPU will have to fill 4 times as many cache lines.

Because so much of our code deals with images (and not just ours - our clients too), if we'd chosen an interleaved channel representation and were now finding that too slow, we'd be pretty stuck. So it's really important to consider these issues up front, but of course you can't if you don't know at least a little bit about how the hardware works.

Hopefully that explains it a bit better?

Post reply on HN