Live data from Hacker News

We chose Java for our high-frequency trading application

medium.com

71–80 of 287 posts

Re: We chose Java for our high-frequency trading application

#71
post #16

Except that HFT firms don't call themselves HFT any more, they re-branded to ultra-low latency trading. And FPGAs are the norm now, not the exception. Sure, Java still has its place, but implying that it is anywhere in the hot loop of a contemporary ULL system is wrong at best and disingenuous at worst. I mean just read the abstracts from this years STAC conference. https://www.stacresearch.com/spring2020

The importance of FPGAs is generally oversold. They definitely have their place, but the problem is due to the complexity you can only evaluate very simple triggers and/or must sacrifice a ton of agility where strategy development is measured in years instead of weeks.

The biggest manifestation of this is that FPGAs really struggle with sizing. Say there's a given market event that produces a very obvious opportunity, but it's not so obvious how big that opportunity is. Archetypical case is when the price changes, and there's a new level formed on the book.

It's almost certainly the case that adding an epsilon of liquidity at the first position of the queue is profitable. What's harder to determine is where the breakeven point is. Should you add 1 lot? 100 lots? 10,000 lots? That's a lot more complex because you generally have to be sensitive to the typical sizes in that particular market and that particular time, as well as how strong the level formation event was. E.g. was it in response to something that looks like a very big order, or just some random liquidity depletion.

What happens is that the FPGAs know they should quote an epsilon, but can't really confidently quote anything bigger than that. So if the right answer is a thousand lots, the FPGA will quickly capture first queue position with something like a hundred lots, then the software-based players with take the rest of the 900 shares for themselves.

What you see with FPGAs is a segment that makes very consistent, very good money relative to the volume that it executes. But fails to capture the sizable majority of the market share.

Re: We chose Java for our high-frequency trading application

#72
post #26

My friend spent two years working for a large bank as a (his words) "trader's bitch". Every day the trader would request a new feature and my friend had at most two hours to implement it. They used a combination of Python and highly GC-tweaked Java. This was six years ago, but I see that similar techniques are still used to this day.

Damn, seems like a good way to have a ton of buggy code in production.

Re: We chose Java for our high-frequency trading application

#73

I think I actually saw these folks present at JavaOne a couple years ago? Either that or there's more than one shop branding itself as "HFT" that uses Java. I worked in the industry and it's always a little funny to see who calls themselves HFTs vs quants. Basically, there's a bit of a spectrum of fast vs smart. In general it's hard to do incredibly smart stuff fast enough to compete in the "speed-critical" bucket of…

That's right, the more to the fast end you are, the more your strategy is obvious. I worked at an HFT where it was literally buy-here-sell-there for one of the strategies, and of course you can't do that without being ridiculously fast.

DSquare is reasonably famous in London, somehow everyone knows them. They are on the smarter end of things, because the founders were plugged into the early development of the electronic FX market at the beginning of this century.

There's a sort of engineering-vs-business culture thing to this as well, but I think it blurs over time.

Re: We chose Java for our high-frequency trading application

#74

I think I actually saw these folks present at JavaOne a couple years ago? Either that or there's more than one shop branding itself as "HFT" that uses Java. I worked in the industry and it's always a little funny to see who calls themselves HFTs vs quants. Basically, there's a bit of a spectrum of fast vs smart. In general it's hard to do incredibly smart stuff fast enough to compete in the "speed-critical" bucket of…

I discovered something amazing when working with some people who were writing HFT software.

Why do you need 1TB of RAM in these machines? Because when you're Java based, you want to avoid stop-the-world GC pauses. These trading systems only have to be up from 9:30AM-4:30PM EST, so they simply disable GC altogether! At the end of a trading day, restart the app or reboot the system.

Re: We chose Java for our high-frequency trading application

#75

I think I actually saw these folks present at JavaOne a couple years ago? Either that or there's more than one shop branding itself as "HFT" that uses Java. I worked in the industry and it's always a little funny to see who calls themselves HFTs vs quants. Basically, there's a bit of a spectrum of fast vs smart. In general it's hard to do incredibly smart stuff fast enough to compete in the "speed-critical" bucket of…

You're making good points, but one thing I want to emphasize is that latency is not the sole dimension of competition in HFT space.

Certain things may just be "table-stakes", but for many strategies table-stakes is the only requirement. Simply being "fast enough" may be fine if you have a smarter model, exploit niche opportunities overlooked by others, or are willing to shoulder certain risks that other HFTs are trying to offload.

To take this to the extreme, look at Renaissance's Medallion fund. It's certainly the case that much of what they're doing is "HFT-ish" in the sense that they're executing high turnover, high Sharpe strategies with short holding periods. Yet they've managed to continue to be successful from the 90s well until the age of HFT, without competing very hard on latency at all.

Markets are an ecosystem, and like biological ecosystems there isn't just one trait that predicts dominance.

Re: We chose Java for our high-frequency trading application

#76
post #7

There's a major historically-contingent component to this, too. The Island ECN, later absorbed into NASDAQ's ECN, originally ran its matching engine as one of these low-latency Java processes, writing everything using explicit object pools to avoid doing any GC on the critical path, as this was long before Zing was around. This was a pretty reasonable choice at the time, as this was long before C++11 was around, and…

I bet they used a bit of sun.misc.Unsafe too https://blogs.oracle.com/javamagazine/the-unsafe-class-unsaf...

Less than you'd think – it comes up in serializing/deserializing data, somewhat, because JNI has a fair bit of overhead, but this isn't really like Rust's "unsafe". To have fast Java, you really need the JIT to be happy, and that's not a safe bet if you're doing random things with Unsafe.

Re: We chose Java for our high-frequency trading application

#77
post #7

There's a major historically-contingent component to this, too. The Island ECN, later absorbed into NASDAQ's ECN, originally ran its matching engine as one of these low-latency Java processes, writing everything using explicit object pools to avoid doing any GC on the critical path, as this was long before Zing was around. This was a pretty reasonable choice at the time, as this was long before C++11 was around, and…

And before Java, it used Microsoft FoxPro! Here's the source code:

https://josh.com/notes/island-ecn-10th-birthday/ https://josh.com/notes/island-ecn-10th-birthday/ISLAND.PRG.T...

Re: We chose Java for our high-frequency trading application

#78
post #63

I have a question: When i finished my theoretical Physics PhD, HFT was THE way out of academia for making money in my field. I didn't follow this path as it felt at the time a pretty evil thing to do. Does HFT provide any benefits for our society?

No, but there are many things that don't provide any net value that is even more prolific. All the sugary drinks that just give you health issues, the big monopolistic companies that replaced smaller decent-wage shops with big low-wage machines that suck the value out and doles it out at the top. If everything is designed to take value from one place and never give back, is that beneficial? Of course not. If you only look at the product, you see only half the picture.

Evil is everywhere, really. Our brains are not made to really understand these things well. The easiest thing is to resign yourself to at least make you and your familys life easier. Which is acceptable enough.

Re: We chose Java for our high-frequency trading application

#79
post #30
post #16

Except that HFT firms don't call themselves HFT any more, they re-branded to ultra-low latency trading. And FPGAs are the norm now, not the exception. Sure, Java still has its place, but implying that it is anywhere in the hot loop of a contemporary ULL system is wrong at best and disingenuous at worst. I mean just read the abstracts from this years STAC conference. https://www.stacresearch.com/spring2020

It's an entire industry devoted to wasting enormous energy, resources and intellectual capacity. There are simply no material external gains from HFT etc, it's completely zero-sum other than maybe a 'job creation program'. The site ridiculously talks about 'levelling the playing field' by extending HFT opportunities to other players, which is rich, considering the playing field could be 'levelled immediately' by some…

It helps with price discovery just like human traders do. Is there something inherently better about droves of people shouting into telephones?

Re: We chose Java for our high-frequency trading application

#80
post #2

I've used Java and C++ in a few low latency applications and I do prefer Java in certain scenarios. The whole ecosystem around Java makes rapid development easier meaning we could make safe changes quicker which is a big benefit in trading systems where you need to react to unpredictable market conditions. The article focuses quite heavily on Zing vs Hotspot but it'd be interesting to see an analysis of a variety of…

where I've worked you avoid triggering the GC by making sure you have enough memory for the entire day's operations

with a modern JVM you can even disable it entirely (Epsilon collector)

Post reply on HN