Why are we still building high frequency trading apps ? Wasn’t it part of the old world we meant to change ?
We chose Java for our high-frequency trading application
101–110 of 287 posts
Re: We chose Java for our high-frequency trading application
#102There's a major historically-contingent component to this, too. The Island ECN, later absorbed into NASDAQ's ECN, originally ran its matching engine as one of these low-latency Java processes, writing everything using explicit object pools to avoid doing any GC on the critical path, as this was long before Zing was around. This was a pretty reasonable choice at the time, as this was long before C++11 was around, and…
And before Java, it used Microsoft FoxPro! Here's the source code: https://josh.com/notes/island-ecn-10th-birthday/ https://josh.com/notes/island-ecn-10th-birthday/ISLAND.PRG.T...
More color on the original FoxPro matching engine:
"Levine found a simple but hugely consequential 'trick' to speed matching. In Levine’s later paraphrase, what the 'enter2order' procedure did when the Island system received a new order was to: 'See if there was a record from a recently cancelled order that we can reuse for this new order. This is hugely important because that record will likely still be in the cache [fast internal memory] and using it will be much faster than making a new one. (Levine, n.d.)'"
http://www.sps.ed.ac.uk/__data/assets/pdf_file/0003/97500/Is...
Re: We chose Java for our high-frequency trading application
#103I have a question: When i finished my theoretical Physics PhD, HFT was THE way out of academia for making money in my field. I didn't follow this path as it felt at the time a pretty evil thing to do. Does HFT provide any benefits for our society?
High margin businesses rarely if ever provide (equitable) benefits to someone other than shareholders, but the innovations they create may/usually trickle down to the rest of us over time, one way or another.
Re: We chose Java for our high-frequency trading application
#104I think I actually saw these folks present at JavaOne a couple years ago? Either that or there's more than one shop branding itself as "HFT" that uses Java. I worked in the industry and it's always a little funny to see who calls themselves HFTs vs quants. Basically, there's a bit of a spectrum of fast vs smart. In general it's hard to do incredibly smart stuff fast enough to compete in the "speed-critical" bucket of…
I discovered something amazing when working with some people who were writing HFT software. Why do you need 1TB of RAM in these machines? Because when you're Java based, you want to avoid stop-the-world GC pauses. These trading systems only have to be up from 9:30AM-4:30PM EST, so they simply disable GC altogether! At the end of a trading day, restart the app or reboot the system.
This is similar to how missiles don't need GC.
https://devblogs.microsoft.com/oldnewthing/20180228-00/?p=98...
Re: We chose Java for our high-frequency trading application
#105I think I actually saw these folks present at JavaOne a couple years ago? Either that or there's more than one shop branding itself as "HFT" that uses Java. I worked in the industry and it's always a little funny to see who calls themselves HFTs vs quants. Basically, there's a bit of a spectrum of fast vs smart. In general it's hard to do incredibly smart stuff fast enough to compete in the "speed-critical" bucket of…
I discovered something amazing when working with some people who were writing HFT software. Why do you need 1TB of RAM in these machines? Because when you're Java based, you want to avoid stop-the-world GC pauses. These trading systems only have to be up from 9:30AM-4:30PM EST, so they simply disable GC altogether! At the end of a trading day, restart the app or reboot the system.
I guess what you mean is the no-op garbage collector which is available in Java 11 http://openjdk.java.net/jeps/318
Even this isn't fail proof right ?
Last-drop latency improvements. For ultra-latency-sensitive applications, where developers are conscious about memory allocations and know the application memory footprint exactly, or even have (almost) completely garbage-free applications, accepting the GC cycle might be a design issue. There are also cases when restarting the JVM -- letting load balancers figure out failover -- is sometimes a better recovery strategy than accepting a GC cycle. In those applications, long GC cycle may be considered the wrong thing to do, because that prolongs the detection of the failure, and ultimately delays recovery.
So essentially you need to know the memory footprint of your applications. To err on the side of caution just get as much memory that is available in the market.
At this point like another reply mentions here aren't you just better off writing C++ code ?
Re: We chose Java for our high-frequency trading application
#106Earlier quoted context omitted.
??? I run ads and campaigns for my company and they are an absolutely essential form of communication. ... as almost everyone running and actual business knows. Yes - it can be very inefficient - but it's not inherently wasteful. You're making the wrong comparison - the 'Digital Ad Industry' could be compared to 'Digital Banking Industry' perhaps - i.e. useful overall but deficiencies exist. HFT is one of the ares in…
What sort of targeting do you use for your campaigns, and how privacy-violating are those targetings?
If you have to ask the question I would encourage you to take $50, open a FB ad account, same for Google, and run some ads to drive some traffic to your blog or whatever.
Running an ad campaign, or having to 'get the word out' for something possibly meaningful can be an enlightening and transformative experience for so many people who live adjacent to, but otherwise 'totally outside' the ostensibly opaque world of marketing.
It's one of the most stupidly represented issues in pop culture, like 'reefer madness' memes form the 1960s' where 'smoking weed will make you gay!'. Well - just 'smoke weed' once and you realize it's frankly just not that big of a deal.
Ads are ancient, normal, universal. The industry has serious efficiency problems, we're starting to face privacy issues (which are overstated but nevertheless legit), and I don't like some mediums (I for one, think we should dump most billboards) - but the nature of the industry is not wrong or evil, it's actually beneifical.
'Good ads' are a win for everyone. HFT is only a win for those directly involved and it's zero sum.
Re: We chose Java for our high-frequency trading application
#107Re: We chose Java for our high-frequency trading application
#108> This is why, according to Oracle, Java is the #1 programming language, with millions of developers and more than 51 billion Java Virtual Machines worldwide.
Oof this didn't need to be added to the end of the article.
Re: We chose Java for our high-frequency trading application
#109Earlier quoted context omitted.
I discovered something amazing when working with some people who were writing HFT software. Why do you need 1TB of RAM in these machines? Because when you're Java based, you want to avoid stop-the-world GC pauses. These trading systems only have to be up from 9:30AM-4:30PM EST, so they simply disable GC altogether! At the end of a trading day, restart the app or reboot the system.
Even if you completely disable GC, Java is still allocating tons of ephemeral objects on the heap. Which in turn is leading to expensive and unpredictable page faults. In contrast, C++'s default is to allocate objects on the heap unless you knowingly call new/malloc. Of course, it's possible to write Java in such a way to minimize this type of heap-thrashing. But by that point, you're already doing the equivalent of…
That's covered in Java by escape analysis and allocation on the stack.
Re: We chose Java for our high-frequency trading application
#110I may be acting overly pessimistic, but given the arms race like nature of HFT this could be intentional disinformation.