Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We chose Java for our high-frequency trading application

medium.com

Post reply on HN
Print view View original

121–130 of 287 posts

Re: We chose Java for our high-frequency trading application

#121

Post by benjaminjackman »

oppositelock wrote: post #74
cashewchoo wrote: post #5

I think I actually saw these folks present at JavaOne a couple years ago? Either that or there's more than one shop branding itself as "HFT" that uses Java. I worked in the industry and it's always a little funny to see who calls themselves HFTs vs quants. Basically, there's a bit of a spectrum of fast vs smart. In general it's hard to do incredibly smart stuff fast enough to compete in the "speed-critical" bucket of…

I discovered something amazing when working with some people who were writing HFT software. Why do you need 1TB of RAM in these machines? Because when you're Java based, you want to avoid stop-the-world GC pauses. These trading systems only have to be up from 9:30AM-4:30PM EST, so they simply disable GC altogether! At the end of a trading day, restart the app or reboot the system.

Speaking from experience with JVM HFT applications (we used Scala).

There are a lot of tricks though to not require 1TB.

And allocation in general is a bad idea even if you don't collect because it scatters stuff all over memory and messes up cache locality. You really, really don't want to allocate in a performance sensitive jvm application if you can avoid it. It's the opposite of a lot of what I was told and taught (e.g. never do object pooling), but empirically, in my experience, allocations are the biggest slowdown. You can get an application a lot faster just by opening up the memory allocation tab in a jmc flightrecording and refactoring the biggest allocators, usually there is a lot of easy to optimize low hanging fruit that will give good performance improvements, even better than focusing on hot spots in code (in my personal experience).

By far the biggest allocator in trading is going to be marketdata and calculations on it. For reading marketdata from the exchange it's best to leave raw data in memory and access it with a ByteBuffer / sun.misc.unsafe. Under this pattern classes have 1 value, the memory address to pass into sun.misc.unsafe, then everything from there on is done with offsets onto that address. For calculations it's better to write things as static functions, or use object pooling.

In the course of optimizing a trading engine I wrote lots and lots of code to get allocations down to zero. It's definitely doable, but best done from the start, I refactored an existing trading engine to do that, it was not very fun.

Re: We chose Java for our high-frequency trading application

#122

Post by lostcolony »

jariel wrote: post #50
posnet wrote: post #41

Earlier quoted context omitted.

Better or worse than Ad tech based companies? Most traders I've met have been fairly honest about contributing nothing to society, or that the world would probably be a better place is large segments of the finance industry disappeared tomorrow. I can't say the same for the majority of developers from Facebook or Google I've met.

??? I run ads and campaigns for my company and they are an absolutely essential form of communication. ... as almost everyone running and actual business knows. Yes - it can be very inefficient - but it's not inherently wasteful. You're making the wrong comparison - the 'Digital Ad Industry' could be compared to 'Digital Banking Industry' perhaps - i.e. useful overall but deficiencies exist. HFT is one of the ares in…

Actual campaigns can be good, or bad, and aren't really being referenced here.

The ad tech in question is the issue; the data collected, the platform itself. Think Facebook - in pursuit of creating a platform and getting data that allows for very specific targeting, they have created bubbles that magnify extremist messages.

Re: We chose Java for our high-frequency trading application

#123

Post by zarkov99 »

otabdeveloper4 wrote: post #95
zarkov99 wrote: post #49

Earlier quoted context omitted.

That is not un an entirely unfair characterization. However in my mind HFT practicioners are model citizens when compared to so many of the bright young minds working in the worlds largest tech companies. HFT is a neutral force in the world, while Ad driven tech is causing harm at a scale that is hard to overstate. History will judge the companies profiting from manipulating the worst in us in a much harsher light th…

Complaining about advertising is like complaining about money. Business absolutely cannot exist without advertising. The need must be filled one way or another. The same cannot be said about HFT and stock markets in general.

Its not the need for advertising. Its the unreasonable effectivness and devastating side-effects of the attention sequestration techniques developed in the last 10 years. Its not only that years of our of lives are being directly stollen by vapid bullshit, its that the effects of this manipulation extend well beyond the actual time spent engaging with it, with absolutely horrific consequences to our mental health, our coesion as a people and ultimately our physical safety. In a generation, if not earlier, the ad bussiness will be looked upon in a worst light than the tabaco industry. And this will be entirely appropriate.

Re: We chose Java for our high-frequency trading application

#124

Post by pak9rabid »

winrid wrote: post #66
pak9rabid wrote: post #51

Speaking about JVM garbage collection, why isn't automatic reference counting being used instead of traditional garbage collection? It seems kind of like the best of both words in a automatic memory-managed environment of providing safety as well as performance. Languages like Objective-C and Swift have proven that this is very viable. I wonder what's keeping the JVM from adopting it?

Because allocation is slower with just reference counting, and the JVM is all about throughput. The downside is predictability, but newer memory managers are almost fully concurrent so soon it may not matter.

Interesting. Is the slower allocation due to it having to increase a reference count somewhere, or is there more to it than that?

Re: We chose Java for our high-frequency trading application

#125

Post by xyst »

didip wrote: post #113

This is off tangent, I am surprised that the largest of FAANGM companies aren't trying to muscle into HFT or Quant industries. It seems right on their wheelhouse: highly technical and makes a lot of money.

FAANG isn't interested in entering a highly regulated industry.

Re: We chose Java for our high-frequency trading application

#126

Post by proverbialbunny »

I'm surprised this article doesn't talk about the common issue with writing a trading bot in Java, which is the floating point types. I worked on a trading bot in Java over a decade ago when it was common in the industry to do so. We wrote a fixed precision decimal library which was about 4x faster than the alternative BigDecimal type. It worked out quite well, but Java doesn't allow operator overloading, so the whole code base looked like a mess with tons of a.mul(b.add(c)) looking lines of code.

When C++14 came out the project was ported to C++. It needed to go fast, but more importantly it needed to be stable and readable. An error in code could cost us, which wasn't something we could accept. Moving to modern C++ wasn't fun, but the code is clean, easy to read, works and works well.

Today, the code base has started to be ported to Rust, but unlike going from Java to C++, this move isn't as big of a deal, but the added safety is appreciated given the nature of the project.

Today it's pretty common for firms to create their own lisp like or Haskell like programming language. By having it FPP it can mirror mathematics a bit closer which reduces bugs, yet by having it in house it can be fast, otherwise these companies would use a common language. We are the only ones I know of moving towards Rust. I imagine a lot of firms who are aready C++ are going to stay with modern C++.

Re: We chose Java for our high-frequency trading application

#127

Post by eshyong »

mumblemumble wrote: post #35

One bit of useful background knowledge: The technical demands for high frequency trading can vary wildly depending on both what kind of trading strategies you're using, and what market you're in. This translates into real-time needs that vary considerably, depending on context. At one firm I used to work at, the spread was several orders of magnitude. On one end, people were counting nanoseconds, and even C++ wasn't…

What language was the team counting in nanoseconds using?

Re: We chose Java for our high-frequency trading application

#128

Post by unethical_ban »

traK6Dcm wrote: post #82
immmmmm wrote: post #63

I have a question: When i finished my theoretical Physics PhD, HFT was THE way out of academia for making money in my field. I didn't follow this path as it felt at the time a pretty evil thing to do. Does HFT provide any benefits for our society?

Some people argue that HFT provides liquidity for retail investors, but it's debatable whether that liquidity is real or not since it'll be gone during black swan events. At the same time HFT profit from uninformed/retail flow. So it's debatable wether the actual activity provides values. I'd say probably not. But there can be indirect value in working in HFT, just like there is with other demanding jobs. There's int…

I think that's getting a little high minded for the purpose of the OP's question. I think it's fair to interpret the question as "Is there a good faith argument that HFT is beneficial to markets in a way that also benefits everyday citizens? Are there positive downstream effects? Is the intent of HFT strictly selfish?"

I don't think your comparison to academia holds up there.

Re: We chose Java for our high-frequency trading application

#129

Post by gpsx »

traK6Dcm wrote: post #82
immmmmm wrote: post #63

I have a question: When i finished my theoretical Physics PhD, HFT was THE way out of academia for making money in my field. I didn't follow this path as it felt at the time a pretty evil thing to do. Does HFT provide any benefits for our society?

Some people argue that HFT provides liquidity for retail investors, but it's debatable whether that liquidity is real or not since it'll be gone during black swan events. At the same time HFT profit from uninformed/retail flow. So it's debatable wether the actual activity provides values. I'd say probably not. But there can be indirect value in working in HFT, just like there is with other demanding jobs. There's int…

I'm not an economics or finance expert, so I may not have a very extensive view of this, but I think there is a meaning to the expression of providing value. If a product has more value to a person than the price of the product, assuming they have correct knowledge, that is the creation of value. In high frequency trading, or stock trading in general, people may be willing to pay more than the listed price, but I don't think value is created because this is generally due to incorrect knowledge. I think stocks are pretty much a zero sum game. If one person is making money, someone else is losing money.

As I mentioned, I'm not an economics or finance expert, so I'd be interested to hear what more experienced people would have to say.

Re: We chose Java for our high-frequency trading application

#130

Post by pacoverdi »

CraigJPerry wrote: post #24

Reminds me of one of the best blogs i ever subscribed to, now defunct unfortunately https://mechanical-sympathy.blogspot.com/

This Google group is the offspring of this blog and is still very much alive: https://groups.google.com/g/mechanical-sympathy
Post reply on HN