Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Using Reinforcement Learning in the Algorithmic Trading Problem

arxiv.org

Post reply on HN
Print view View original

51–60 of 139 posts

Re: Using Reinforcement Learning in the Algorithmic Trading Problem

#51

Post by MasterScrat »

fnbr wrote: post #15

An additional problem with this is that they use A3C here for trading. A3C is known to not be suitable for adversarial environments (e.g. board games, like Chess). I wrote a paper that demonstrated that A3C is as exploitable as a uniform random strategy in board games (specifically, some poker variants): https://arxiv.org/abs/2004.09677 (Exploitable is a technical term that is defined in the paper; basically, it's "h…

> A3C is known to not be suitable for adversarial environments

Interesting! What are the main papers in this area?

Any intuition why this is the case? is it because A2C generally results in brittle policies?

Re: Using Reinforcement Learning in the Algorithmic Trading Problem

#52

Post by twic »

traderthrow454 wrote: post #36
amiga_500 wrote: post #34

Earlier quoted context omitted.

I'm certain HFT vols are up in the volatility. Not sure that proves or disproves anything. HFT is hugely expensive to maintain. Staffing and equipment costs are massive. HFT choose not to trade on exchanges with delays because they want to exploit latency advantages. That's why govt should regulate this on all exchanges to just wipe it out. Consolidation is not good because it will collapse into a monopoly. Which wil…

> I'm certain HFT vols are up in the volatility. Not sure that proves or disproves anything. You said that HFTs jump out when things get volatile. But then I show you evidence that they trade more in high volatility. Do you not see the contradiction? > HFT is hugely expensive to maintain. Staffing and equipment costs are massive. HFT firm equipment costs are a few racks of high end servers and some expensive networki…

> You said that HFTs jump out when things get volatile. But then I show you evidence that they trade more in high volatility. Do you not see the contradiction?

We should distinguish between volatility at different timescales. An HFT might well be very active over a volatile month, but may still turn off over a very volatile second. I've always assumed that the "HFTs jump out during volatility" complaint was about the latter; it means that when some shocking news hits the market, the HFT firms providing most of the liquidity pull it, and so the manually-entered market orders wanging around end up moving the market further, exacerbating the volatility. Maybe that's not what people are actually complaining about, though.

> HFT firm equipment costs are a few racks of high end servers and some expensive networking equipment

And FPGAs. FPGA development is really not cheap.

Re: Using Reinforcement Learning in the Algorithmic Trading Problem

#53

Post by yorwba »

nightski wrote: post #22
Qworg wrote: post #12

Earlier quoted context omitted.

Trading is inherently zero sum.

That's only true in the sense of opportunity cost. I may buy something at $10 and sell it at $15 making a $5 profit. Then it may go to $20. Did I lose $5/share? Sure. But in reality I wasn't a "loser". I find that in reality opportunity cost rarely matters.

If you buy something at $10 and sell it at $15, where are the $5 profit coming from? From the other market participants, e.g. someone selling to you for $10 and later buying it back for $15, losing $5 in the process. Your profit and their loss sum to zero, which is what "zero-sum" means.

It has absolutely nothing to do with opportunity cost, or whether you, personally, are a "loser". But if you're a "winner", someone else must be the "loser".

Re: Using Reinforcement Learning in the Algorithmic Trading Problem

#54

Post by idohft »

amiga_500 wrote: post #27
traderthrow454 wrote: post #24

Earlier quoted context omitted.

I think thats an overly simplistic view of things. The market is big and many participants trade at different frequencies. Large pension funds need liquidity to move big blocks of stock for their quarterly and monthly rebalances, and the big medium term statistical arbitrage traders provide liquidity for them to do so. HFT players provide liquidity for the stat arb players. The classes of participants with different…

I knew someone would come in with "liquidity". Many HFT jump out when things get volatile, when liquidity is actually required. Ultimately HFT is doing nothing of societal value, the race down to zero is never-ending and we are wasting huge amounts of resources on a totally pointless march towards zero. Exchanges should introduce random delays to allow market participants who really want to hedge / buy / sell, then w…

You were replied to, but I'm going to ask some questions of this moralizing.

> Many HFT jump out when things get volatile, when liquidity is actually required.

This feels almost like a "no true Scotsman" situation. Why is liquidity not "actually required" when volatility is low? Is it a moral obligation for any trader to catch a falling knife? I see this condition of "when liquidity is actually required", but I never understood why there was such a strong feeling for it. Why do you believe this?

> Ultimately HFT is doing nothing of societal value, the race down to zero is never-ending and we are wasting huge amounts of resources on a totally pointless march towards zero.

I don't know, I could probably take a similar view of so many jobs in tech. What does society really get from Snapchat, what do they get from HQ Trivia, what do they get from people making powerpoint presentations with arrows that point to synergies. What's the point of any job with some amount of abstraction?

> Exchanges should introduce random delays to allow market participants who really want to hedge / buy / sell, then we can shift some of the resources to the real world.

Why?

> The system is hugely inefficient

Do you know how efficient the system was before HFT started up? And, do you know how many people were working in trading before, and how many are, for a similar fraction of stock volume?

> The law of diminishing returns.

OK.

Re: Using Reinforcement Learning in the Algorithmic Trading Problem

#55

Post by traK6Dcm »

yorwba wrote: post #53
nightski wrote: post #22

Earlier quoted context omitted.

That's only true in the sense of opportunity cost. I may buy something at $10 and sell it at $15 making a $5 profit. Then it may go to $20. Did I lose $5/share? Sure. But in reality I wasn't a "loser". I find that in reality opportunity cost rarely matters.

If you buy something at $10 and sell it at $15, where are the $5 profit coming from? From the other market participants, e.g. someone selling to you for $10 and later buying it back for $15, losing $5 in the process. Your profit and their loss sum to zero, which is what "zero-sum" means. It has absolutely nothing to do with opportunity cost, or whether you, personally, are a "loser". But if you're a "winner", someone…

In this simple example, yes, but you are assuming that monetary value = utility. That's not always the case. People have all kinds of different incentives for participating in the markets.

Let's say I am a market maker offering to buy Apple shares at $99 and sell them at $100. Let's take an ex-Apple employee who owns some shares. He just had a family emergency and wants to liquidate his shares to get cash, and he needs it quickly. He doesn't care about paying a few dollars extra in exchange for a quick trade because he needs to pay a bill tomorrow. I buy his shares for $99. He is happy because he immediately got his cash.

On the other side, there is a a retail investor doing long-term investment and wants to add Apple to their portfolio. They also don't care about a few cents because they're holding the stock for a decade and love the new CEO. They buy my Apple shares from me for 100.0. They are happy because I can guarantee them a stable price for a decent number of shares.

All participants are happy. I just made $1 from the spread for providing liquidity, the investor got the long-term investment they wanted, and the ex-Apple employee got his cash.

Sure, both sides of the market could have made more optimal trades if they had put in more effort and "optimized" their trades with algos and somehow skipped the middle-man, but they would've sacrificed convenience and time, which may be worth more to them than the little bit of extra $ they paid. Aren't we all winners?

When you go buy bananas in your grocery store you also don't complain about them taking a cut for providing liquidity. You don't say the farmer has "lost" money because the consumer paid more than what the farmer originally sold for to the grocery store. The farmer is happy because otherwise he may not have traded at all or his bananas may have gone bad (= needs to trade quickly). This is no different.

Re: Using Reinforcement Learning in the Algorithmic Trading Problem

#56

Post by buzzkillington »

vecter wrote: post #29
Qworg wrote: post #12

Earlier quoted context omitted.

Trading is inherently zero sum.

Zero-sum in wealth, but not zero-sum in utility. Otherwise, people wouldn't trade at all.

That's not true, it just needs to cost you more to not play than it does to play.

Something like:

    |            | Play | Don't Play |
    |------------+------+------------|
    | Play       |   -5 |        +10 |
    | Don't Play |  -10 |          0 |

Re: Using Reinforcement Learning in the Algorithmic Trading Problem

#58

Post by amiga_500 »

twic wrote: post #52
traderthrow454 wrote: post #36

Earlier quoted context omitted.

> I'm certain HFT vols are up in the volatility. Not sure that proves or disproves anything. You said that HFTs jump out when things get volatile. But then I show you evidence that they trade more in high volatility. Do you not see the contradiction? > HFT is hugely expensive to maintain. Staffing and equipment costs are massive. HFT firm equipment costs are a few racks of high end servers and some expensive networki…

> You said that HFTs jump out when things get volatile. But then I show you evidence that they trade more in high volatility. Do you not see the contradiction? We should distinguish between volatility at different timescales. An HFT might well be very active over a volatile month, but may still turn off over a very volatile second. I've always assumed that the "HFTs jump out during volatility" complaint was about the…

Exactly, that was my point and I thought that would be understood by someone in the domain. Same for HFT vs "electronic" in general, again someone in the domain would get that distinction immediately.

Re: Using Reinforcement Learning in the Algorithmic Trading Problem

#59

Post by astrofinch »

young_rutabaga wrote: post #44
traderthrow454 wrote: post #20

Earlier quoted context omitted.

If you want to read useful academic papers about trading there is one author in particular who is actually not bad - Zura Kakushadze. Most of his stuff is applicable to mid-frequency trading, not HFT. He worked at WorldQuant (reputable trading firm) and the founder of WQ, Igor Tulchinsky, is a coauthor on one of his papers. Example of a pretty interesting and accessible one - is "101 Formulaic Alphas" [0]. [0] - http…

This paper is a hilarious dump of WQ's randomly generated formulas that (hopefully) happen to pass in-sample test. Alpha#33: rank((-1 * ((1 - (open / close))^1))) This formula trivially reduces to rank(open/close - 1) which is an example of a mean-reversion strategy. But: 1) nobody bothered to simplify this formula, 2) as any mean reversion, it is extremely difficult to trade.

Why is mean reversion difficult to trade?

Re: Using Reinforcement Learning in the Algorithmic Trading Problem

#60

Post by amiga_500 »

twic wrote: post #50
traderthrow454 wrote: post #31

Earlier quoted context omitted.

My initial comment was discussing speculative trading in general, but since you mostly brought up some common anti-HFT tropes I might as well address them. > Many HFT jump out when things get volatile, when liquidity is actually required. Do you have a citation on that? If you look the preliminary Q1 results of Virtu Financial [0] (only publicly traded HFT) they seem to be doing more trading than ever in these volati…

> If you look the preliminary Q1 results of Virtu Financial [0] (only publicly traded HFT) they seem to be doing more trading than ever in these volatile markets. Similar story from Flow Traders: https://www.flowtraders.com/sites/flow-traders/files/quarter...

Everyone is, volumes are hugely up. The point about liquidity is during the sudden market shifts, not over a quarter!
Post reply on HN