Intruder at the top of the 20 meter amateur band?

If prices are out of line, that definitely seems like a place where automated trading would be valuable. But HFTs' tech advantage isn't just about finding a better arbitrage algorithm or being smarter: it's also quite clearly about exploiting a pure technical advantage over other traders. The fact that HFTs are so willing to invest in speed even to achieve a tiny advantage over other HFTs kind of gives the game away.

And when people quite rightly observe that having a speed advantage over other traders obviously allows extractive behavior like frontrunning, a bunch of people on HN come out with irate counter-takes that claim HFT are an unalloyed good -- never something nuanced like, "yes HFT could allow for some extractive behavior, but it's counterbalanced by these advantages which I will explain in detail." (And the corollary, which is an explanation of "why a world where all traders have the same speed advantages wouldn't have all the claimed advantages of HFT but be even more efficient.")

I guess it's also worth pointing out that these responses are usually from people who are involved in the HFT industry in some way, and usually they start out by accusing people of "not understanding the industry". Which is precisely the accusation many make against HFT: that it's so deliberately opaque that people outside the industry can't possibly determine how much extraction there is compared to value being added. Saying "trust us" or "you couldn't possibly know because you aren't on the inside profiting from it" is not the compelling argument you think it is.

Why suspect HFTs over other potential interfering applications? Could be anything, and unless I missed something, TFA doesn't show any evidence that the usage is HFT-motivated

Exchanges generally like (and provide incentives to) orders that add liquidity, which means orders that do not execute immediately. HFT orders are generally intended to not execute immediately, so exchanges would tend to like them. Of course, HFT also usually pays higher connection fees to get closer to the exchange, which probably doesn't hurt their relations with the exchange.

The dark pools that HFT firms setup to trade with retail investors are probably less enjoyed by the exchanges, but at the same time, it does reduce load at the exchange, so maybe it's mixed because it reduces the exchanges' income from commissions, but also reduces their expense from operations.

Padding orders by a random amount doesn't really discourage racing to get their first; if it's a uniform random, across all orders, you still want to get there first, which means you still want to get your order there quickly; and depending on implementation, you might break your order up into many smaller orders, so that some of them get there with less delay. If you wanted to disincentivize speed, what you really want to do is group orders that arrive in a given interval and treat them as arriving at the same time; you could do that with fixed intervals (1 minute, 1 second, 100 ms, whatever) or adaptive intervals based on the number of orders over the last interval, or randomly, if you like random.

From what I've seen from selling employer stock over the years, the commissions have gone down (to zero), and the bid/offer spreads have gone down, and both are due to HFT firms, so as far as I can see, HFT is helping me. I can't say I've noticed a difference in execution speed, trading always seemed pretty fast, as long as I was able to setup the order via the browser.

Speed and frontrunning are not equivalent. Frontrunning is an illegal practice. Here is an article from FINRA stating as much [0]. Of course, people are always looking for ways to accomplish this. Hell, sometimes the exchanges try to sell it as a feature [1]. Again, no one debates that these types of practices should be banned and penalized (they already are). I'm not inside "the industry" so I don't know if there are some firms that make profit from illicit means by abusing bugs and/or other loopholes where speed matters. However, speed is always an advantage when there are multiple exchanges that sell inter-related products. The same reason why computers are used instead of telephones.

You mention that HFT firms are willing to invest to beat other HFT firms gives the game away. We are both in agreement here but come to different conclusions. The arbitrage "algorithm" is simple in most cases. Futures contracts being sold in Chicago are at odds with equity prices in NYC. The first one to correct them wins. Speed is the only thing that matters. How exactly is people competing in the race "give the game away"?

[0] - https://www.finra.org/investors/insights/getting-speed-high-... [1] - https://www.sec.gov/news/press-release/2012-2012-189htm

Both yes and no. For most exchanges:

HF traders are big data consumers and they _have_ to have all the orderbook daa, so they're big customers of exchanges. They also pay for stuff like colocation.

Many exchanges also provide rebates for liquidty providing orders, which HF traders are incentivised to use.

One notable exception is the Investors Exchange (IEX), are vocally anti-HF.