I think at this point the benefits of HFT to markets (lower spreads, more liquidity, faster incorporation of information into prices) is pretty undisputed. I'll let you do your own research, but just to address your point about spreads:
When you participate in the market, the spread is the "price" you have to pay to transact. When this goes down, it benefits all participants in the market, and especially the ones that are doing "long-term capital allocation" you are talking about. Rather than take my word for it, here's a quote from the CEO of Vanguard: “From a data perspective, we can see what’s happened to our fund shareholders over the last 20 years, and they’ve benefited by that reduction in transaction costs.” [1]. If you're managing trillions of dollars of 401ks, pension funds, etc, and you are constantly rebalancing your assets, buying new allocations, etc, any tiny reduction in average spread is a huge savings on net. A big part of the reason why there's been an almost universal reduction in fund management fees, saving retirement savers an enormous amount of money over the last 10 years or so is this reduction in spreads. This is absolutely a huge benefit to society, and its almost entirely attributable to HFTs.
You're correct that it HFT is a zero-sum arms race, and maybe you could make the argument that in an optimal allocation of society's resources, perhaps you could have less than the current number of participants. But I think you could actually make this argument about almost all lucrative & highly competitive fields. If anything, HFT is more productive per unit of labor, relative to other sectors of society - that's why the compensation is so high!
The total number of programmers and quants that work in HFT, at least for the 5-10 significant players, is probably less than the number of programmers Google alone employs (last I checked, there are about 30k programmers working at google. There's almost certainly less than 30k quants and programmers working at the main HFT firms). How many programmers do you "need" to optimize ads for eyeballs? Or how many programmers do you "need" to make mobile phone games? Or work on social networks? etc etc.
HFT is a small, niche, industry, and I think extremely productive per employee relative to most other industries. It replaced the tens of thousands of manual traders that used to be responsible for arbitrage and market making with automated robots, dramatically increasing market efficiencies while reducing the amount of human capital required to provide those services. It seems pretty misguided to make the argument that HFT is somehow "bad" or a "waste" of resources, given how much of an improvement it was to what there was before, and also given how small the industry really is.
[1] https://www.cnbc.com/2014/04/25/vanguard-chief-defends-high-...