Insider trading by, say, having direct access to the particular wind vane that grounds the market and being able to manipulate it is a problem. Being better at trading because you have better models or better computers isn't per se a problem. Any more than it's a problem if someone is a better poker player.
In a broader context this may contribute to income inequality and regulatory capture, it can have negative externalities. But it isn't on it's face a problem. And the alleged positive externalities (realtime forecasts and a liquid market for hedging) are driven by such skilled traders.
Not all prediction markets are equally easy to manipulate. The bar was supposed to be (according to my hazy recollection) that a market had to be run in collaboration with an outside authority who, along with delivering a verdict that was already part of their normal duties, would help you exclude insiders from the market. That's a defensible position (not morally, tactically).
But "will so and so tweet about such and such" is a total joke. That's indefensible. That I do think that should never have been allowed or should be regulated as gambling. If there's no conceivable natural hedger, it isn't a real financial market and doesn't have a function (or the function is gambling, whichever).
Don't get me wrong, I think this turned into an end run around gambling regulations (that's certainly what happened with PredictIt), but I don't think that was always inevitable or will necessarily always be the case. That's a strong attractor for sure but it was pushed that way by the COVID era meme stock frenzy, the deregulation of online gambling, and ultimately competitive pressure from the sketchier markets. Once those forces were aligned it was inevitable, but it's possible that in 10 years there will have been a crackdown and tightening down of regulation and things will be very different. I'm not that optimistic, but it's a real possibility.