If I understood correctly, the whole damn scam is only possible because this Venmo's terms of service prohibit something called "merchant transactions" and allow for reversing or cancelling transfers if they deem or someone reports a payment to fall in "merchant" category?
If so, then the problem is only a completely artificial one and made of mere policy decisions. Venmo could just act as a blind intermediary, no questions asked, completing any transactions once both parties have verified their intention to transfer money.
That's generally how banks work for at least any reasonable sums of money: I can send or receive thousands of euros to whoever and from whoever without worrying that the transfers might be cancelled by some vague policy. With tens of thousands, I'd have to ask my bank to lift the daily transfer cap beforehand, but still no questions asked. I assume that it goes in a relatively similar way in the USA.
Thus, any mobile money transfer service ought to be at least as trustworthy as that. There needs to be a point where the transaction is either complete or discarded, after which both parties can trust it. The service also shouldn't discriminate, it shouldn't matter if it's a private transfer or a merchant transfer. If they want the money they could sell transaction history imports for bookkeeping at a higher price to merchants or something.
Venmo could still do their instant transfer magic and add value there, but them considering whether a transfer is ok or not is akin to an internet service provider who would want to consider which web pages I can view before letting me download them.