Easier said then done. Who's "we" here? The handful of idealists that actually understand the legal issues? Or think they do?
The main issue with patents is not that they exist but that the system for filing and enforcing them has been eroded to the point where it has become a highly effective tool for big corporations to enforce patents they arguably should not have received against smaller competitors. That's technically illegal but since it is super hard to argue that in a court, companies get away with anti competitive behavior. And of course they are actively supported in this by bought and payed for politicians. In the US that's called lobbying, in much of the rest of the world we'd label that corruption, nepotism, and graft. Not a good thing. But as old as power and politics is.
The issue is not patents but the corruption around it.
The system in the US is flooded with bull shit patents that would not be acceptable else where. A lot of those have issues with their claims being overly broad, prior art, or a complete and utter lack of novelty. This does not matter. Getting patents rubber stamped is a numbers game. You just send more patents in and eventually a few will get approved. Of course, it costs a money to register and keep patents but once you have them the system is rigged in your favor. Challenging a patent is super expensive relative to filing one. And since patent offices are under staffed, they bias in favor of approving. And also they don't like admitting they were wrong to approve.
None of this happened by accident. The system was lobbied into the shape it is in today by the largest patent holders in the US (i.e. every big mega corporation you can name). They use patents as a tool to keep competitors out of the market. And they do so very successfully. To the extent that this is becoming a problem.
The solution is fixing the system, not getting rid of it. A big motivation for this could simply be that patents are not globally enforceable. So, you might convince companies in the US to not compete but e.g. Chinese companies will happily take your patented thing and run with it. Good luck complaining about that in a Chinese court of law. And they seem to have bootstrapped a pretty nice high tech industry in recent decades there. Bad patents actually erode the competitive position of US based corporations. While they are tied up in courts being sued by patent trolls, their foreign competitors are free to compete without concern for such things. Great argument for getting politicians off their ass. The problem exists to a lesser extent in other jurisdictions of course. But the US is the most paralyzed by this.