How many cases do you know of that people were unable to advance the arts because of a patent?For one concrete example, anyone who worked on data compression outside of purely academic areas has been harmed, and by extension their customers have been harmed, by early software patents in that area. The LZW patent (infamously associated with the .GIF file format) and IBM's patents on arithmetic coding both rendered entire fields of computer science radioactive for years.
It's perhaps even more enlightening to spend some thinking about what life would be like if everything that could be patented under the incredibly loose standards of the USPTO was patented. It's safe to say you wouldn't be able to afford the computer or tablet you're typing on, nor the Internet service necessary to convey your thoughts. But this argument is also moot because neither of those technologies would be available in their present form, at any price.
Your impression is likely based on what you hear in a) tech media, which is largely supported by ads and hence welcome rage-views, and b) tech forums like this where open source is the dominant religion.
My impression is based on decades of work in the industry.
I'll give you some numbers less than 1% of patents are ever asserted
How many land mines ever explode? Does that make land mines a good thing? Patents are intellectual land mines, nothing more, nothing less.