Its a very intersting issue and sadly the debate is always very limited.
Fundamentally it makes absolute sense to have different priorities for different packets. Just like in any logistics you have different priorities for different application. In logistics usually the costumer demands some level of services and if he pays the needed amount he can get what he wants. Most people don't think this is bad. So fundamentally there is nothing wrong.
The IP protocol has this idea backed in already, and inside controlled organisation this is already pretty standard.
The problem happens when you have ISP who without their costumers input make arbitrary choices about the importance of packets. Not only because they don't really know what my needs are also because it requires the ISP to open up each package and analyse what is inside.
Even that might still be OK if there were a truly competitive market in ISPs. If one ISP does not suit you you can move on, enforcing discipline in the whole system. That would still be problematic because applications would then have to make specialised deals with the ISP to get high quality service for specific application rather then making that choice when you send a package.
In practice however ISP for historical, regulatory and economic reason are in a monopoly position allowing them sit in the middle and make money on both sides. This primary what Net Neutrality tries to fix. Providers that are in these positions, like Comcast, would like nothing more then to have Net Neutrality removed. Providers that are in a highly competitive market have far less to gain.
Even worse, then strictly about net neutrality, once the technology is in place ISP also start to act in other ways, for example injecting advertisements into your websites, breaking TLS and so on.
So really I am against Net Neutrality in general, but I would still not have removed it in the current situation.
The solution for me is actually a different one. Quite simply, ISP should not be allowed to open up your packages, just like the post office is also not allowed to do so. They should be allowed to sell you different quality of service in whatever pricing model they wish but it is the costumer who makes the choice about what he wants to do.
Monopoly would still be an issue in that case the potential damage would be way lower.