Which "we" are you speaking for here? I live and work in Manhattan and I definitely am negatively affected by all the vehicle congestion, along with everyone I know. The vast majority of us here don't own or drive cars, and having fewer of them around will be much better for walking, biking, and taking the bus.
You can't possibly deny that the intent of congestion charges is to reduce congestion. That's been the intent (and effect) in every other major city it's been implemented in across the world. The simple fact of the matter is that single occupancy vehicles can't possibly scale to meet all the transit demands of an urban area as dense as Manhattan, so they need to be deprioritized in favor of more scalable modes of transport. Reclaiming car lanes for dedicated bus lanes and cycle lanes would be a great first start.
Frankly, this doesn't go nearly far enough. All of the free on-street parking in this zone needs to be removed as well, with people paying the true cost of storing a vehicle on the most valuable property in the US. That'll get rid of a lot of the vehicles on the streets and we can reclaim that space for much more productive uses. Many of these vehicles only move on alternate side parking days anyway, so there's really no reason they should be here at all. Charge the $500+/month that space is actually worth and they'll be gone.