The debate is very complex, with many people discussing orthogonal issues.
Some are the cypherpunks that want security at all costs. They see Bitcoin as a failure unless just about anyone can operate the network. They already see mining as a failure because it's too centralized, and are staunchly opposed to anything that might increase that centralization.
Ironically, I think increasing the block size would provide advantages to miners outside of the China firewall, which may have the effect of decentralizing mining a little. That's a very simple take on it, though, and only one of many factors.
Interestingly, many who hold this view argue that changes can only be made to the network through strict consensus, ultimately relying on very centralized development.
Others see Bitcoin as a currency that should be used at the consumer level, like cash. They want a network that grows and is predictable for investors. The possibility of expensive transactions is dangerous to business models and will stall adoption.
There are some focused on the technology, and what can be built on top of bitcoin. Recognizing the network effect (more users, more value), they see using Bitcoin as a settlement layer for the various implementations on top of that. Some of these require some changes to the Bitcoin protocol, and there's much controversy in what is permissible. Lightning Network, side chains, etc. could all operate at different frequencies of Bitcoin settlement transactions. Other services would like to store information on the blockchain, using a small transaction fee to pay for immutable information storage. This could be used for contracts of all sorts. Some would see this is spam on the network, and others a huge innovation and source of value.
Some will look to the early development for guidance. Satoshi wrote a lot about how the network might evolve, and it's quite clear that he thought network capacity would improve with technology. Basically, some use the argument that the 1MB limit was a stopgap, temporary limit that has passed its usefulness. The 'Bitcoin people signed up for' is one with a growing block size.
You can add to that concerns about many of the core developers working at Blockstream, so might have perverse incentives to cripple Bitcoin to generate demand for their products.
It's ... a little complex.