>> Most of the reason chemical weapons don't get used is because they're not very effective.
Against a modern military equipped and prepared to withstand them, perhaps. Against a civilian population or a less well-funded force, not so much. This is doubly the case when the attacker is a non-state agent (like a terrorist). Remember for instance the Sarin gas attack in the Tokyo metro by the Shinning Path [1]. An air force bombing a major city with chemical weapons would also cause vast casualties.
I think what you're trying to say is that chemical weapons are primarily designed to be terrifying with lethality considerations coming next. However, you could argue the same about nuclear weapons. After all, the attacks in Hiroshima and Nagasaki were not acutally meant to wipe the Japanese apart, only to scare them into submission. And nuclear deterrence works on the principle that an enemy won't dare attack a nuclear power.
In general, you could think of all weapons ever as aimed primarily at the morale of the enemy rather than their life and that thought may have some merit. After all, an army that breaks and runs is beaten faster and with fewer losses to the other side than one that fights to the last [2].
However, the important point to keep in mind is that weapons, even when they're primarily designed to be terrifying, are designed to be terrifying in the manner they kill. So I think you'll find that most people would agree that chemical weapons should be banned because they kill in a horrible manner, not relative to the number of casualties they may or may not inflict.
I do think you underestimate their potency btw.
__________________________
[1] https://en.wikipedia.org/wiki/Tokyo_subway_sarin_attack
[2] Cough cough. In war games, certainly.