Still reading the comments, but is anyone actually saying passwords are more secure in general? I think most people are saying that they get locked out of their account if they lose the token. How do you validate your account is your account if the only secret you have is lost? If you 100% require a hardware token, you need at least two and/or a method to auth that is not a hardware token so you can recover in a mode where you lost one.
But on the security note. There are several types of security. Security from the people directly around you and security from everyone on the internet.
Without any malware, I could pickup my friends keys and log into their account on my computer in seconds and without their knowledge. This is harder to do with a password. At the same time a password is much easier for someone who doesn't even know who I am to attack.
It's why 2FA is so necessary, it helps defend against both methods of attack.