The problem is non security minded users probably won't take the time to verify the numbers in the first place. They should look into a method that verifies identities without user interaction. Possibly by having signal store users public keys after they verify their phone numbers or better yet provide users with signed certificates for verification purposes.
Congratulations, you've invented the iMessage security model.
The cryptographers behind signal are very smart; they have thought about these things.
The signal model has the advantage of not having to trust a third party server. With iMessage, it's possible for the government to force Apple to add pubkeys to a user's identity (although the user will be notified).