I'm not deaf, I only studied ASL in HS and it included some useful lessons on Capital-D Deaf Culture. So I'm not much more of an expert on the subject by any means, but I love language studies.
Efficiency is only one spectrum in play. Sign languages are languages distinct and evolutionarily unique from spoken languages. Early attempts such as "Signed English" attempted to minimize the distinction between the spoken language and signed language, but living languages evolve to fit their medium, and sign languages evolved some truly unique grammars and even their own words that don't always reflect relatives in nearby spoken languages.
Even though sign languages for the most part haven't adopted their own written languages that better reflect their grammar and uniqueness (and there's really interesting arguments on both sides whether they should), there's an increasing difference between a sign language and its "written language neighbor". One particularly fun example is that American Sign Language (ASL) is much closer to French Sign Language (FSL) on the language family tree than ASL is to British Sign Language (BSL). Despite what you might presume from the spoken language "neighborhoods" of the sign languages, an ASL signer is going to have an easier conversation with an FSL signer than a BSL signer.
Which leads to the fact that there are even native speakers whose first and primary language in their lives is a sign language, and with a culture to match (Capital-D Deaf Culture). Written English will always be a foreign language to them, even if it is a second language they have a lot of reason to study from early in life.
So it's a quality of life question as much as or more than just an efficiency question. (Just as with most language issues, really. We could probably find a more efficient use of language for English medical/scientific jargon than smashing lots of random bits of Latin and Greek together. But we have what we have for a lot of familiarity and cultural reasons.)