This seems to be partly a technological problem. We will soon need secured, authenticated modes of communication that can verify a person’s voice. A similar problem is emerging for photos and videos. We also soon need cryptographically signed devices in order to be used in journalism or to be admissible in court. Otherwise we are going back 150 years where we depend on in-person communication and eyewitness accounts.
Scammers aren't looking to defeat or even challenge voice identification. They're looking for that one person who's having a bad day and is susceptible to getting tricked. All they need is to find that person to earn their quota for the day. They'd actually appreciate it if 99% of the population used Voice Supr-Sure-Auth 3000™ technology, because that would make it more efficient for them to reach the 1% who don't.
This is why the Nigerian prince emails have typos. They're not trying to convince you their email is authentic. They're trying to find the person who isn't sophisticated enough to think in terms of email authenticity.