Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Terrifying A.I. Scam That Uses Your Loved One's Voice

newyorker.com

Post reply on HN
Print view View original

21–30 of 47 posts

Re: The Terrifying A.I. Scam That Uses Your Loved One's Voice

#21

Post by hiddencost »

devjab wrote: post #9

As horrible as this is, I suspect this will be far worse in enterprise companies. We’ve already seen issues with email impersonation where top managers or CEOs order money transfers, and since this works better than it should, we now use confirmation of some sort. Imagine if you’d get both the email and a call, maybe even a video call in the future with the confirmation… Especially CEOs will be very “easy” to imperso…

Japan is an interesting example. My understanding is that Japan relies heavily on the Hanko (a physical stamp), which makes business difficult as agreements need to be stamped in person.

Re: The Terrifying A.I. Scam That Uses Your Loved One's Voice

#23

Post by verisimi »

Well, I feel sorry for real kidnappers. Who's going to take them seriously now, when it is so easy to fake?! :)

In all seriousness, the solution to this and other tech dystopia issues seems obvious to me. It's un-teching things. Having to go to the bank in person, etc. Not believing screens, even corporate or government ones. (We never should have believed those anyway, so this should be a net gain for humanity!)

Re: The Terrifying A.I. Scam That Uses Your Loved One's Voice

#24

Post by codingminds »

BOOSTERHIDROGEN wrote: post #13
devjab wrote: post #9

As horrible as this is, I suspect this will be far worse in enterprise companies. We’ve already seen issues with email impersonation where top managers or CEOs order money transfers, and since this works better than it should, we now use confirmation of some sort. Imagine if you’d get both the email and a call, maybe even a video call in the future with the confirmation… Especially CEOs will be very “easy” to imperso…

Interesting any article about the systems on denmark

I don't know about the danish system, but I guess it's similar to the ones used by Norway and Sweden

https://bankid.no/ (Norway) https://www.bankid.com/ (Sweden)

Re: The Terrifying A.I. Scam That Uses Your Loved One's Voice

#25

Post by godelski »

I wonder if these things will be the breaking point where we start to take security and identity verification seriously. I mean banks are still using SMS 2FA. Will tools like Keybase become well known and used? Yubi keys? Push into homomorphic encryption? Signal? Matrix? Don't get me started on SSNs...

I think one of the unfortunate things about our societies is that we so strongly believe "if it ain't broke, don't fix it." It can be a useful piece of advice, but we should always be looking to improve things. Hell, it's what low level research is about. Just technically everything is broken, just how much. Similarly like "move fast and break things." It can be great but it leaves a wake of destruction and tech debt. Sometimes you go faster by fixing things first instead of hacking around things and compounding.

How I think about it is that it is cheaper to perform maintenance rather than fixing things. Yet it is always easier to push off maintenance because other things are high priority. But in the long run this is far more costly. So we should often be fixing things before they are broken. But like all things, nothing is absolute ;)

Re: The Terrifying A.I. Scam That Uses Your Loved One's Voice

#26

Post by djur »

This scam was being done years before AI voice cloning was possible (it's called a "grandparent scam") and so far I have not seen evidence that such technology is being used. The victims claim it was exactly like their relative's voice, but victims were saying the same thing two decades ago. The attorney mentioned in the article testified that his "son" was sobbing and claimed to have had his nose broken in the accident. Most people are not going to be able to confidently tell apart even a loved one's voice from someone around the same age if they're crying, screaming, being distorted by the phone transmission, etc.

I think it's possible that scams like this could be done with voice cloning, don't get me wrong, but it's not necessary. It's probably good that people are alert to the risk of these scams, regardless.

Re: The Terrifying A.I. Scam That Uses Your Loved One's Voice

#27

Post by idank »

lynx23 wrote: post #16
notTooFarGone wrote: post #12

Anyone believing AI will be a nice thing in the future: With data from social media it's a fucking nightmare - billions of people having their relationships visible to everyone and a singular voice recording/video is enough to scam your friends, family, workplace. I don't see a future of the current communication channels without strong authentication measures - people already get scammed with shitty tactics by billi…

I doubt a single voice recording could generate speech with enough emotional depth that I'd be fooled by a fake call from my better half. Do people really have such a shallow connections to their loved ones that they wouldn't instinctively feel that something isn't right? C'mon, we humans operate with a lot of extra clues. Its not like we just STT and "read" the resulting output.

It doesn't have to work 100% of the time to be a successful scam.

Re: The Terrifying A.I. Scam That Uses Your Loved One's Voice

#28

Post by bboygravity »

LadyCailin wrote: post #10
kombookcha wrote: post #8

Earlier quoted context omitted.

It's insanely cruel and fucked up. Imagining what this is like for the victims goes way, way beyond what calling it a "scam" implies.

It’s actual terrorism. It’s just small beans compared to other things which we usually call terrorism, but that’s what quite a large number of these “scams” are, small scale terrorism.

Ok, that's fine, but then we're going to need a new word to describe actual terrorism. This had/has a meaning that involves specifically targetting governments and destabilizing them.

Re: The Terrifying A.I. Scam That Uses Your Loved One's Voice

#29

Post by godelski »

lynx23 wrote: post #16
notTooFarGone wrote: post #12

Anyone believing AI will be a nice thing in the future: With data from social media it's a fucking nightmare - billions of people having their relationships visible to everyone and a singular voice recording/video is enough to scam your friends, family, workplace. I don't see a future of the current communication channels without strong authentication measures - people already get scammed with shitty tactics by billi…

I doubt a single voice recording could generate speech with enough emotional depth that I'd be fooled by a fake call from my better half. Do people really have such a shallow connections to their loved ones that they wouldn't instinctively feel that something isn't right? C'mon, we humans operate with a lot of extra clues. Its not like we just STT and "read" the resulting output.

You're right to be suspicious but wrong in your interpretation of reality. The issue is that there's a big difference in how you'll interpret things in different environments and stress levels. Under pressure a lot of your normal checks will go out the window. This is why conmen often put you under time pressures, because they're stressing your system. Under normal circumstances, you're prefectly right, but under duress weird things seem normal. I mean duress is abnormal to begin with.

Also enshitification is an important aspect here. We readily overlook things like sharp cuts in video, degraded signals in video and voice, distortion, etc. Because that stuff happens normally. Post hoc it is often easy to see scams and think how dumb someone must have been to fall for something so obvious, but that's actually a tool of the scammers themselves. They love that because the more embarrassed you are, the less likely you will report it due to shame. Be careful with over confidence, because that's what they target.

Re: The Terrifying A.I. Scam That Uses Your Loved One's Voice

#30

Post by godelski »

atleastoptimal wrote: post #17

this is why i believe the only possible route forward for humanity in the AI age is a biometric digital footprint

Biological fingerprints are not that unique. Besides, how are you operating on it without digitizing it? What data cannot be replicated? The quantum no cloning theorem? Good luck getting that level of precision.
Post reply on HN