Live data from Hacker News

The Terrifying A.I. Scam That Uses Your Loved One's Voice

newyorker.com

31–40 of 47 posts

Re: The Terrifying A.I. Scam That Uses Your Loved One's Voice

#31

Earlier quoted context omitted.

It's insanely cruel and fucked up. Imagining what this is like for the victims goes way, way beyond what calling it a "scam" implies.

It’s actual terrorism. It’s just small beans compared to other things which we usually call terrorism, but that’s what quite a large number of these “scams” are, small scale terrorism.

[deleted]

Re: The Terrifying A.I. Scam That Uses Your Loved One's Voice

#32

Earlier quoted context omitted.

It’s actual terrorism. It’s just small beans compared to other things which we usually call terrorism, but that’s what quite a large number of these “scams” are, small scale terrorism.

Ok, that's fine, but then we're going to need a new word to describe actual terrorism. This had/has a meaning that involves specifically targetting governments and destabilizing them.

Perhaps "psychological torture and extortion" also works as a phrase for this sort of crime.

Re: The Terrifying A.I. Scam That Uses Your Loved One's Voice

#33
post #20

Earlier quoted context omitted.

Interesting any article about the systems on denmark

It's more than Denmark, the Nordics in general use digital signatures for accessing banking, government services (tax agencies, healthcare, pension systems, etc.), online payment ID checks, and so on. In Denmark I believe NemID/MitID is the most used one, here in Sweden we have BankID (most used), and Freja eID, there are a few smaller ones as well; Norway has their own BankID that is independent from the Swedish one…

The original Finnish one was TUPAS, it has been available since at least 1998. It was phased out in 2019 as it was not compatible with eIDAS. For end users there isn't really any difference but the old system required separate contracts between each bank and service provider. Due to that service providers also couldn't use 3rd party (think service like Auth0) who would handle contracts with individual banks. From my understanding it was also fairly expensive (in terms of per authentication event). The new one is called Finnish Trust Network and uses OIDC and in some cases SAML, TUPAS had it's own protocol (I'm not sure if there even was a standard protocol for that kind of authentication back then, even SAML 1.0 came out only in 2002).

Re: The Terrifying A.I. Scam That Uses Your Loved One's Voice

#34
post #9

As horrible as this is, I suspect this will be far worse in enterprise companies. We’ve already seen issues with email impersonation where top managers or CEOs order money transfers, and since this works better than it should, we now use confirmation of some sort. Imagine if you’d get both the email and a call, maybe even a video call in the future with the confirmation… Especially CEOs will be very “easy” to imperso…

I wonder if the problem is with impersonating C-level executives or if the problem is that they have power to just send money.. In recent John Oliver episode there was banker who simply bankrupted his bank by sending money to the scammer.

Or this guy https://www.npr.org/2019/03/25/706715377/man-pleads-guilty-t.... Do these companies not have proper process for sending so much money with multiple levels of people involved?

Re: The Terrifying A.I. Scam That Uses Your Loved One's Voice

#35
post #9

As horrible as this is, I suspect this will be far worse in enterprise companies. We’ve already seen issues with email impersonation where top managers or CEOs order money transfers, and since this works better than it should, we now use confirmation of some sort. Imagine if you’d get both the email and a call, maybe even a video call in the future with the confirmation… Especially CEOs will be very “easy” to imperso…

As a CEO, this is one of the many reasons why I do not do money transfers myself.

Ps: your insights are spot on. Been plagued by this issue for years. Deep fake is merely a refinement of a pre-existing fraud pattern.

Re: The Terrifying A.I. Scam That Uses Your Loved One's Voice

#36
post #16

Earlier quoted context omitted.

I doubt a single voice recording could generate speech with enough emotional depth that I'd be fooled by a fake call from my better half. Do people really have such a shallow connections to their loved ones that they wouldn't instinctively feel that something isn't right? C'mon, we humans operate with a lot of extra clues. Its not like we just STT and "read" the resulting output.

You're right to be suspicious but wrong in your interpretation of reality. The issue is that there's a big difference in how you'll interpret things in different environments and stress levels. Under pressure a lot of your normal checks will go out the window. This is why conmen often put you under time pressures, because they're stressing your system. Under normal circumstances, you're prefectly right, but under dur…

As someone who works in IT for roughly 30 years, and who never fell for any kind of phishing, I beg to differ. I think my interpretation of reality, as you put it, isn't as bad as you might believe. Also, all the phishing-victims I know have something in common, a certain lack of rigour. You seem to claim that everyone can fall for phishing. I beg to differ, its a certain class of not very thorough (l)users.

Your comment regarding enshitifcation has some truth to it I didn't really think about yet, thanks.

However, my original comment still stands as it is. I challenge anyone to create a voice scam that fools me.

Re: The Terrifying A.I. Scam That Uses Your Loved One's Voice

#38
post #6

[flagged]

It's insanely cruel and fucked up. Imagining what this is like for the victims goes way, way beyond what calling it a "scam" implies.

And yet the top comment here is:

"As horrible as this is, I suspect this will be far worse in enterprise companies."

To give the "strongest interpretation" I presume by worse they mean more prevalent. But I also think a fundamental lack of human perspective in the tech world is the real root cause of cybersecurity problems threatening society. I mean, in this context who seriously cares about "enterprise" or money?

Re: The Terrifying A.I. Scam That Uses Your Loved One's Voice

#40
post #16

Anyone believing AI will be a nice thing in the future: With data from social media it's a fucking nightmare - billions of people having their relationships visible to everyone and a singular voice recording/video is enough to scam your friends, family, workplace. I don't see a future of the current communication channels without strong authentication measures - people already get scammed with shitty tactics by billi…

I doubt a single voice recording could generate speech with enough emotional depth that I'd be fooled by a fake call from my better half. Do people really have such a shallow connections to their loved ones that they wouldn't instinctively feel that something isn't right? C'mon, we humans operate with a lot of extra clues. Its not like we just STT and "read" the resulting output.

> that I'd be fooled by a fake call

Can you say the same thing about all of your friends and loved ones?

You don't have to personally fall for the scam for it to negatively impact you. Maybe your mom falls for it, and now she can't pay her mortgage.

Fake call scams already generate billions of dollars in stolen money. Telecoms let spam phone calls run rampant for a decade before being forced to fix it. AI video/audio deepfakes at scale could 10x this industry.

Post reply on HN