The golden age of scammers: AI-powered phishing • Hacker News Board

Re: The golden age of scammers: AI-powered phishing

#101

Post by kkielhofner » Wed, Jul 17, 2024, 3:16 PM UTC

dbspin wrote: post #75

I'm kind of amazed how slow 'AI phishing' has been to roll out. The technology for customised text based attacks at scale has been available at least since Llama was open sourced. The tech for custom voice and image based attacks is basically there too with whisper / tortoise and stable diffusion - though clearly more expensive to render. I'm honestly not sure why social networks aren't being leveraged more to target…

The ROI on scams based in Indian call centers is already huge.

They recently took my mom for $25k for what was a few hours of “work” over the span of two days. When I reviewed their communications and got the full story from my mom they’re in some ways laughably bad and in other ways very cunning.

Turns out it all started with a comically bad initial e-mail, pop-up, and then remote access. Then follow-up calls and text messages (Bitcoin QR code with her bank logo) and multiple people impersonating banks, various government agencies, etc. The end to end pipeline to replicate this via AI would be very complex and difficult (today).

I imagine the increased scale, additional opportunity, and reduced “payroll” that could be realized utilizing AI (given the initial level of development effort) just isn’t there. Yet.

Re: The golden age of scammers: AI-powered phishing

#102

Post by meroes » Wed, Jul 17, 2024, 3:16 PM UTC

manishsharan wrote: post #90

This is going to become so much worse https://news.ycombinator.com/item?id=40942307 Imagine old people getting phone calls from frantic children. They won't know real from fake. Add tech like this to SIM forgery ..and we will devolve from a high trust society to a no trust society.

Happened to my family. Grandfather got a call from a panicky grandchild that sounded like them. The teller at Western Union is the only reason it was stopped. The scary thing is this happened more than 5 years ago so it’s only getting worse.

Re: The golden age of scammers: AI-powered phishing

#103

Post by ryandrake » Wed, Jul 17, 2024, 3:22 PM UTC

manishsharan wrote: post #90

This is going to become so much worse https://news.ycombinator.com/item?id=40942307 Imagine old people getting phone calls from frantic children. They won't know real from fake. Add tech like this to SIM forgery ..and we will devolve from a high trust society to a no trust society.

I think everyone with elderly parents already needs to have "the talk" with those parents, to help them to understand and deal with common (and less common) scams that prey on old people, what forms of communication to trust, what capabilities scammers have, and so on. All that is changing is the scammers' capabilities.

Re: The golden age of scammers: AI-powered phishing

#104

Post by dbspin » Wed, Jul 17, 2024, 3:32 PM UTC

kkielhofner wrote: post #101

dbspin wrote: post #75

I'm kind of amazed how slow 'AI phishing' has been to roll out. The technology for customised text based attacks at scale has been available at least since Llama was open sourced. The tech for custom voice and image based attacks is basically there too with whisper / tortoise and stable diffusion - though clearly more expensive to render. I'm honestly not sure why social networks aren't being leveraged more to target…

The ROI on scams based in Indian call centers is already huge. They recently took my mom for $25k for what was a few hours of “work” over the span of two days. When I reviewed their communications and got the full story from my mom they’re in some ways laughably bad and in other ways very cunning. Turns out it all started with a comically bad initial e-mail, pop-up, and then remote access. Then follow-up calls and te…

I'm so sorry, that's absolutely awful. Hope your moms OK.

Re: The golden age of scammers: AI-powered phishing

#105

Post by dbspin » Wed, Jul 17, 2024, 3:36 PM UTC

jjulius wrote: post #87

dbspin wrote: post #75

I'm kind of amazed how slow 'AI phishing' has been to roll out. The technology for customised text based attacks at scale has been available at least since Llama was open sourced. The tech for custom voice and image based attacks is basically there too with whisper / tortoise and stable diffusion - though clearly more expensive to render. I'm honestly not sure why social networks aren't being leveraged more to target…

Forgive my ignorance, but why are we surprised that voice messages aren't being spoofed more often? Doesn't this require a pretty darn decent dataset for training? Unless they've got a ton of videos of themselves shared on a public social media profile, I don't know that this is going to be a thing.

I'm not a data science / AI person at all, but AFAIK while initial model training is enormously processor intensive, customising a trained model is not. This has been my experience playing with custom training stable diffusion on very low memory home GPU. I'd assume it's true for voice generators like Tortoise [1] also. Moreover, while Eleven Labs isn't in my opinion as good, they let you custom train voices for an incredibly low cost and with a tiny amount of sample data [3].

For perfect audio spoofing, lots of audio would be needed. Bear in mind there are literally millions of podcasts available [2], and billions of youtube videos. Should be trivial to grab biographic data and voice samples from a subset of them.

[1] https://github.com/neonbjb/tortoise-tts

[2] https://whatsthebigdata.com/podcasts-statistics/

[3] https://elevenlabs.io/

Re: The golden age of scammers: AI-powered phishing

#106

Post by dbspin » Wed, Jul 17, 2024, 3:41 PM UTC

diatone wrote: post #89

dbspin wrote: post #75

I'm kind of amazed how slow 'AI phishing' has been to roll out. The technology for customised text based attacks at scale has been available at least since Llama was open sourced. The tech for custom voice and image based attacks is basically there too with whisper / tortoise and stable diffusion - though clearly more expensive to render. I'm honestly not sure why social networks aren't being leveraged more to target…

It’s not enough for the method to be possible, it has to be economical. If it’s not prevalent yet it means one of two things: (1) cheaper methods work too well for now to even bother with sophisticated approaches (2) it’s too expensive to be worth the effort It would be interesting to plot out the cost of an integrated AI stack for this over time.

There's a third possibility. It's just not turnkey enough yet for criminal enterprises to bother. I'm sure cartels and mafia level gangsters sometimes have great tech people, but the level of op sec and technical knowhow exhibited by most professional criminals seems low.

https://www.npr.org/2024/05/31/1197959218/fbi-phone-company-...

Re: The golden age of scammers: AI-powered phishing

#107

Post by nanidin » Wed, Jul 17, 2024, 3:43 PM UTC

savanaly wrote: post #91

jjulius wrote: post #87

Earlier quoted context omitted.

Forgive my ignorance, but why are we surprised that voice messages aren't being spoofed more often? Doesn't this require a pretty darn decent dataset for training? Unless they've got a ton of videos of themselves shared on a public social media profile, I don't know that this is going to be a thing.

The dataset you need to train in the first place is indeed huge, but I think the idea is once them model is trained, new "voices" can be acquired with much less data than was required to train it in the first place. Just like you can instruct ChatGPT to talk about topics never heard of on the internet and in a dialect you customize and invent on the spot and it can comply, despite not consuming an internet's worth of…

Soon the role of the Indian call centers will change from running the scam directly to making spam calls to trusted contacts of the intended mark to collect voice data for TTS model fine tuning.

Re: The golden age of scammers: AI-powered phishing

#108

Post by diob » Wed, Jul 17, 2024, 3:45 PM UTC

WesternWind wrote: post #2

Hey, just going to say what I've been telling folks IRL, if you are reading this, and your parents and family members aren't tech savvy, you need to set them up with two factor authentication now. Because you know how to do that, and it's so much easier than helping them when they get hacked.

It's not just that either.

Talk to them about investment / romance scams as well. Unfortunately, most folks do these things "willingly" and get in deep.

Re: The golden age of scammers: AI-powered phishing

#109

Post by nanomonkey » Wed, Jul 17, 2024, 3:47 PM UTC

jjulius wrote: post #87

dbspin wrote: post #75

I'm kind of amazed how slow 'AI phishing' has been to roll out. The technology for customised text based attacks at scale has been available at least since Llama was open sourced. The tech for custom voice and image based attacks is basically there too with whisper / tortoise and stable diffusion - though clearly more expensive to render. I'm honestly not sure why social networks aren't being leveraged more to target…

Forgive my ignorance, but why are we surprised that voice messages aren't being spoofed more often? Doesn't this require a pretty darn decent dataset for training? Unless they've got a ton of videos of themselves shared on a public social media profile, I don't know that this is going to be a thing.

I've had this nagging worry that my voice will be harvested during a random call acting like a sales person or survey. Besides getting money from family members, I really can't think of a scenario where my voice would allow a scammer to get into one of my accounts, but I'm sure it would make social engineering such an attach that much easier.

Re: The golden age of scammers: AI-powered phishing

#110

Post by burningChrome » Wed, Jul 17, 2024, 3:56 PM UTC

kkielhofner wrote: post #101

dbspin wrote: post #75

I'm kind of amazed how slow 'AI phishing' has been to roll out. The technology for customised text based attacks at scale has been available at least since Llama was open sourced. The tech for custom voice and image based attacks is basically there too with whisper / tortoise and stable diffusion - though clearly more expensive to render. I'm honestly not sure why social networks aren't being leveraged more to target…

The ROI on scams based in Indian call centers is already huge. They recently took my mom for $25k for what was a few hours of “work” over the span of two days. When I reviewed their communications and got the full story from my mom they’re in some ways laughably bad and in other ways very cunning. Turns out it all started with a comically bad initial e-mail, pop-up, and then remote access. Then follow-up calls and te…

Banks are actually getting much better at ferreting out these scams.

My Mom got one of those calls where the guy said my son was jail and they needed to pay his cash bail to get him out. They never called me or him, just panicked and went to the bank to get the cash out and the bank refused to give them the money because they knew it was a scam. They started asking my parents questions and pointing out how the scam worked when they got agitated they weren't giving them the money.

They called me from the bank and I texted my son and he was completely confused since he was at work and was just fine. I thanked the people at the bank profusely for their intervention.

It was a great learning experience for my parents as well. They are way more leery about strangers calling and have already hung up on several scammers. They also filter out email messages and won't click on any links in emails or pop-ups.

The whole experience really put their radar up for this stuff now.