Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Deepfake Offensive Toolkit (real-time deepfakes for virtual cameras)

github.com

Post reply on HN
Print view View original

111–120 of 336 posts

Re: Deepfake Offensive Toolkit (real-time deepfakes for virtual cameras)

#111

Post by mike_hock »

coldcode wrote: post #85
throw457 wrote: post #54

Earlier quoted context omitted.

Deepfakes are for deceiving humans. There are still way too many artifacts in the outputs that are easy to detect in code.

The problem comes that even with extremely good deepfake detection, political operations can still use the deepfakes to fool the average voter into believing that is it real, irrespective of any successful analysis. You and I might not be fooled but we don't matter if 10000X people are fooled and affect our lives anyway in a negative way.

For propaganda purposes, the deepfake needs to be neither real-time, deep, nor fake, nor does it need to exist at all. A false message with zero supporting evidence repeated endlessly by high-profile influencers is a million times more effective than a single bombshell video ignored by mainstream media.

Re: Deepfake Offensive Toolkit (real-time deepfakes for virtual cameras)

#113

Post by bjt2n3904 »

mensetmanusman wrote: post #107
bjt2n3904 wrote: post #98

Earlier quoted context omitted.

They literally have a demonstration of the tool being used to evade an intrusive "send us a video of your face, turn your head both ways, blink". That isn't a defensive use!?

Definitely useful for scammers.

Then build a cultural revolution that instills distrust in what you see. We already teach the elderly not to trust a link that says, "you are the millionth visitor, click to win your prize". Teach them also to look past the video call and what looks (and sounds) like their grandson to what is actually being said. He's asking for your bank password, that behavior doesn't match!

The cat is already out of the bag. Going "nooo, unpublish this" doesn't protect people from scammers, who will continue to use it.

Society must unlearn that seeing is believing. That is what is morally irresponsible, not this software.

Re: Deepfake Offensive Toolkit (real-time deepfakes for virtual cameras)

#114

Post by sva_ »

To those who ask about the ethics of releasing something like this, I'd say that this technology already exists, and bad actors probably already can get access if they really want to and are sophisticated enough. Making this available to the general public will spread awareness of the existence of such tools, and can then possibly have a preventive effect.

Re: Deepfake Offensive Toolkit (real-time deepfakes for virtual cameras)

#115

Post by belter »

Planktonne wrote: post #7

> Authors and contributing developers assume no liability and are not responsible for any misuse or damage caused by the use of this program. Anything that can be created, will be created. However, that doesn't free you from all moral culpability. If you create something, make it freely accessible and easy to use, then I think you are partly responsible for its misuse. I'm not saying that they shouldn't have created…

Should Ron Rivest, Adi Shamir, and Leonard Adleman have responsibility if criminals use their discovery?

Re: Deepfake Offensive Toolkit (real-time deepfakes for virtual cameras)

#116

Post by VMG »

jacquesm wrote: post #67
VMG wrote: post #63

Earlier quoted context omitted.

Isn't the situation completely analogous to other "ethical hacking" software? I distinctly remember a very similar discussion around firesheep 12 years ago: https://www.computerworld.com/article/2469667/firesheep-fire... https://news.ycombinator.com/item?id=1828955 > There are probably going to be a lot of people negatively affected by this for quite some time to come. One thing to point out is that there are grades…

Indeed it is, but I'm interested in this particular author's stance on this, prompted by their disclaimer which clearly indicates that they realize that there is the risk of abuse, and an AMA seems to be an excellent opportunity to gain some insight.

I agree.

I'm not OP but I would answer like this: abuses of this technology are inevitable and can only be mitigated by counter-software (which leads to an arms race).

The release of this source code could kickstart the development of deepfake detection software.

Or maybe in general people need to put less weight on video evidence.

Re: Deepfake Offensive Toolkit (real-time deepfakes for virtual cameras)

#117

Post by Planktonne »

jacquesm wrote: post #61
yosito wrote: post #60

Earlier quoted context omitted.

It doesn't matter what anyone thinks about it.

I actually disagree with that. I think software and ethics are closely intertwined and too often we see them as disjoint. It's interesting to try to find a software project that does not have an ethically dubious angle to it, that isn't all that simple.

Fully agreed. I think software directly addresses ethical questions far more often than we tend to be comfortable admitting - almost everything involving personal data runs full tilt into questions of identity and ownership, for example.

There's no right answer to a lot of the dilemmas raised, and people will reach their own conclusions and choose particular principles to follow. I don't want to tell people how to think, but I do think that everyone has a responsibility to actually think about these issues, wherever they end up landing on them.

Re: Deepfake Offensive Toolkit (real-time deepfakes for virtual cameras)

#118

Post by Planktonne »

krageon wrote: post #24
Planktonne wrote: post #7

> Authors and contributing developers assume no liability and are not responsible for any misuse or damage caused by the use of this program. Anything that can be created, will be created. However, that doesn't free you from all moral culpability. If you create something, make it freely accessible and easy to use, then I think you are partly responsible for its misuse. I'm not saying that they shouldn't have created…

This has a clear benign use. Of course it sucks that you can also use it in a hostile manner - but the fact that this tool is publicly available rather than hidden in the pocket of some unscrupulous blackhat means that every space that uses verification with these methods can now incorporate this type of testing. That's a net benefit for society. I do think disclaimers like this are a little juvenile (it reeks of a U…

This is a tool with a clear benign use, along with a bunch of malign ones. If you create something with significant potential for harm, then you should at least think about that, the potential consequences, and possible mitigations.

I wouldn't term it a failure of empathy to ask that people consider the impact of their actions on others. I totally get why they have the disclaimer, and I wouldn't ask them to remove it, but I don't think it's enough. Given the clear potential for harm here, the potential uses and misuses of this tool should have been addressed directly.

Re: Deepfake Offensive Toolkit (real-time deepfakes for virtual cameras)

#119

Post by garblegarble »

darkwater wrote: post #91
tgv wrote: post #30

Earlier quoted context omitted.

You don't need to detect a deepfake in the same time as it takes to generate one, do you?

If you get a Zoom call from your CFO telling you to issue a money transfer to that bank account, that it is 100% legit, how much time do you have to detect a deepfake?

Does your company instantly execute payment instructions to new supplier accounts based on a remote verbal instruction?

Re: Deepfake Offensive Toolkit (real-time deepfakes for virtual cameras)

#120

Post by Planktonne »

Reubensson wrote: post #90
Planktonne wrote: post #48

Earlier quoted context omitted.

> there is a moral imperative to create and release tools like this as free and open source software so that _anyone_ has access to them That's an argument I have a lot of time for, but it needs to be made, rather than - as at present - having the whole issue sidestepped. With any tool like this, I think there's a need for ethical due diligence as much as there is for engineering, and there's no evidence that this as…

To me it sounds like it boils down to either you release the software or you don't. If you decide to release it into public, I don't think there is much you can realistically do to prevent abuse.

Release/don't release is definitely the choice with the highest impact, but it's not the only thing that can be considered.

Who you release things to, in what form, with what framing - all of these impact how something will be used. You can never eliminate bad actors, but you can encourage responsible use.

Post reply on HN