Deepfake Offensive Toolkit (real-time deepfakes for virtual cameras) • Hacker News Board

Re: Deepfake Offensive Toolkit (real-time deepfakes for virtual cameras)

#171

Post by driverdan » Tue, Jun 07, 2022, 1:01 PM UTC

throw9871928 wrote: post #76

I work at Axis, which makes surveillance cameras.[0] This comment is my own, and is not on behalf of the company. I'm using a throwaway account because I'd rather not be identified (and because surveillance is quite controversial here). Axis has developed a way to cryptographically sign video, using TPMs (Trusted Platform Module) built into the cameras and embedding the signature into the h.264 stream.[1] The video c…

I don't see how that would be useful here. I'm not giving out info about my webcam to anyone. There would be no way to verify the signature.

Re: Deepfake Offensive Toolkit (real-time deepfakes for virtual cameras)

#172

Post by [deleted] » Tue, Jun 07, 2022, 1:04 PM UTC

Planktonne wrote: post #7

> Authors and contributing developers assume no liability and are not responsible for any misuse or damage caused by the use of this program. Anything that can be created, will be created. However, that doesn't free you from all moral culpability. If you create something, make it freely accessible and easy to use, then I think you are partly responsible for its misuse. I'm not saying that they shouldn't have created…

[deleted]

Re: Deepfake Offensive Toolkit (real-time deepfakes for virtual cameras)

#173

Post by bko » Tue, Jun 07, 2022, 1:12 PM UTC

barnabee wrote: post #169

bko wrote: post #148

Earlier quoted context omitted.

Honest question, do you apply the same reasoning to gun control?

Good question. I would apply that reasoning to the information about how a gun works for sure, but IMO there is some (admittedly hard to define) amount of danger or perhaps destructive power beyond which we might or perhaps should choose as a society to restrict the ability to wield a tool and certainly at least some guns exceed that limit for me. I’m not sure that any software, on its own, does though, at least I ca…

I agree its a very hard line to draw. But I think certain technology is more contagious and dangerous. For instance, a single person with a gun can inflict damage locally with dire consequences on his or her own personal safety in the process. But if someone has a virus (physical or digital), then it caries a kind of contagion that can disproportionately hurt many more people. That's why we can't equate something like deaths due to car accidents to pandemic deaths. A single car accident doesn't increase the risk of all car accidents. It's localized. But a single infection increases the risk of future infections due to contagion. Whether you think any technology falls into the dangerous contagious category or not is up for debate.

Re: Deepfake Offensive Toolkit (real-time deepfakes for virtual cameras)

#174

Post by barnabee » Tue, Jun 07, 2022, 1:13 PM UTC

mastersummoner wrote: post #167

rolisz wrote: post #94

Earlier quoted context omitted.

For nuclear weapons, even if plans were publicly available, getting the right kind of uranium would still be a major hurdle for 99.9% of the people.

But what if they weren't? What if one could create something with similar destructive force with materials easily accessible, and the only hurdle is knowledge of how to do so? It's honestly an interesting moral question.

Over time the number of single humans that have the power to wipe out a sizeable portion of life on Earth (which is almost certainly non-zero already) will almost certainly increase due to advances in technology and dissemination of information.

Do we aim for security through obscurity or work out how to deal with that increasingly terrifying reality? I think at some point we will be forced to do latter.

Re: Deepfake Offensive Toolkit (real-time deepfakes for virtual cameras)

#175

Post by scoutt » Tue, Jun 07, 2022, 1:13 PM UTC

I can imagine more scenarios other than "bad actors" (scammers and such) for this software...

For example: what about using this in an interview for a remote job (possibly intercontinental)? It could tip the balance in some situations (due to biases). For example, beating ageism by projecting a 25 years old version of ourselves. Or removing a tatoo or birth defect (and possibly other changes too, you can imagine).

Re: Deepfake Offensive Toolkit (real-time deepfakes for virtual cameras)

#176

Post by solsane » Tue, Jun 07, 2022, 1:14 PM UTC

Planktonne wrote: post #7

> Authors and contributing developers assume no liability and are not responsible for any misuse or damage caused by the use of this program. Anything that can be created, will be created. However, that doesn't free you from all moral culpability. If you create something, make it freely accessible and easy to use, then I think you are partly responsible for its misuse. I'm not saying that they shouldn't have created…

I disagree. The moral responsibility really rests on the person who uses the tool.

For instance, I once cheated by using gcc/godbolt to generate assembly output for a class from C code. By this logic, Richard Stallman should be blamed for my misconduct.

There are any number of reasons for supplanting another face onto your own, many of which are simply good fun. If you choose to use this for scamming or perverted reasons, so be it.

Moral posturing aside, perhaps there could be an invisible watermark or something included by default to easily identify less technically inclined actors as users of this tool.

Re: Deepfake Offensive Toolkit (real-time deepfakes for virtual cameras)

#177

Post by oliver910 » Tue, Jun 07, 2022, 1:17 PM UTC

sva_ wrote: post #114

To those who ask about the ethics of releasing something like this, I'd say that this technology already exists, and bad actors probably already can get access if they really want to and are sophisticated enough. Making this available to the general public will spread awareness of the existence of such tools, and can then possibly have a preventive effect.

I agree that raising awareness that tools like this are possible is important and that sufficiently advanced actors can do this anyway, however I don't think in this case releasing pre-trained weights to the general public is responsible. This could probably be used to help bypass crypto exchange KYC for moneylaundering purposes. I'm not sure what the best access model is - email us with a good reason to get access to the weights perhaps - but what alarms me is there seems to be no consideration for misuse or responsible release at all.

Re: Deepfake Offensive Toolkit (real-time deepfakes for virtual cameras)

#178

Post by ethanbond » Tue, Jun 07, 2022, 1:25 PM UTC

barnabee wrote: post #174

mastersummoner wrote: post #167

Earlier quoted context omitted.

But what if they weren't? What if one could create something with similar destructive force with materials easily accessible, and the only hurdle is knowledge of how to do so? It's honestly an interesting moral question.

Over time the number of single humans that have the power to wipe out a sizeable portion of life on Earth (which is almost certainly non-zero already) will almost certainly increase due to advances in technology and dissemination of information. Do we aim for security through obscurity or work out how to deal with that increasingly terrifying reality? I think at some point we will be forced to do latter.

This trend is clearly real, but what exactly do you mean “work out how to deal with that?” One of the obvious ways to deal with it, albeit imperfectly, is in fact to keep these technologies as obscure as possible for as long as possible.

Maybe an interesting norm to try to generate in this community would be, “if you plan on publishing attack X, you should publish immediately alongside it your best guess as to the effective counter to X.”

Re: Deepfake Offensive Toolkit (real-time deepfakes for virtual cameras)

#179

Post by pragmatick » Tue, Jun 07, 2022, 1:26 PM UTC

giorgiop wrote: post #6

Dot was used for performing vulnerability assessments on many biometric KYC vendors on 2022. The Verge covered this study in this article https://www.theverge.com/2022/5/18/23092964/deepfake-attack-...

That article is linked in the second paragraph of the readme.

Re: Deepfake Offensive Toolkit (real-time deepfakes for virtual cameras)

#180

Post by registeredcorn » Tue, Jun 07, 2022, 1:27 PM UTC

I'm really excited to see what could be done with this! I think the primary benefits of this being released are two fold:

1) It will give security researchers more freely available technology to work with in order to try and fight the malicious use of deepfakes. (I saw some interesting comments in this thread about TPM. It'd be interesting to see what other solutions are out there.)

2) It would raise the overall awareness of the general population about the existence and advancements that deepfake technology has made. I would argue that a small subset of the overall population know what the term "deepfake" means, and even fewer are aware at how far it has progressed in only a few short years. (I'm not super well versed in the topic myself, I just know that I've heard a lot of progress has been made.)

I think that since this tech is already actively being used by bad actors, the best course of action that we can take until at least a somewhat good counter to it has been adopted (and then quickly defeated) is to make as many people aware that this is something that could affect them, or their families. That this is something that could be used to get someone fired, or hurt, or killed. I think that the more that people are aware of its existence, the less impactful the overall effect of deepfakes becomes. People learn to look twice before making a call on something, because of how easy it has become to fake audio and video.