Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Deepfake Offensive Toolkit (real-time deepfakes for virtual cameras)

github.com

Post reply on HN
Print view View original

141–150 of 336 posts

Re: Deepfake Offensive Toolkit (real-time deepfakes for virtual cameras)

#141

Post by IshKebab »

jacquesm wrote: post #93
throw9871928 wrote: post #76

I work at Axis, which makes surveillance cameras.[0] This comment is my own, and is not on behalf of the company. I'm using a throwaway account because I'd rather not be identified (and because surveillance is quite controversial here). Axis has developed a way to cryptographically sign video, using TPMs (Trusted Platform Module) built into the cameras and embedding the signature into the h.264 stream.[1] The video c…

That will just move the hack one level further down and will create even more confusion because then you'll have a 'properly signed video stream' as a kind of certificate that the video wasn't manipulated. But you don't really know that, because the sensor input itself could be computer generated and I can think off the bat of at least two ways in which I could do just that.

"That will just make it more difficult to make fakes."

Yes that's kind of the point. Plus I'm sure they could put the whole camera in a tamper resistant case. They could make it very difficult to access the sensor.

Including focus data should make "record a screen" a bit harder too. I guess recording a projection would be pretty hard to detect, but how likely is it that people would go to those lengths, vs using a simple deep fake tool?

Re: Deepfake Offensive Toolkit (real-time deepfakes for virtual cameras)

#142

Post by FollowingTheDao »

Hendrikto wrote: post #136
Planktonne wrote: post #7

> Authors and contributing developers assume no liability and are not responsible for any misuse or damage caused by the use of this program. Anything that can be created, will be created. However, that doesn't free you from all moral culpability. If you create something, make it freely accessible and easy to use, then I think you are partly responsible for its misuse. I'm not saying that they shouldn't have created…

If I stab someone with a knife, who is responsible? The inventor of knives? The knife‘s manufacturer? The store who sold me the knife? I would say the responsibility lays 100% with myself. I do not think it makes sense to pursue inventors for what happens to their creations, unless they actively encourage misuse.

They arrest people for making and selling meth everyday. You have a bias because making deep fakes is not illegal yet. Knives are used for cooking, what alternative good do deep fakes provide?

Re: Deepfake Offensive Toolkit (real-time deepfakes for virtual cameras)

#143

Post by FollowingTheDao »

Francesco_deep wrote: post #82
ramblerman wrote: post #68

Earlier quoted context omitted.

> If you create something, make it freely accessible and easy to use, then I think you are partly responsible for its misuse. That's a dangerous precedent. Would you apply the same logic to a kitchen knife? Or if for some reason only freemium products count (not sure why), then a pentesting tool? I understand the underlying point you are trying to make but what are you proposing as an alternative exactly? Who gets to…

I sell airplanes do I am to feel responsible for the 9/11?

Do you sell meth?

Re: Deepfake Offensive Toolkit (real-time deepfakes for virtual cameras)

#145

Post by bko »

sva_ wrote: post #114

To those who ask about the ethics of releasing something like this, I'd say that this technology already exists, and bad actors probably already can get access if they really want to and are sophisticated enough. Making this available to the general public will spread awareness of the existence of such tools, and can then possibly have a preventive effect.

I agree with everything you said, but I we shouldn't deny that opportunistic bad actors don't exist. Or it might get on their radar and be exploited. Open source tools also tend to be better maintained, documented and reliable, so the bad guys will have a better tool.

That being said, bringing it to light also has benefits like you said. If the tool is out in the open and state of the art techniques are used, technology to detect its use will also benefit.

Re: Deepfake Offensive Toolkit (real-time deepfakes for virtual cameras)

#146

Post by pixl97 »

FollowingTheDao wrote: post #134
rolisz wrote: post #94

Earlier quoted context omitted.

For nuclear weapons, even if plans were publicly available, getting the right kind of uranium would still be a major hurdle for 99.9% of the people.

But what are the consequences of the .1% of people who get over that hurdle?

Having something that can still be detected by most of the planet and making them very angry.

Re: Deepfake Offensive Toolkit (real-time deepfakes for virtual cameras)

#147

Post by Planktonne »

jcims wrote: post #65
Planktonne wrote: post #51

Earlier quoted context omitted.

Absolutely, and there's definitely a strong argument to be made for increasing awareness of existing tools and attack vectors. Importantly though, that aim is served by proofs-of-concept and information sharing more than it is by releasing point-and-shoot attacks. This comment puts it rather well [1]. As above, I'm not saying this shouldn't have been created, or even that it shouldn't have been released. I am saying…

OpenAI attempted this with GPT-2’s incremental release strategy and was lambasted on moral and ethical grounds as well. https://openai.com/blog/better-language-models/ So individuals working on a technology like this have no clear societal consensus to use as a guide when making the decision. Strong arguments can be made in both directions.

I thought that was actually a positive sign from OpenAI; potentially a disingenuous one, but it at least showed an awareness of the potential issues.

There will always be people who find the very idea of thinking about things from an ethical perspective offensive; that doesn't mean they're right. A lot of valuable discussion in this space is drowned out by people shouting down attempts to explore nuance and shine a light on the grey areas [1].

I'm not asking for censorship or a government crackdown; I do think that it's irresponsible to dodge the issues raised, act as though they are settled, or dismiss any concerns as anti-progress. As you say, there is no clear societal consensus, and there are strong arguments in both directions; what I want to see is those discussions taking place, in good faith, and those issues being acknowledged, rather than ignored.

[1] https://twitter.com/emilymbender/status/1532699418281009154

Re: Deepfake Offensive Toolkit (real-time deepfakes for virtual cameras)

#148

Post by bko »

barnabee wrote: post #43
Planktonne wrote: post #7

> Authors and contributing developers assume no liability and are not responsible for any misuse or damage caused by the use of this program. Anything that can be created, will be created. However, that doesn't free you from all moral culpability. If you create something, make it freely accessible and easy to use, then I think you are partly responsible for its misuse. I'm not saying that they shouldn't have created…

I’d argue there is a moral imperative to create and release tools like this as free and open source software so that anyone has access to them and can choose whether to use them, rather than only sophisticated and well resourced adversaries. IMO the creators should feel good about their actions, even if they feel bad or apprehensive about the direction of the world because this technology exists at all.

Honest question, do you apply the same reasoning to gun control?

Re: Deepfake Offensive Toolkit (real-time deepfakes for virtual cameras)

#149

Post by jacquesm »

pbhjpbhj wrote: post #131
jacquesm wrote: post #93

Earlier quoted context omitted.

That will just move the hack one level further down and will create even more confusion because then you'll have a 'properly signed video stream' as a kind of certificate that the video wasn't manipulated. But you don't really know that, because the sensor input itself could be computer generated and I can think off the bat of at least two ways in which I could do just that.

Isn't the point to prove physical access to the camera. As in, this stream originated at this TPM which was sold as part of this camera. So, the best you get is that the stream shows it wasn't produced with access to a particular camera. Then impersonating a YouTuber, say, requires access to their physical camera.

> Isn't the point to prove physical access to the camera.

Yes. But since you can't really prove who had and who did not have physical access to the camera (what are you going to use, video evidence ;) ) that carries little weight.

Let's say we're talking about a death penalty or a multi billion $ lawsuit, the stakes would be so high that to rely on something as trivial as a signature on a video stream would be way too risky because of the consequences.

Which expert would sign on the dotted line to the statement that 'the equipment has not been accessed by someone who wasn't authorized to do so'? And that's before we get into the question of whether this camera really is the only way that this particular video could be generated (can duplicates of the TPM be made by the manufacturer?), you'd have to do a lot of work to make guarantees that would stand the test of time.

All it takes is one 'DVD Jon' and you can kiss your carefully constructed scheme goodbye.

Finally, the whole chain of that video would have to preserve the signatures, which in a world that tries to balance privacy with other concerns may not be feasible and in fact could well have downsides all its own.

Personally I would much rather see the level of skepticism against all kinds of digital evidence go up than to see the trust go up due to supposedly magic signatures and other tricks to pretend that we have a perfect lock on this stuff. Every time we believe we do someone comes along that proves us wrong, usually after the damage is done. See also: fingerprints, face recognition, DNA evidence, lie detectors, bomb detectors and so on.

Re: Deepfake Offensive Toolkit (real-time deepfakes for virtual cameras)

#150

Post by blagie »

sva_ wrote: post #114

To those who ask about the ethics of releasing something like this, I'd say that this technology already exists, and bad actors probably already can get access if they really want to and are sophisticated enough. Making this available to the general public will spread awareness of the existence of such tools, and can then possibly have a preventive effect.

As someone with a stalker, I can't emphasize this enough. A stalker will go to all sorts of lengths to do bizarre shit. People don't believe it. I would guess governments will do some equivalent there-of.

Democratizing access to things -- including bad things -- has a preventative effect:

1) I can guard against things I know about

2) People take me seriously if something has been democratized

The worst-case scenario is if my stalker got her hands on something like deep fake technology before the police / prosecutor / jury didn't knew it existed. I'd probably be in jail by now if something like that ever happened. She's tried to frame me twice before. Fortunately, they were transparent. She'll try again.

Best case scenario is that no one has access to this stuff.

Worst case scenario is only a select group have access, and most people don't know about it.

Universal access is somewhere in between.

Post reply on HN