Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Deepfake Offensive Toolkit (real-time deepfakes for virtual cameras)

github.com

Post reply on HN
Print view View original

251–260 of 336 posts

Re: Deepfake Offensive Toolkit (real-time deepfakes for virtual cameras)

#251

Post by kube-system »

throw9871928 wrote: post #76

I work at Axis, which makes surveillance cameras.[0] This comment is my own, and is not on behalf of the company. I'm using a throwaway account because I'd rather not be identified (and because surveillance is quite controversial here). Axis has developed a way to cryptographically sign video, using TPMs (Trusted Platform Module) built into the cameras and embedding the signature into the h.264 stream.[1] The video c…

That will help someone determine that a fake video didn't come from their own camera.

But most videos where we're worried about deep fakes are videos that come from other people's cameras, where we don't know which signature should be valid, nor whether it should have a signature.

Re: Deepfake Offensive Toolkit (real-time deepfakes for virtual cameras)

#252

Post by colejohnson66 »

tinus_hn wrote: post #130
throw9871928 wrote: post #76

I work at Axis, which makes surveillance cameras.[0] This comment is my own, and is not on behalf of the company. I'm using a throwaway account because I'd rather not be identified (and because surveillance is quite controversial here). Axis has developed a way to cryptographically sign video, using TPMs (Trusted Platform Module) built into the cameras and embedding the signature into the h.264 stream.[1] The video c…

The surveillance states wet dream, where you can just look up who took that ‘unfair’ video of your agent breaking the rules.

You're being uncharitable. I read it as a way for a person to prove they took the video, not as a database of personsigning key. In other words, the government would only be able to see that video 123 was signed by signature 456. It would be up to the poster to prove they own the camera that produced signature 456.

Re: Deepfake Offensive Toolkit (real-time deepfakes for virtual cameras)

#253

Post by ospray »

Planktonne wrote: post #7

> Authors and contributing developers assume no liability and are not responsible for any misuse or damage caused by the use of this program. Anything that can be created, will be created. However, that doesn't free you from all moral culpability. If you create something, make it freely accessible and easy to use, then I think you are partly responsible for its misuse. I'm not saying that they shouldn't have created…

The main value in releasing tools like this is to demonstrate weakness in our current security controls. A key weakness of biometrics is that there is no secret data. Open source tooling like this help people understand that.

Re: Deepfake Offensive Toolkit (real-time deepfakes for virtual cameras)

#254

Post by phkahler »

ethanbond wrote: post #64
barnabee wrote: post #43

Earlier quoted context omitted.

I’d argue there is a moral imperative to create and release tools like this as free and open source software so that anyone has access to them and can choose whether to use them, rather than only sophisticated and well resourced adversaries. IMO the creators should feel good about their actions, even if they feel bad or apprehensive about the direction of the world because this technology exists at all.

Does this apply to nuclear weapons? To biological weapons? “Tool is expensive” is a real, effective deterrent to frivolous use of a tool. There’s a reason we pool resources into governments, which is explicitly so we (via a government) have capabilities that we (as individuals) don’t have.

>> Does this apply to nuclear weapons? To biological weapons?

That may not be a fair comparison. Nuclear weapons are difficult to make, and using them will not go undetected - the damage would be enormous. Deep fakes are a real threat today, and anyone willing to pay for the expertise (which isn't that hard to find) can have them made. By putting this out there, deepfakes are now something almost anyone can make which means the threat is imminent and needs to be addressed. I understand the argument, but still not sure about it. Just wanted to say it's different than nuclear weapons.

Re: Deepfake Offensive Toolkit (real-time deepfakes for virtual cameras)

#255

Post by thingification »

haswell wrote: post #242
thingification wrote: post #229

Earlier quoted context omitted.

I think you describe the conventional wisdom well here. But: are those really the only options? Paranoid stop-the-world wishful thinking or all-new-trends-are-inevitable? I realise I'm almost certainly wildly misrepresenting your point of view with the latter label! I don't really mean to ascribe it to you. But I think the conversation often ends up tacitly as a debate between those two positions, both of which are m…

I think in many ways, the continued evolution of technology will turn back the clock on progress in other areas of our lives. In ages past, impersonating another person was not nearly as difficult as it is today. Advances in identity verification and such have eliminated a certain class of problems. The emergence of deepfakes doesn’t necessarily introduce “new” problems, it just resets the progress on some very old o…

> I think in many ways, the continued evolution of technology will turn back the clock on progress in other areas of our lives.

I don't disagree that it can have that effect.

Just to note: this use of "turn back the clock" is quite distinct from "turn back the clock on technology".

> Assuming this is solved not by banning the tech but by making progress elsewhere, some pretty worrisome implications come with that.

My point was not that we can make progress elsewhere to solve resulting problems (we can) but that we can also not use the technology in the "inevitable" bad way. It happens!

> One antidote to deepfakes would be even more progress on identity verification and tracking the source of digital media for purposes of authentication / validity.

One other antidote (of many I'm sure) would be to do less identity verification in the first place.

Re: Deepfake Offensive Toolkit (real-time deepfakes for virtual cameras)

#256

Post by phkahler »

IfOnlyYouKnew wrote: post #84
barnabee wrote: post #43

Earlier quoted context omitted.

I’d argue there is a moral imperative to create and release tools like this as free and open source software so that anyone has access to them and can choose whether to use them, rather than only sophisticated and well resourced adversaries. IMO the creators should feel good about their actions, even if they feel bad or apprehensive about the direction of the world because this technology exists at all.

I can’t see a way this technology could be used defensively. Wide access just leads to more abuse. There’s no principle by which it serves some sort of justice to make some crime more accessible. It’s surprising how often this argument is made compared to, say, equal-opportunity access to fancy cars or good housing.

>> I can’t see a way this technology could be used defensively. Wide access just leads to more abuse. There’s no principle by which it serves some sort of justice to make some crime more accessible.

Should that argument be made to the original research? I don't think it applies to one but not the other.

Re: Deepfake Offensive Toolkit (real-time deepfakes for virtual cameras)

#257

Post by munk-a »

rolisz wrote: post #94
ethanbond wrote: post #64

Earlier quoted context omitted.

Does this apply to nuclear weapons? To biological weapons? “Tool is expensive” is a real, effective deterrent to frivolous use of a tool. There’s a reason we pool resources into governments, which is explicitly so we (via a government) have capabilities that we (as individuals) don’t have.

For nuclear weapons, even if plans were publicly available, getting the right kind of uranium would still be a major hurdle for 99.9% of the people.

So then what about releasing data on how to enrich uranium at home or selling it on ? I know usually there are multi-million dollar facilities required for enriched uranium production but why shouldn't the free market allow people to buy enriched uranium - it isn't our place to morally prejudge what people may use it for and the bad people will have access to it anyways... so why can't I buy a kilo of U-235 on Amazon?

I'm sure there are some very worthy technical efforts to improve mankind that are being hampered by the lack of easily accessible U-235 so why are we restricting this material from Chemists that want to Chem and Physicists that want to Fizz?

Re: Deepfake Offensive Toolkit (real-time deepfakes for virtual cameras)

#258

Post by munk-a »

franga2000 wrote: post #249
ryandrake wrote: post #217

Earlier quoted context omitted.

Even an extreme example -- The technology to build an automated drone with a camera for targeting and a gun on a gimbal and deploy it to do a mass-shooting -- all exists today. All it would take is for some madman to pick the pieces up and assemble them. In fact, I'd be willing to bet $100 that we'll actually see some kind of tragedy to that effect in the USA within the next 5 years. The technology cats are very quic…

Many airsoft players are mounting fully automatic airsoft rifles on drones and some Russian dude already put a glock on a 1st gen DJI phantom years ago. My guess is a big thing stopping such an attack from happening is that the people doing these attacks are usually not well educated, so they don't have the arduino-starter-kit level of electronics knowledge required to pull it off.

Mental instability happens on all portions of the education and intelligence spectrum though - so if that's what we're banking on we're going to have a very rude awakening.

Re: Deepfake Offensive Toolkit (real-time deepfakes for virtual cameras)

#259

Post by toss1 »

light_hue_1 wrote: post #200
sva_ wrote: post #114

To those who ask about the ethics of releasing something like this, I'd say that this technology already exists, and bad actors probably already can get access if they really want to and are sophisticated enough. Making this available to the general public will spread awareness of the existence of such tools, and can then possibly have a preventive effect.

That's like saying that nuclear weapons exist, and bad actors can potentially get them, so let's lower the bar so that anyone can. Making such tools accessible is reprehensible. It will lead to more bad actors, to less trust in media and in any objective reality, and more erosion of our institutions and society. There is absolutely no reason whatsoever for this. It's unethical and frankly downright evil.

>>It will lead to . . . less trust in media

Trust in media is already very low, and in fact should go lower. Deepfake tech exists, and the fact that it does, and is broadly available to bad actors, should be widely known.

Obviously the best case by far is that such weapons (in this case disinformation weapons) do not exist, but the worst case is them existing but hidden — THAT is the recipe for fooling people in the greatest numbers.

These tools existing, with widespread knowledge of their existence is sort of the least-worst case for the real world in which we live.

Yes this does have the potential to kill pretty much anything related to video and photography (everything from art to news to documentation), but that was the same when spam was a literal threat to existence of email. Unless we manage it, video and photography will be trusted for nothing but boring amusement; but better than than mass deception.

Re: Deepfake Offensive Toolkit (real-time deepfakes for virtual cameras)

#260

Post by thingification »

ethanbond wrote: post #239
thingification wrote: post #231

Earlier quoted context omitted.

I honestly am puzzled: the both of you (if I understand you right), do you really think some fake videos will cause there to "be no such thing as trust"? Why? Videos aren't the only way to understand the world, and there was never any such thing as "direct observation" in the first place.

Videos are considered one of the more authoritative sources of factual information. E.g. Imagine how disruptive it’d be if a deepfake of Biden announcing support for Putin’s invasion of Ukraine were to circulate during those first hours of the invasion? Sure, there are other communications channels that would be able to dispute the authenticity of that video, but would those channels be as fast and as (apparently) au…

All of this is not "no such thing as trust".

I too dread a little all the mischief that's going to happen while the world learns about this. But people don't have to uncritically believe "the evidence of their own eyes", even though that has been very useful corrective to many mistakes, and will I guess be less so now. Evidence doesn't work that way: people always have to interpret everything in terms of their understanding of the world, and they can get better at that in response to changes like this.

Post reply on HN