Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Deepfake Offensive Toolkit (real-time deepfakes for virtual cameras)

github.com

Post reply on HN
Print view View original

241–250 of 336 posts

Re: Deepfake Offensive Toolkit (real-time deepfakes for virtual cameras)

#241

Post by haswell »

wussboy wrote: post #196
Karliss wrote: post #187

Earlier quoted context omitted.

Even without deepfakes any kind of system relying on a person (or computer) not being tricked by webcam video seems quite questionable. People could still be tricked with a spliced video fragments of the real person or makeup especially if the set of face expressions used during "liveness check" is known ahead of time.

I try to imagine how society will deal with this. What if deep fakes are so perfect that anyone can generate real-time footage of anyone else doing anything? As a society we’d need to move out of the virtual and back into the physical. Would that be such a bad thing? I suspect this perfect deep fake technology might be a real boon to society.

I’ve often wondered if it’s possible to put modern tech “back in the box”. At first, this seems impossible. It permeates our daily lives in so many ways.

Continued development towards a tech dystopia might be the only way. But I’m worried it will require the dystopia part before people will wake up and accept that moving back to the physical world might be required.

The next few decades will certainly be interesting, at the very least.

Re: Deepfake Offensive Toolkit (real-time deepfakes for virtual cameras)

#242

Post by haswell »

thingification wrote: post #229
teakettle42 wrote: post #208

Earlier quoted context omitted.

On par with nuclear weapons? Downright evil? You’re being absurd. The technology exists, and it’ll get better. Pretending that it doesn’t exist or banning it won’t make it go away — it’ll just be used by the least scrupulous and most powerful. Disruptive efforts like this are most upsetting to anxiety-ridden people who think that if they could just control things firmly enough, everyone and everything will be safe. T…

I think you describe the conventional wisdom well here. But: are those really the only options? Paranoid stop-the-world wishful thinking or all-new-trends-are-inevitable? I realise I'm almost certainly wildly misrepresenting your point of view with the latter label! I don't really mean to ascribe it to you. But I think the conversation often ends up tacitly as a debate between those two positions, both of which are m…

I think in many ways, the continued evolution of technology will turn back the clock on progress in other areas of our lives.

In ages past, impersonating another person was not nearly as difficult as it is today. Advances in identity verification and such have eliminated a certain class of problems. The emergence of deepfakes doesn’t necessarily introduce “new” problems, it just resets the progress on some very old ones.

Assuming this is solved not by banning the tech but by making progress elsewhere, some pretty worrisome implications come with that. One antidote to deepfakes would be even more progress on identity verification and tracking the source of digital media for purposes of authentication / validity.

It’s not hard to imagine the negatives in such improvements at a time where we’re already tracked far too much.

I think a more ideal development would be a return to pre-tech habits. Meeting people in person. Less reliance on virtual communication, etc. But one need only look at the backlash against returning to the office to see that such a move would require something truly existential.

I suspect you are right in the long run, but I’m curious what other factors or forces might nullify the threat introduced by this tech in the short term, or if things really have to run off the rails before people decide that fundamental changes in how we interface with tech are required.

Re: Deepfake Offensive Toolkit (real-time deepfakes for virtual cameras)

#243

Post by [deleted] »

sva_ wrote: post #114

To those who ask about the ethics of releasing something like this, I'd say that this technology already exists, and bad actors probably already can get access if they really want to and are sophisticated enough. Making this available to the general public will spread awareness of the existence of such tools, and can then possibly have a preventive effect.

[deleted]

Re: Deepfake Offensive Toolkit (real-time deepfakes for virtual cameras)

#244

Post by gambler »

sva_ wrote: post #114

To those who ask about the ethics of releasing something like this, I'd say that this technology already exists, and bad actors probably already can get access if they really want to and are sophisticated enough. Making this available to the general public will spread awareness of the existence of such tools, and can then possibly have a preventive effect.

The entirety of deep fake technology was developed mostly in mainstream academia using "raising awareness" as an excuse. Paper after paper, model after model, repository after repository. Every single time the excuse was "if we don't do it, someone else will". This was going on for years and the explanation is absolutely laughable. Without countless human-hours put into this by academia, it's pretty obvious that this technology would be nowhere near its current state. Maybe some select military research agencies could develop something analogous. Currently this is accessible to literally every crook and prankster with internet access.

Also, the notion that "raising awareness" is going to prevent deep fakes from being used in practice shows complete and utter disconnect from reality. Most people who are skeptical are already aware how imminently fakeable all the media really is. Most people who still are unaware will remain so, no matter how many GitHub repositories some dipshits will publish.

Re: Deepfake Offensive Toolkit (real-time deepfakes for virtual cameras)

#245

Post by pbhjpbhj »

jacquesm wrote: post #149
pbhjpbhj wrote: post #131

Earlier quoted context omitted.

Isn't the point to prove physical access to the camera. As in, this stream originated at this TPM which was sold as part of this camera. So, the best you get is that the stream shows it wasn't produced with access to a particular camera. Then impersonating a YouTuber, say, requires access to their physical camera.

> Isn't the point to prove physical access to the camera. Yes. But since you can't really prove who had and who did not have physical access to the camera (what are you going to use, video evidence ;) ) that carries little weight. Let's say we're talking about a death penalty or a multi billion $ lawsuit, the stakes would be so high that to rely on something as trivial as a signature on a video stream would be way to…

TPM is supposed to be a secure enclave that's resistant to electronic access, so non-duplicable (in theory). So as much as anything, you're pricing access by the signing of a file with the TPM. That means continuity of 'proof' -- perhaps we'll get to the point where famous people perpetually film themselves using a third-part verifiable module to sign the data, just so they can show "that wasn't me" (ie technological deniability).

I don't think you can show the reverse (it was you), nor can you make a 'proof' without an established chain (you're only proving that you verified something with the same equipment you previously used).

Reputation seems like it could become much more a part of social media, like, this video was signed by all these witnesses to endorse it as a true account.

Re: Deepfake Offensive Toolkit (real-time deepfakes for virtual cameras)

#246

Post by trasz »

mod wrote: post #66
mjbeswick wrote: post #50

Earlier quoted context omitted.

The difference is that guns are literally deigned for killing people. There are nonlethal alternatives, so when people buy guns (the exception being hunting) the purpose it to be able to kill someone.

There are non-lethal alternatives to devices "literally deigned [sic] for killing people?" This logically does not compute. I guess you mean in situations where guns are currently used, other strategies could be employed? Since the goal of a firearm is to hurl a chunk of metal through the air at high speed, I can't really think of alternatives. Besides air rifles, we just have traditional powder guns filling that nic…

>The usefulness of firearms extends beyond killing people

Not really, not for the vast majority of them. For hunting or sport shooting a bolt action makes sense, but a 9mm pistol is a tool specifically designed to kill people, just like an automatic rifle is.

Re: Deepfake Offensive Toolkit (real-time deepfakes for virtual cameras)

#247

Post by nonbirithm »

mastersummoner wrote: post #167
rolisz wrote: post #94

Earlier quoted context omitted.

For nuclear weapons, even if plans were publicly available, getting the right kind of uranium would still be a major hurdle for 99.9% of the people.

But what if they weren't? What if one could create something with similar destructive force with materials easily accessible, and the only hurdle is knowledge of how to do so? It's honestly an interesting moral question.

This is explored further in Nick Bostrom's Vulnerable World Hypothesis. He believes that not even a total global surveillance system would be able to stop a single teenager with the recipe for a homemade pathogen. Nothing changes about the laws of the universe in that case, only the fact that the information was not in our minds before and then it is, and we collectively cannot forget or uninvent it.

We as a species have only survived ~75 years since we became aware of nuclear weapons, which is only a small portion of our overall history.

This has taught me to be mindful of the state of being unaware, because once you are aware of something it is impossible to reverse unless you're an amnesiac. Collectively forgetting something as a society then becomes infeasible without the technology and the willingness to revert our knowledge to a safer state, and a single outlier can ruin the entire system. When Jobs held up the first smartphone and we all started cheering and fantasizing, did any of us think it would lead to this?

(I also think that a pill to forget the experience of [addictive thing] would be revolutionary, but sadly our ingrained curiosity might undo the effect shortly afterward. A lot of tech takes advantage of our curiosity.)

This makes me wonder: what compels us to keep researching AI despite us also being able to hypothesize all these game-over scenarios? Why does it feel like we have no choice but to be crushed by unbounded progress?

Perhaps 50 years from we may find that nothing short of halting all research in the name of curiosity was the only option that could have preserved our culture/existence/values for another few decades.

Re: Deepfake Offensive Toolkit (real-time deepfakes for virtual cameras)

#248

Post by warble »

Planktonne wrote: post #40
wdroz wrote: post #13

Earlier quoted context omitted.

> If you create something, make it freely accessible and easy to use, then I think you are partly responsible for its misuse. I don't agree, but people will never reach consensus on this moral topic. So please don't call the other side "at best naive".

If there's no duty to consider potential harm when releasing tools, then why would there be a duty to avoid criticising people who release them?

Considering potential harm is one thing, being liable for misuse is another. If you're releasing something that can only harm others and has a arguable net downside, that may be a good reason to not release it.

What we're talking about (I think) is releasing something with a net upside, but with potential for misuse and the resulting liability?

Am I misunderstanding your point?

Re: Deepfake Offensive Toolkit (real-time deepfakes for virtual cameras)

#249

Post by franga2000 »

ryandrake wrote: post #217
ethanbond wrote: post #178

Earlier quoted context omitted.

This trend is clearly real, but what exactly do you mean “work out how to deal with that?” One of the obvious ways to deal with it, albeit imperfectly, is in fact to keep these technologies as obscure as possible for as long as possible. Maybe an interesting norm to try to generate in this community would be, “if you plan on publishing attack X, you should publish immediately alongside it your best guess as to the ef…

Even an extreme example -- The technology to build an automated drone with a camera for targeting and a gun on a gimbal and deploy it to do a mass-shooting -- all exists today. All it would take is for some madman to pick the pieces up and assemble them. In fact, I'd be willing to bet $100 that we'll actually see some kind of tragedy to that effect in the USA within the next 5 years. The technology cats are very quic…

Many airsoft players are mounting fully automatic airsoft rifles on drones and some Russian dude already put a glock on a 1st gen DJI phantom years ago. My guess is a big thing stopping such an attack from happening is that the people doing these attacks are usually not well educated, so they don't have the arduino-starter-kit level of electronics knowledge required to pull it off.
Post reply on HN