Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI-Generated Voice Evidence Poses Dangers in Court

lawfaremedia.org

Post reply on HN
Print view View original

81–90 of 171 posts

Re: AI-Generated Voice Evidence Poses Dangers in Court

#81

Post by gowld »

qwertox wrote: post #32
tgv wrote: post #23

I'll say it again, even though it is rather unpopular here: there has never been a need to develop these tools, nor one to make them easy to deploy, nor one to make them easy to use. Yet all this has happened, and now it may occur that someone is acquitted because AI generated media is so good, the evidence might be artificial. If that happens, and the suspect commits another crime, it's on the conscience of the peop…

I do want a computer which talks to me like a person, with that agility. I am used to listening to voice. I am tired of reading so much stuff which could well be spoken to me. Like this comment here, which you now need to read. It's only very recently that living beings on this world have learned to read and write, it's not normal. It's normal to communicate through sound.

You don't need writing, if you want to live in a cave or wander a prairie, I guess?

Re: AI-Generated Voice Evidence Poses Dangers in Court

#82

Post by triceratops »

qwertox wrote: post #32
tgv wrote: post #23

I'll say it again, even though it is rather unpopular here: there has never been a need to develop these tools, nor one to make them easy to deploy, nor one to make them easy to use. Yet all this has happened, and now it may occur that someone is acquitted because AI generated media is so good, the evidence might be artificial. If that happens, and the suspect commits another crime, it's on the conscience of the peop…

I do want a computer which talks to me like a person, with that agility. I am used to listening to voice. I am tired of reading so much stuff which could well be spoken to me. Like this comment here, which you now need to read. It's only very recently that living beings on this world have learned to read and write, it's not normal. It's normal to communicate through sound.

Reading and writing isn't "normal" but using a computer is? Humans have been doing the former for way longer.

Re: AI-Generated Voice Evidence Poses Dangers in Court

#83

Post by PolieBotics »

asddubs wrote: post #80
PolieBotics wrote: post #79

I've developed a novel approach to creating tamper-evident video via cryptographic feedback loops between projectors and cameras. The process works as follows: 1. A projector displays a challenge pattern (Perlin noise derived from of a hash) 2. A camera captures this projection 3. The system hashes the captured image concatenated with the previous hash and uses it to derive the next projection 4. This chain demonstra…

pretty clever, kind of like cipher block chaining

Thank you! It is indeed a little like a signature based on proof-of-projection.

As they say, once you have a signature, you have a most of a cryptosystem. I've been experimenting with those and other applications of non-linear functions in projector-camera systems.

https://github.com/poliebotics/PolieBotics

Re: AI-Generated Voice Evidence Poses Dangers in Court

#84

Post by stogot »

zusammen wrote: post #68
bluGill wrote: post #17

Earlier quoted context omitted.

I don't know what the latest is, but often judges are supposed to not allow "expert testimony" without checking that the person is an expert. However this is a really complex area. Judges don't want to be the one deciding a case, but not allowing some "expert" is in a way deciding the case.

There’s a built-in design paradox. How does a judge assess an expert in a field where he is not one? There’s probably some improvement that comes from experience but it’s not perfect.

Qualifications=academics usually & career experience. No way to know if they actually learned something and aren’t a fraud. Even corporations that do thousands of interviews get duped

Re: AI-Generated Voice Evidence Poses Dangers in Court

#85

Post by o11c »

PolieBotics wrote: post #79

I've developed a novel approach to creating tamper-evident video via cryptographic feedback loops between projectors and cameras. The process works as follows: 1. A projector displays a challenge pattern (Perlin noise derived from of a hash) 2. A camera captures this projection 3. The system hashes the captured image concatenated with the previous hash and uses it to derive the next projection 4. This chain demonstra…

There are actually (at least) 3 different places where cryptography is needed here:

* Proof that this starts after a given time. Traditionally this has used methods like "this is the headline of a major newspaper today", which is limited to 1-day granularity and has problems if you can just generate a large number of expected headlines and use them in parallel. But with crypto, we can just query any random-number-timestamp-signing server, and a network of such servers can mutually sign each other's previous packets so it's very reliable both against downtime and against attacks.

* Proof of sequencing. This is trivial with a chain of hashes, though it does prevent recompression.

* Proof that this ends before a given time. This requires actively submitting your signature data to a timestamp server for additional signing, which is a much more complicated task than the initial half. It is still possible to eliminate the single source of vulnerability, but much more work.

"Camera looks at monitor" is going to be a much cheaper way to make this air-gapped than adding a projector. And this doesn't strictly need to be continuous; most things are tolerant of one-day granularity and almost everything of 15-minute granularity.

Re: AI-Generated Voice Evidence Poses Dangers in Court

#86

Post by hippari2 »

adiabatichottub wrote: post #46
LPisGood wrote: post #39

Earlier quoted context omitted.

This is unironically a usecase for blockchain.

Who needs a whole blockchain? Just basic public-key cryptography would do the job. Imagine if you will, that the NVR (recording system) has a unique private key flashed in during manufacturing, with the corresponding public key printed on it's nameplate. The device can sign a video clip and its related meta-data before exporting. Now, any decent hacker could see possible holes in this system, but it could be made tam…

Unfortunately consumer devices often have weak cryptography built into them. The one properly implemented are just out of reach for average consumer.

Re: AI-Generated Voice Evidence Poses Dangers in Court

#87

Post by bilbo0s »

nostrademons wrote: post #73
bilbo0s wrote: post #69

Earlier quoted context omitted.

Keep in mind that being able to demonstrate your innocence to the government is also just an individual change in behavior. A person might not necessarily care if you use the tracking technologies or submit yourself to all the cameras in society. But they'll choose to do it themselves just so that there's that record out there. The government coming up with one way to track everyone is not really necessary to get peo…

I think the issue demonstrated by this article is that technology is getting such that demonstrating your innocence to the government is not an individual change of behavior, and that there are ways to abuse the demonstration mechanisms that effectively feed false information to the government. Imagine that the populace supports widespread surveillance techniques, and so cameras are setup everywhere. Some hacker grou…

Which speaks to why the law enforcement types prefer multiple surveillance and tracking channels. A hacker group compromising one may be possible, but to get away with a crime, it would be necessary to compromise the feeds coming from all of the channels at the same time. Extremely unlikely. The deep fake of someone else robbing a store that you put on the store's security camera system is nice. But the police are likely to be more convinced by the other ten thousand smart phones, smart glasses, and security and door bell cameras in the neighborhood that recorded you and your buddies running out of that store with guns at the time of the robbery.

This is how the "watcher" types are trained. Information is only valid if they can get it from multiple independent sources. So they love when new tracking and surveillance channels are released. (Social media apps, or smart glasses, or doorbell cameras or what have you.)

The bar would be much higher than compromising a single channel. With multiplying channels, the task of compromising them all approaches impossibility.

Re: AI-Generated Voice Evidence Poses Dangers in Court

#88

Post by PolieBotics »

o11c wrote: post #85
PolieBotics wrote: post #79

I've developed a novel approach to creating tamper-evident video via cryptographic feedback loops between projectors and cameras. The process works as follows: 1. A projector displays a challenge pattern (Perlin noise derived from of a hash) 2. A camera captures this projection 3. The system hashes the captured image concatenated with the previous hash and uses it to derive the next projection 4. This chain demonstra…

There are actually (at least) 3 different places where cryptography is needed here: * Proof that this starts after a given time. Traditionally this has used methods like "this is the headline of a major newspaper today", which is limited to 1-day granularity and has problems if you can just generate a large number of expected headlines and use them in parallel. But with crypto, we can just query any random-number-tim…

Largely true, although submitting timestamp hashes to the blockchain is probably the easiest bit.

"Camera looking at a monitor": While that might be simpler in some setups, it doesn’t really solve my main issue. I want the signal to permeate the entire scene, not just appear in the corner of a display or overlaid on the video. By projecting the challenge onto all visible surfaces, we create a physical environment that’s difficult to fake (since you’d have to convincingly generate or remove those patterns in real time). Air-gapping isn’t really the goal right now.

Finally, we're need much finer granularity than 15 minutes! The point is to lower the generation time below what is achievable via generative model.

Thank you for the comment, and I hope these clarifications are useful. It's a new concept, so please forgive the clumsiness with which I may be communicating it.

Re: AI-Generated Voice Evidence Poses Dangers in Court

#89

Post by o11c »

PolieBotics wrote: post #88
o11c wrote: post #85

Earlier quoted context omitted.

There are actually (at least) 3 different places where cryptography is needed here: * Proof that this starts after a given time. Traditionally this has used methods like "this is the headline of a major newspaper today", which is limited to 1-day granularity and has problems if you can just generate a large number of expected headlines and use them in parallel. But with crypto, we can just query any random-number-tim…

Largely true, although submitting timestamp hashes to the blockchain is probably the easiest bit. "Camera looking at a monitor": While that might be simpler in some setups, it doesn’t really solve my main issue. I want the signal to permeate the entire scene, not just appear in the corner of a display or overlaid on the video. By projecting the challenge onto all visible surfaces, we create a physical environment tha…

"The blockchain" is just a wasteful way of doing things compared to the plain crypto.

"Overlay the entire scene" doesn't actually appear to add any information-theoretic value compared to simply bounding the timestamp at which the video was made. Nothing either of us is talking about will actually prevent fakes (before the camera signs it), only constrain the time at which the fakes are made.

Slower-than-real-time generation of fakes is still significantly inhibited by the fact that the hash sequence can be checked for continuity across long lengths of time whose bounds are verified using the other steps.

Re: AI-Generated Voice Evidence Poses Dangers in Court

#90

Post by dghlsakjg »

Majromax wrote: post #50
dghlsakjg wrote: post #27

Earlier quoted context omitted.

I understand what you are saying, but my point is that I could make plausible sounding recordings that did not reflect reality by, for example, cutting recordings up with freeware like Audacity, or even using a consumer level double tapedeck before that. It wasn't Manhattan project levels of effort before this. This seems more like people losing their minds over 3d printed guns, when hobbyists with a drill press have…

> I could make plausible sounding recordings that did not reflect reality by, for example, cutting recordings up with freeware like Audacity, Cutting up what recordings, exactly? If you're mixing-and-matching, you need a pretty broad corpus of source material with fairly consistent recording quality (background noise, etc), and even still you're limited to reproducing words that are already present. You can't cut-and…

I'm giving limited examples, not a conclusive list of how audio could be manipulated. I understand that there are circumstances where my examples don't work.

My broader point is that audio evidence has never been perfect, just like every other medium that evidence can exist in.

Generative audio technology is not the first time that audio has been corruptible as evidence, and any lawyer who believed otherwise was naive at best.

Post reply on HN