Live data from Hacker News

AI-Generated Voice Evidence Poses Dangers in Court

lawfaremedia.org

121–130 of 171 posts

Re: AI-Generated Voice Evidence Poses Dangers in Court

#121
post #12

So... those talking head "influencers" who leave multiple hours of voice and video samples on social networking for anyone to download and clone are the most at risk for an attack like this?

Not necessarily. Recent advancements suggest that just a few soundbites of a typical person's voice already allow modern AI to synthesize it fairly accurately.

Re: AI-Generated Voice Evidence Poses Dangers in Court

#122
post #89

Earlier quoted context omitted.

Largely true, although submitting timestamp hashes to the blockchain is probably the easiest bit. "Camera looking at a monitor": While that might be simpler in some setups, it doesn’t really solve my main issue. I want the signal to permeate the entire scene, not just appear in the corner of a display or overlaid on the video. By projecting the challenge onto all visible surfaces, we create a physical environment tha…

"The blockchain" is just a wasteful way of doing things compared to the plain crypto. "Overlay the entire scene" doesn't actually appear to add any information-theoretic value compared to simply bounding the timestamp at which the video was made. Nothing either of us is talking about will actually prevent fakes (before the camera signs it), only constrain the time at which the fakes are made. Slower-than-real-time ge…

I’m using a blockchain because it’s currently the most practical Byzantine Fault Tolerant (BFT) network available for timestamping the Initialization Vector and final hash. If you prefer a web-of-trust or another cryptographic framework, feel free to use that instead.

By projecting across the entire scene, we significantly increase the complexity of any real-time forgery attempt. A single display that barely interacts with the surroundings is comparatively easy to spoof and not particularly convenient to deploy. Of course, if you think that simpler setup is worth exploring, go ahead and experiment!

Keep in mind, this method differs from merely signing an image. It creates an optical puzzle that reality solves faster than a simulation can. The physical interaction with the environment adds complexity that’s difficult to replicate artificially. I tried placing a display in front of the camera about a decade ago, but found that approach too “analytically solvable,” so it lost its appeal for robust authentication.

Re: AI-Generated Voice Evidence Poses Dangers in Court

#123
post #51

Earlier quoted context omitted.

The point is the camera maker certifies in court under perjury penalty that their cameras are not compromised and that is their image. "Other camera systems are compromised, but ours is not...".

Cameras are an IoT device. The S is for security. What good is having the company certify something that isn't true? It seems like it would just be a mechanism to lend undeserved credibility to something that still isn't trustworthy.

Here the camera manufacture is asserting under oath that they (not their competitors) are secure. Just because IoT is almost always insecure doesn't mean it 100% is, being the exception can allow you to charge a premium price at times - but you better really be the exception.

Re: AI-Generated Voice Evidence Poses Dangers in Court

#124
post #17

Earlier quoted context omitted.

I don't know what the latest is, but often judges are supposed to not allow "expert testimony" without checking that the person is an expert. However this is a really complex area. Judges don't want to be the one deciding a case, but not allowing some "expert" is in a way deciding the case.

There’s a built-in design paradox. How does a judge assess an expert in a field where he is not one? There’s probably some improvement that comes from experience but it’s not perfect.

Opposing lawyers have an incentive to help the judge. Of course the lawyers will lie, but they still will point out important details for the judge to look at.

Re: AI-Generated Voice Evidence Poses Dangers in Court

#125

Earlier quoted context omitted.

Have you ever heard some of the wiretap or hidden microphone recordings used to convict mafia bosses back in the 1980s? It was so bad I can't believe it was accepted. It could easily have been faked. The only thing that made it work was the sworn statements of authenticity from the people who did the recording, and the chain of custody thereafter.

I know a lawyer who was convicted on one of those. The detectives had the mic + transmitter taped to his groin so it would get through a pat down. Then the undercover just spoke both sides of the conversation (the defendant's side was whispered and muffled). Someone testified it sounded like the defendant. Conviction was overturned on appeal on a simpler issue unrelated to the evidence. In fact, I remember a drug dea…

So what happened to him?

Re: AI-Generated Voice Evidence Poses Dangers in Court

#126

Here's my plan. 1. Cryptographically hash each piece of media when it's recorded. 2. Submit the hash to a "trusted" authority. 3. It will add a timestamp and sign the result. 4. Now, as long as you keep the original, without re-compressing, and you trust the authority, you have some evidence that the media existed at a timestamp. On or before. This doesn't prove authenticity, but in many cases, establishing a timesta…

Thieves are planning an inside job. They forge the surveillance video ahead of time, do the theft and submit the forgery to be timestamped while it's happening. Also, a lot of surveillance systems are purposely kept offline to prevent them from being compromised, but your system doesn't allow that because they would need external connectivity to get signatures.

This technique is known as an Ella Fitzgerald to those in the heisting industry, as popularized in the documentary Oceans 11.

Re: AI-Generated Voice Evidence Poses Dangers in Court

#127
post #32
post #23

I'll say it again, even though it is rather unpopular here: there has never been a need to develop these tools, nor one to make them easy to deploy, nor one to make them easy to use. Yet all this has happened, and now it may occur that someone is acquitted because AI generated media is so good, the evidence might be artificial. If that happens, and the suspect commits another crime, it's on the conscience of the peop…

I do want a computer which talks to me like a person, with that agility. I am used to listening to voice. I am tired of reading so much stuff which could well be spoken to me. Like this comment here, which you now need to read. It's only very recently that living beings on this world have learned to read and write, it's not normal. It's normal to communicate through sound.

>It's only very recently that living beings on this world have learned to read and write, it's not normal. It's normal to communicate through sound.

It's normal for humans to communicate via sight and sound combined.

That being said, we invest a lot of calories and brain power into vision. It's our primary mode of interacting with the planet. This is what we have evolved to rely on.

I think, therefore, you're wrong. We rely heavily on eyesight, naturally. It's why our eyes are focused on the front, allowing precise binocular vision, whereas our ears are on the side, allowing for broader coverage, but less specific information received. Reading and writing (or at least some form of image communication) is probably more natural for communicating ideas than talking, for humans.

Even if you are still trying to average us out across other life on earth (which doesn't really make a bit of sense), I think vision is the way to go. Colors and visual attractants and other visual forms of communication are common even in plants, as well as animals.

Re: AI-Generated Voice Evidence Poses Dangers in Court

#128
post #6

Earlier quoted context omitted.

I agree. The article didn't touch on this aspect, but we're now at the point where even authentic recordings could be plausibly denied and claimed to be fake. So the entire usage of recordings as evidence will suffer a hit. We may essentially be knocked back to an 18th century level of reliance on eyewitness testimony. One wonders what the consequences for justice will be.

I wouldn't say we'd be quite back to pre-photo evidence days. I feel a lot if not most of the value in a video/audio recording is not just that the medium has traditionally been difficult to edit, but that it's attesting to a lot of details with high specificity. There's a lot to potentially get caught out on with not a lot of wiggle room when inconsistencies are spotted (compared to recalling from memory). Document…

I think forensic analysis of photographs would reveal pretty quickly if they’re AI anyway.

AI images of real life still don’t pass intense scrutiny.

Re: AI-Generated Voice Evidence Poses Dangers in Court

#129
post #128
post #6

Earlier quoted context omitted.

I wouldn't say we'd be quite back to pre-photo evidence days. I feel a lot if not most of the value in a video/audio recording is not just that the medium has traditionally been difficult to edit, but that it's attesting to a lot of details with high specificity. There's a lot to potentially get caught out on with not a lot of wiggle room when inconsistencies are spotted (compared to recalling from memory). Document…

I think forensic analysis of photographs would reveal pretty quickly if they’re AI anyway. AI images of real life still don’t pass intense scrutiny.

And we have been able to edit photos nearly as long as we have been able to make them. Yet photo evidence continues being useful

Re: AI-Generated Voice Evidence Poses Dangers in Court

#130
post #128
post #6

Earlier quoted context omitted.

I wouldn't say we'd be quite back to pre-photo evidence days. I feel a lot if not most of the value in a video/audio recording is not just that the medium has traditionally been difficult to edit, but that it's attesting to a lot of details with high specificity. There's a lot to potentially get caught out on with not a lot of wiggle room when inconsistencies are spotted (compared to recalling from memory). Document…

I think forensic analysis of photographs would reveal pretty quickly if they’re AI anyway. AI images of real life still don’t pass intense scrutiny.

It's just another thing you'll have to pay for as the defendant.
Post reply on HN