Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI-Generated Voice Evidence Poses Dangers in Court

lawfaremedia.org

Post reply on HN
Print view View original

111–120 of 171 posts

Re: AI-Generated Voice Evidence Poses Dangers in Court

#111

Post by echoangle »

HPsquared wrote: post #109
recursive wrote: post #28

Here's my plan. 1. Cryptographically hash each piece of media when it's recorded. 2. Submit the hash to a "trusted" authority. 3. It will add a timestamp and sign the result. 4. Now, as long as you keep the original, without re-compressing, and you trust the authority, you have some evidence that the media existed at a timestamp. On or before. This doesn't prove authenticity, but in many cases, establishing a timesta…

Isn't this basically achieved by standard cloud storage on phones? Photos and videos get uploaded automatically (with authentication for the user account) and presumably that action is logged somewhere the user can't edit. Just need to prove those logs are secure, and there you go.

Unless you’re they police in a murder case, you’re probably not going to get google or apple to give you something that certifies that a file was uploaded before a specific time and not modified after. And just showing me the modification date in the UI wouldn’t convince me.

Re: AI-Generated Voice Evidence Poses Dangers in Court

#112

Post by gblargg »

bluGill wrote: post #20
carra wrote: post #7

Wait some more time and photos or even video recordings will be deemed just as dangerous. And then what? Even if there is real evidence, it will have to be discarded unless it can't be sufficiently validated. It will get very hard to prove anything.

Or perhaps camera manufactures will start putting traces in. Anyone who makes surveillance systems (like stores use) should put some end to end things in so they can say "this camera took that recording and we can see that it wasn't tampered with by...". (if anyone works for a surveillance company please run with this!) With encryption we can verify a lot of things, but it sets the bar higher than someone took a pict…

I think the best proof of authenticity is already there, in the quirks each particular camera model has in the recording encoding. It's likely that a forgery would easily be shown to have differences in encoding when compared with a video file from the same camera. It would be more difficult with an audio recorder, as you could just record the AI voice in the room. An audio expert could probably show that the acoustics of the fakery don't match where it was claimed to be recorded.

Re: AI-Generated Voice Evidence Poses Dangers in Court

#113

Post by HPsquared »

echoangle wrote: post #111
HPsquared wrote: post #109

Earlier quoted context omitted.

Isn't this basically achieved by standard cloud storage on phones? Photos and videos get uploaded automatically (with authentication for the user account) and presumably that action is logged somewhere the user can't edit. Just need to prove those logs are secure, and there you go.

Unless you’re they police in a murder case, you’re probably not going to get google or apple to give you something that certifies that a file was uploaded before a specific time and not modified after. And just showing me the modification date in the UI wouldn’t convince me.

Much easier with the account holder's consent, it's probably in the "data download" thing you can request. And probably available by subpoena. It's like phone records.

I don't mean "date modified" in the file metadata when it's uploaded (of course that could easily be spoofed before upload) but the actual "date uploaded". You know something must have been made before a certain time.

Re: AI-Generated Voice Evidence Poses Dangers in Court

#114

Post by almogo »

bdangubic wrote: post #26
tgv wrote: post #23

I'll say it again, even though it is rather unpopular here: there has never been a need to develop these tools, nor one to make them easy to deploy, nor one to make them easy to use. Yet all this has happened, and now it may occur that someone is acquitted because AI generated media is so good, the evidence might be artificial. If that happens, and the suspect commits another crime, it's on the conscience of the peop…

You cannot create something and pretend its use has nothing to do with you. has always worked with guns so it'll always work with anything else :) and guns will kill more people that AI-generated shit for foreseeable future

There is an argument that "AI-generated shit" (on social media, or otherwise) can influence elections, and thus war. The best example I can think of is https://www.bbc.com/news/world-asia-india-68918330.

Re: AI-Generated Voice Evidence Poses Dangers in Court

#115

Post by pmontra »

nine_k wrote: post #14
mitthrowaway2 wrote: post #3

Earlier quoted context omitted.

I agree. The article didn't touch on this aspect, but we're now at the point where even authentic recordings could be plausibly denied and claimed to be fake. So the entire usage of recordings as evidence will suffer a hit. We may essentially be knocked back to an 18th century level of reliance on eyewitness testimony. One wonders what the consequences for justice will be.

I can easily imagine a future where video evidence is only acceptable in the form of chemically developed analog film, at resolutions that are prohibitively expensive to model, and audio recordings of any kind are not admissible as evidence at all. Signatures on paper, faxes, etc are, of course, inadmissible, too.

The point of a signature on paper is that (at least in my country) you can summon somebody in court and ask "is this your signature"? If they say it is, it is, even if it does not look much like any of their other signatures. Then there might be a suspect of false testimony or being blackmailed etc but that's not always the most important issue in a case. E.g.: if it is a contract and all the signers state that they agree with its terms, then there is no much else to discuss.

Re: AI-Generated Voice Evidence Poses Dangers in Court

#116

Post by AnthonyMouse »

condiment wrote: post #71
AnthonyMouse wrote: post #64

Earlier quoted context omitted.

Suppose the store manager is having a dispute with a kid who keeps skateboarding in the parking lot, so the store manager decides to commit insurance fraud by robbing the store herself and then submits forged video of the kid doing it to the police. The store manager is in the chain of custody but isn't a suspect, the accused is the kid. The kid doesn't even know who actually committed the crime. How is the kid suppo…

In this case, chain of custody needs to extend to the capture device itself, and to any software that exists in the supply chain for the video content. There are some experimental specifications that exist to provide attestation as to the authenticity of media. But most of what I’ve seen so far is a “perjury based” approach that just requires a human to say that something is authentic.

> In this case, chain of custody needs to extend to the capture device itself, and to any software that exists in the supply chain for the video content.

There are two major problems with this.

First, is all footage from existing surveillance systems going to be thrown out because it doesn't use this technology? Answer: No, because it would be impractical. But then nobody cares to adopt the technology because using it isn't required. How's that IPv6 transition going?

Second, that sort of thing doesn't actually work anyway. Surveillance cameras are made by the lowest bidder. Their security record is appalling. They're going to publish their private keys on github and expose buffer overflows to the public internet and leave a telnet server running on the camera that gives you a root shell with no password. Does it sound like hyperbole? Those are all things that have actually happened.

There is only one known way to prevent this from happening: Do not allow the hardware vendor to write the software. Any of the software. Instead, demand hardware documentation so that the firmware can be written by open source software people instead of lowest bidder hardware companies. This is incompatible with using the hardware vendor as the root of trust, which is a natural consequence because the hardware vendors are completely untrustworthy.

But let's suppose we find some way to do it. We'll pass a law imposing a $100 fine on any company that has a security vulnerability. Then there will never be a security vulnerability again because security vulnerabilities will be illegal; I'm assured this is how laws work. At that point the forger takes the camera and points it a a high resolution playback of the forgery, and the camera records and signs the forgery.

I kind of wish people would stop suggesting this. It's completely useless but it creates the false impression that it can be solved this way and then people stop trying to find a real solution.

Re: AI-Generated Voice Evidence Poses Dangers in Court

#117

Post by fhd2 »

EGreg wrote: post #91
fhd2 wrote: post #2

Sounds like reasonable changes. Generally speaking, I think evidence tampering is not a new problem, and even though it's easy in some cases, I don't think it's _that_ widespread. Just like it's possible to lie on the stand, but people usually think twice before they do it, because _if_ they are found to have lied, they're in trouble. My main concern is rather that legit evidence can now easily be called into questio…

I kind of want to have an LLM which takes absolutely any criticism of AI or news of it doing something bad and then generates a plausible HN comment that basically goes "I don't think it's a new problem, X has always been possible, there's nothing really new"... because that comment always appears like clockwork beneath it :)

Strangely, I'm not even offended by the notion that an AI can replace this work I apparently do :D Though my thought here was just pure optimism in the face of something bad, not an attempt to frame a bad side effect of something good as not a big deal.

When it comes to generative AI, I personally don't see a lot of good applications, but a plethora of bad ones. The only solution I could imagine would be regulation to the degree that using or distributing models with certain capabilities is just illegal. Judging from the war on file sharing a few decades ago, probably very difficult to enforce, even if it is perhaps still worth doing.

But I don't see any governments line up to do it. Given that, this particular (semi) new development that generative AI is effective for evidence tampering, I think we'll manage to deal with it.

Re: AI-Generated Voice Evidence Poses Dangers in Court

#118

Post by fhd2 »

vasco wrote: post #103
fhd2 wrote: post #2

Sounds like reasonable changes. Generally speaking, I think evidence tampering is not a new problem, and even though it's easy in some cases, I don't think it's _that_ widespread. Just like it's possible to lie on the stand, but people usually think twice before they do it, because _if_ they are found to have lied, they're in trouble. My main concern is rather that legit evidence can now easily be called into questio…

> But ultimately: Humanity has coped without photo, audio or video evidence for most of its existence. I suppose it will cope again. Same argument for electricity, the internet, sanitation, democracy... doesn't seem like a great test for stuff that we just didn't have it before and survived.

Well, I'm not arguing "good riddance". I just optimistically think we'll manage. I wouldn't want to miss any of the things you listed, actually. I'd also prefer evidence tampering to be impossible or at least very difficult. But that's not a call I can make. All I can decide about things out of my control, is how I think about it. And here I'm carefully optimistic.

Re: AI-Generated Voice Evidence Poses Dangers in Court

#119

Post by Ukv »

tgv wrote: post #23

I'll say it again, even though it is rather unpopular here: there has never been a need to develop these tools, nor one to make them easy to deploy, nor one to make them easy to use. Yet all this has happened, and now it may occur that someone is acquitted because AI generated media is so good, the evidence might be artificial. If that happens, and the suspect commits another crime, it's on the conscience of the peop…

> Nothing of value is lost when these models are made private again

To list a few uses I see for voice-generation/cloning tools:

* Real-time translation of a user's voice, maintaining emotion and intonation

* Professional-quality audio from cheap microphone setups (for video tutorials, indie games, etc.)

* Allowing those with speech impairments to communicate using their natural voice again, or:

* Allowing those uncomfortable with their natural voice to communicate closer to how they wish to be perceived

* Customization of voice assistants, such as to use a native accent/dialect

* Movies, podcasts, audiobooks, news broadcasts, etc. made available in a huge range of languages

* And of course: memes, satire, and parody

If it exists but isn't widely accessible, it's likely in the hands of Musk/Zuck and and various state actors. To me that seems possibly the worst alternative - to have the public generally unaware that it's possible and receiving few of the benefits listed, yet still having it available as a tool for competent disinformation.

Re: AI-Generated Voice Evidence Poses Dangers in Court

#120

Post by vasco »

fhd2 wrote: post #118
vasco wrote: post #103

Earlier quoted context omitted.

> But ultimately: Humanity has coped without photo, audio or video evidence for most of its existence. I suppose it will cope again. Same argument for electricity, the internet, sanitation, democracy... doesn't seem like a great test for stuff that we just didn't have it before and survived.

Well, I'm not arguing "good riddance". I just optimistically think we'll manage. I wouldn't want to miss any of the things you listed, actually. I'd also prefer evidence tampering to be impossible or at least very difficult. But that's not a call I can make. All I can decide about things out of my control, is how I think about it. And here I'm carefully optimistic.

But your argument for why we can live without audio or video evidence was that we lived like that before. All I'm saying is that's not a great argument. You can probably come up with other arguments and they might be perfectly valid.
Post reply on HN