Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI-Generated Voice Evidence Poses Dangers in Court

lawfaremedia.org

Post reply on HN
Print view View original

71–80 of 171 posts

Re: AI-Generated Voice Evidence Poses Dangers in Court

#71

Post by condiment »

AnthonyMouse wrote: post #64
iamacyborg wrote: post #40

Earlier quoted context omitted.

Surely chain of custody applies if the accused has access to the evidence? Perhaps I’m missing your point or I’m overly optimistic about the legal system.

Suppose the store manager is having a dispute with a kid who keeps skateboarding in the parking lot, so the store manager decides to commit insurance fraud by robbing the store herself and then submits forged video of the kid doing it to the police. The store manager is in the chain of custody but isn't a suspect, the accused is the kid. The kid doesn't even know who actually committed the crime. How is the kid suppo…

In this case, chain of custody needs to extend to the capture device itself, and to any software that exists in the supply chain for the video content.

There are some experimental specifications that exist to provide attestation as to the authenticity of media. But most of what I’ve seen so far is a “perjury based” approach that just requires a human to say that something is authentic.

Re: AI-Generated Voice Evidence Poses Dangers in Court

#72

Post by gus_massa »

Majromax wrote: post #50
dghlsakjg wrote: post #27

Earlier quoted context omitted.

I understand what you are saying, but my point is that I could make plausible sounding recordings that did not reflect reality by, for example, cutting recordings up with freeware like Audacity, or even using a consumer level double tapedeck before that. It wasn't Manhattan project levels of effort before this. This seems more like people losing their minds over 3d printed guns, when hobbyists with a drill press have…

> I could make plausible sounding recordings that did not reflect reality by, for example, cutting recordings up with freeware like Audacity, Cutting up what recordings, exactly? If you're mixing-and-matching, you need a pretty broad corpus of source material with fairly consistent recording quality (background noise, etc), and even still you're limited to reproducing words that are already present. You can't cut-and…

I agree.

I edited the audio of a few math videos to upload to YouTube and I had to fix "typos" like "then one plus zero is zero". The problem is that there are no spaces in the voice, so it sounds like "thenonepluszeroiszero."

So I had to look for a "one" that is in a similar context, after an "s" and before a ".", or at least a similar one. And hope the speed matches, and adjust the volume.

They were free videos, so it was not necesary to get it perfect, but at least reduce the distraction caused by mistakes.

Re: AI-Generated Voice Evidence Poses Dangers in Court

#73

Post by nostrademons »

bilbo0s wrote: post #69
nostrademons wrote: post #62

Earlier quoted context omitted.

Government is based on the threat of the use of force, not the actual use of force. If you're a government that is regularly using force against a significant proportion of your citizens, you have problems, and probably will not remain the government for long. I suspect that we're saying the same thing but with reversed causality. Both of us agree that non-deterministic enforcement breaks down the incentives needed f…

Keep in mind that being able to demonstrate your innocence to the government is also just an individual change in behavior. A person might not necessarily care if you use the tracking technologies or submit yourself to all the cameras in society. But they'll choose to do it themselves just so that there's that record out there. The government coming up with one way to track everyone is not really necessary to get peo…

I think the issue demonstrated by this article is that technology is getting such that demonstrating your innocence to the government is not an individual change of behavior, and that there are ways to abuse the demonstration mechanisms that effectively feed false information to the government.

Imagine that the populace supports widespread surveillance techniques, and so cameras are setup everywhere. Some hacker group figures out how to hack into the cameras and insert deepfakes in them. Now members of that hacker group have a government-proof alibi whenever they want it, and can commit crimes at will, and get it blamed on others. Justice goes out the window.

Re: AI-Generated Voice Evidence Poses Dangers in Court

#74

Post by AyyEye »

mitthrowaway2 wrote: post #3
fhd2 wrote: post #2

Sounds like reasonable changes. Generally speaking, I think evidence tampering is not a new problem, and even though it's easy in some cases, I don't think it's _that_ widespread. Just like it's possible to lie on the stand, but people usually think twice before they do it, because _if_ they are found to have lied, they're in trouble. My main concern is rather that legit evidence can now easily be called into questio…

I agree. The article didn't touch on this aspect, but we're now at the point where even authentic recordings could be plausibly denied and claimed to be fake. So the entire usage of recordings as evidence will suffer a hit. We may essentially be knocked back to an 18th century level of reliance on eyewitness testimony. One wonders what the consequences for justice will be.

> we're now at the point where even authentic recordings could be plausibly denied and claimed to be fake.

We've been there for at least two years.

https://arstechnica.com/tech-policy/2023/04/judge-slams-tesl...

Re: AI-Generated Voice Evidence Poses Dangers in Court

#75

Post by MichaelDickens »

gortok wrote: post #52

What shocks (and irritates!) me is that Charles Schwab keeps wanting me to set up voice ID. Why would I want to set up a voice ID for something that is now trivially spoofed?

Schwab used to have an 8 character maximum on passwords (although at least they changed that). They have never been a paragon of good security practices.

Re: AI-Generated Voice Evidence Poses Dangers in Court

#76

Post by _DeadFred_ »

condiment wrote: post #71
AnthonyMouse wrote: post #64

Earlier quoted context omitted.

Suppose the store manager is having a dispute with a kid who keeps skateboarding in the parking lot, so the store manager decides to commit insurance fraud by robbing the store herself and then submits forged video of the kid doing it to the police. The store manager is in the chain of custody but isn't a suspect, the accused is the kid. The kid doesn't even know who actually committed the crime. How is the kid suppo…

In this case, chain of custody needs to extend to the capture device itself, and to any software that exists in the supply chain for the video content. There are some experimental specifications that exist to provide attestation as to the authenticity of media. But most of what I’ve seen so far is a “perjury based” approach that just requires a human to say that something is authentic.

Chain of custody isn't real as long as the judiciary gives the government a 'good faith' pass when chain of custody isn't maintained/documentable in court. Go into Lexus Nexus and look up 'good faith' related to 'chain of custody'. Any 'protections' that can be waived away at the judges whim when the standard isn't met by the government are not actually real but pure theater to lend legitimacy to the American judicial system that it doesn't deserve.

Re: AI-Generated Voice Evidence Poses Dangers in Court

#77

Post by MichaelDickens »

qwertox wrote: post #58
krainboltgreene wrote: post #35

Earlier quoted context omitted.

What an incredible line to end on. “It’s not normal to read/write”.

I'm not aware of any other species which is capable of reading and writing. But many can certainly hear and make sounds.

I'm not aware of any other species with a >10,000 word vocabulary, either.

Re: AI-Generated Voice Evidence Poses Dangers in Court

#78

Post by gs17 »

AnthonyMouse wrote: post #36
bluGill wrote: post #20

Earlier quoted context omitted.

Or perhaps camera manufactures will start putting traces in. Anyone who makes surveillance systems (like stores use) should put some end to end things in so they can say "this camera took that recording and we can see that it wasn't tampered with by...". (if anyone works for a surveillance company please run with this!) With encryption we can verify a lot of things, but it sets the bar higher than someone took a pict…

Cryptography doesn't really fix it. There are a zillion camera makers and all it takes is one of them to have poor security and leak the keys. Then the forger uses any of the cameras with extractable keys to sign their forgery. Or they just point a camera at a high resolution playback of a forged video. This also assumes you can trust all the camera makers, because by doing this you're implicitly giving them each aut…

> Or they just point a camera at a high resolution playback of a forged video.

Exactly, it's just like DRM or cheating in video games. Even if everything in software is blocked, there's always the analogue hole.

Re: AI-Generated Voice Evidence Poses Dangers in Court

#79

Post by PolieBotics »

I've developed a novel approach to creating tamper-evident video via cryptographic feedback loops between projectors and cameras. The process works as follows:

1. A projector displays a challenge pattern (Perlin noise derived from of a hash) 2. A camera captures this projection 3. The system hashes the captured image concatenated with the previous hash and uses it to derive the next projection 4. This chain demonstrates true temporal sequentiality that's difficult to forge

By incorporating random noise derived from Byzantine Fault Tolerant networks and using these networks as timestamping servers, the proofs inherit the network's decentralization properties. ML then confirms that the feature distributions in projection-photograph pairs match expected patterns from the training dataset.

Demo video and GitHub repo available here: https://www.reddit.com/r/PoliePals/comments/1j8qm2j/truth_be...

Re: AI-Generated Voice Evidence Poses Dangers in Court

#80

Post by asddubs »

PolieBotics wrote: post #79

I've developed a novel approach to creating tamper-evident video via cryptographic feedback loops between projectors and cameras. The process works as follows: 1. A projector displays a challenge pattern (Perlin noise derived from of a hash) 2. A camera captures this projection 3. The system hashes the captured image concatenated with the previous hash and uses it to derive the next projection 4. This chain demonstra…

pretty clever, kind of like cipher block chaining
Post reply on HN