Live data from Hacker News

Accused murderer wins right to check source code of DNA testing kit

theregister.com

231–240 of 520 posts

Re: Accused murderer wins right to check source code of DNA testing kit

#231

It's an important development. Forensic labs and companies are expert witnesses with black box processes and the incentive to protect the authority of their profession. They are as likely to lie as any other witness. Perhaps even moreso. "Those arguing on behalf of the defense cited past problems with other genetic testing software such as STRmix and FST (Forensic Statistical Tool). Defense expert witnesses Mats Heim…

> Forensic labs and companies are expert witnesses with black box processes and the incentive to protect the authority of their profession. I imagine they're also incentivised to favour a result indicating guilt.

They fill a similar role to prosecutors that field drug tests serve to cops[1]: they aren't meant to be accurate, they exist to give the cops legitimacy as they proceed to do whatever it is they wanted to do to you. In the cops' case, they need legitimacy to search and arrest you, and in the prosecutors' case, they need legitimacy to charge and convict you.

If field drug tests were actually accurate, they wouldn't be bought and used. Similarly, if an expert witness fails to give prosecutors the results and testimony that they want, they wouldn't be hired again.

[1] "How a $2 Roadside Drug Test Sends Innocent People to Jail": https://www.nytimes.com/2016/07/10/magazine/how-a-2-roadside...

Re: Accused murderer wins right to check source code of DNA testing kit

#232

Earlier quoted context omitted.

That seems plausible. I would argue that if prosecutors and/or cops get to pick the tool, false positives would result in more sales. Just as false negatives would result in more sales if the defendant gets to make the pick.

false positives would result in more sales. That seems highly unlikely. Prosecutors, like all attorneys, talk. It wouldn’t take too many getting burned by a false positive before word would get out (regardless of any nda). For example, once an expert gets Dauberted, you’ve got to think long and hard about being the next guy to use them.

A product that doesn't work wouldn't be considered.

Re: Accused murderer wins right to check source code of DNA testing kit

#233
post #90

Earlier quoted context omitted.

A tool designed to find people guilty is biased to find people guilty. As far as I know it is fairly easy to take a generic dna sequencer meant for healtcare diagnostics, and repurpose it for STR analysis. The only major difference between the healthcare versions and the forensic versions is the software i/o.

> A tool designed to find people guilty is biased to find people guilty. I don't see those particular issues make it biased, just inaccurate - it could go either way.

When running an experiment and following poor practices (i.e. p-hacking), results that fit the hypothesis will be accepted more readily and negative results will be debugged or re-ran more often.

i.e. The initial error may be randomly distributed. But the follow-up on the error will have a lot of bias.

Re: Accused murderer wins right to check source code of DNA testing kit

#234
post #160
post #149

Earlier quoted context omitted.

Of course prosecutors will favour expert witnesses that will help them get convictions. Why isn’t that obvious?

There are countries where the job of the prosecutor is to uncover the truth, not to convict someone at all costs: https://en.wikipedia.org/wiki/Inquisitorial_system

Conviction rates are surprisingly high in a lot of the first world.

Re: Accused murderer wins right to check source code of DNA testing kit

#235
These companies are disgusting. They peddle black box "models," that essentially ride the good reputation of DNA as infallible (which it is most certainly not) to get convictions on dubious or no evidence.

The way it works is that if there is a sample from a crime scene, they send it to these guys and they analyze it with their software to detect "statistical" DNA from the sample. These samples are the ones that are too crappy to actually make a definitive match -- they are a statistical match. So you say "I think Jim, Bob, and Alice were on scene," and it says "10% likelihood Jim DNA, 5% likelihood Bob DNA, 45% Alice DNA." Do you think it ever says "99% no DNA" in the sample?

It's basically Theranos, except instead of wasting $50 on a shitty blood test you get life in prison.

Ostensibly, it searches the entire DNA database for matches, and only returns a positive result if there's a positive match.

But it's a statistical model, using inputs that are crappy at best (because if it was an actual DNA match, they would send it off to in house forensics who would be able to do PCR...) and which includes inputs from circumstantial evidence as priors. Like we believe Alice was at the scene therefore if you find any statistical likelihood that this is Alice's DNA boost that.

They often run the model multiple times in a row, and use the result that the DA likes the most to enter into evidence. This is because the models return different results each time -- of course they'd say, iTs StAtiStIcaL, so they can do that...

And the source code is completely impenetrable. They argue that it's a "trade secret" that jeopardizes their ability to make future profits, so it cannot be open-sourced. These guys could have a model that just says "what percentage should the thing read, Señor D.A.?" The entire product is a sham. And because it's 170k LOC, no one has the time or the qualifications (Judges/Attorneys reading source code? Yeah right!) to review it, even if it were open source.

Pure quackery, and often times, decades-long sentences or life in prison for the defendant. These companies are pure filth worthy of the lowest revulsion. It's a wonder any convictions happen at all because of this stuff, but jurors have very inaccurate conceptions of forensic science, thanks to shit like CSI, Law and Order, etc. These companies happily play into that image and people really believe this stuff works.

Re: Accused murderer wins right to check source code of DNA testing kit

#236

Earlier quoted context omitted.

When I was a public defender I made my career by cross-examining expert witnesses. There are a dozen innocent people walking around free today because I challenged the expert despite going into the cross "with no evidence." What you call abuse, I call due process. (And your proposed solution is already part of the expert certification and questioning process in criminal trials, so it isn't a solution. Experts must te…

Testify does not equal independently certified. Also, double-blind testing is not really subject to faking. Its the standard of science. Responses to my comment are all around 'how it is now' which seems not to be very reliable. A responsible testing/certification process would address that.

You've just moved the goal posts from someone who can be crossed in court to someone who can't, which is worse, because then you have now way of verifying that the "independent" certifier is actually independent or that the certification was properly done.

You still need to allow the "independent certification" to be examined in court to validate both the independence and the certification. The means testimony on the applicable standards, and how the specific equipment used satisfied those standards. And right now, the expert witness already does the latter with respect to calibration, etc., for the equipment they used.

Re: Accused murderer wins right to check source code of DNA testing kit

#237

Earlier quoted context omitted.

> Forensic labs and companies are expert witnesses with black box processes and the incentive to protect the authority of their profession. I imagine they're also incentivised to favour a result indicating guilt.

Why? The risk to their reputation & livelihood for a false positive should temper that.

The risk to their business of negatives whether or not they are false negatives, is likely higher. The resource asymmetry between a tax payer funded prosecution and a defendant funded defense is enormous.

Re: Accused murderer wins right to check source code of DNA testing kit

#238
post #205

Earlier quoted context omitted.

The defendant has a right to “confront” their accuser. So when an algorithm is the one doing the accusing it seems only fair that the defendant should be able to understand how and why the algorithm arrived at that conclusion. Running the analysis through every software package available wouldn’t satisfy the defendant’s 6th amendment rights.

It wouldn't satisfy them, but if one of the software packages gave a different answer they can't both be correct.

Or worse, they could give the same incorrect answer.

Re: Accused murderer wins right to check source code of DNA testing kit

#239
post #225
post #160

Earlier quoted context omitted.

There are countries where the job of the prosecutor is to uncover the truth, not to convict someone at all costs: https://en.wikipedia.org/wiki/Inquisitorial_system

I doubt there's any system in the world where it's officially the job of a prosecutor to "convict someone at all costs". It certainly isn't in the US. Nevertheless....

Most of the very real dynamics that affect all of our lives at a policy-making / systemic level are never official. If you think the official stuff is a reflection of reality ... I don't know how to finish this sentence. Safe from it being "official", it is very much the job of a prosecutor in the US to convict someone at all costs. This is common knowledge.

Re: Accused murderer wins right to check source code of DNA testing kit

#240

It's an important development. Forensic labs and companies are expert witnesses with black box processes and the incentive to protect the authority of their profession. They are as likely to lie as any other witness. Perhaps even moreso. "Those arguing on behalf of the defense cited past problems with other genetic testing software such as STRmix and FST (Forensic Statistical Tool). Defense expert witnesses Mats Heim…

> Forensic labs and companies are expert witnesses with black box processes and the incentive to protect the authority of their profession. I imagine they're also incentivised to favour a result indicating guilt.

This absolutely depends on the agency/situation.

My agency used to rate our agents based on the amount of cases that led to convictions. Years back, we recognized that disproving an allegation was equally worthwhile, and settings promotions/evaluations based on "proving or disproving" the allegation was much more objectively reasonable than focusing strictly on convictions. There is still a reference to case completion speed, value of recovered money/property, and conviction results (providing a thorough enough investigation that the suspect is convicted for X years, for example), but disproving an allegation or proving that a different person committed the crime is far more appreciable to the agency.

We're fortunate in that we are a federal agency, instead of local/state level, where they can be significantly more busy with countless lesser offenses, while still running some extremely high-profile stuff, but I think it would be a huge boon to law enforcement if states mandated an approach focused on this objective metric that isn't strictly on "getting him".

Post reply on HN