When they warn of rare disorders, these prenatal tests are usually wrong • Hacker News Board

Re: When they warn of rare disorders, these prenatal tests are usually wrong

#61

Post by mnw21cam » Sun, Jan 02, 2022, 9:11 PM UTC

halpert wrote: post #40

ellisv wrote: post #36

Earlier quoted context omitted.

I disagree. It is clear from the title, “When They Warn of Rare Disorders, These Prenatal Tests Are Usually Wrong”, and the lead that they’re focusing on false positives.

It's true they are focusing on false positives, but the authors are using the ratio of false positives to true positives to paint a picture that the tests are inaccurate, when in reality the tests are accurate. What this article is looking at is called the "sensitivity" of a test: https://en.wikipedia.org/wiki/Sensitivity_and_specificity

No, the article isn't talking about sensitivity. We don't actually know what the sensitivity is from the data the article gives us. We are told that lots of people were screened and a small number had a positive result, of which a proportion were actually positive. You can't calculate sensitivity from that because you don't know how many actually positive cases were missed.

This article is talking about precision, which is the proportion of positive results that are true. And it's okay for precision to be awful, especially when the condition is so rare. But it's only okay if the result is communicated alongside a statement saying what the precision is, which it seems these were not.

Re: When they warn of rare disorders, these prenatal tests are usually wrong

#62

Post by mnw21cam » Sun, Jan 02, 2022, 9:15 PM UTC

andreilys wrote: post #56

isoprophlex wrote: post #41

Earlier quoted context omitted.

Going through something like this is very VERY stressful. When you get a negative you immediately forget about it. When you get a positive you die inside. Speaking from experience here. 84% wrong sounds, to me, as an accurate description. Experiencing this from the inside out, only the false/true positive ratio matters. (Given sufficiently low false negative rates, of course) 84% of people whose world is turned upsid…

You’re talking about precision (true positive / true positive + false negative) but that’s only one part of the story. There is a real human cost to having a child born with a rare genetic disease (and I would argue is immensely more stressful). You can easily adjust the sensitivity to the test but at the cost of detecting actual true positive cases. The correct response to receiving a positive is to do another test…

No, precision is true positive / (true positive + false positive).

Your first equation is sensitivity.

Re: When they warn of rare disorders, these prenatal tests are usually wrong

#63

Post by fshbbdssbbgdd » Sun, Jan 02, 2022, 9:21 PM UTC

isoprophlex wrote: post #41

halpert wrote: post #30

Earlier quoted context omitted.

Their graphics say the tests are “84% wrong.” Do you really feel that’s an accurate description? That doesn’t feel like an accurate description to me, and their usage of “wrong” in this context highlights that they don’t understand the distinction and importance of true positives, false positives, true negatives, and false negatives when measuring accuracy.

Going through something like this is very VERY stressful. When you get a negative you immediately forget about it. When you get a positive you die inside. Speaking from experience here. 84% wrong sounds, to me, as an accurate description. Experiencing this from the inside out, only the false/true positive ratio matters. (Given sufficiently low false negative rates, of course) 84% of people whose world is turned upsid…

If you get a negative result and then your child is born with the condition, you won’t forget quickly either.

Re: When they warn of rare disorders, these prenatal tests are usually wrong

#64

Post by tinbad » Sun, Jan 02, 2022, 9:28 PM UTC

siganakis wrote: post #26

My wife and I went through this a couple of years ago, with a 10 week NIPT calling a rare trisomy (chr 9), which is always fatal within a few weeks of birth. It was absolute hell. The key problem here is the waiting and uncertainty. You have the NIPT at 10w, but you can’t have the amniocentesis until several weeks later. When that came back fine, there were questions about whether it was a “mosaic” meaning only a sma…

Having gone through two twin pregnancies (where the odds of these tests being correct are especially low) we declined all of them. Anecdotally, I know of several parents who had a positive test for genetic disorder, went ahead with the pregnancy anyway and children were perfectly healthy. Until these tests are close to 100% reliable I don’t see the point.

Re: When they warn of rare disorders, these prenatal tests are usually wrong

#65

Post by mcguire » Sun, Jan 02, 2022, 9:51 PM UTC

gwern wrote: post #9

This article is a confused mess. It's something of a Gish gallop in conflating all the different issues they could come up with, while leaving out all the necessary vocabulary (C-f "Bayes" "posterior" "decision theory" [Phrase not found]) making it almost impossible to consider each issue in adequate detail. It mixes up poor communication (reporting false-positive/negative rates as if posterior probabilities, & exagg…

So you are saying the testing companies in the article aren't fraudulently claiming much more effective tests than they are providing?

Re: When they warn of rare disorders, these prenatal tests are usually wrong

#66

Post by mcguire » Sun, Jan 02, 2022, 9:54 PM UTC

halpert wrote: post #20

How did this article, written by someone who clearly lacks an understanding of basic statistics, make it into the Upshot? They try to make it seem like the test is wrong 85% of the time, but that's not necessarily the case. All we know from the article is that 85 / 100 positive results are false positives, which means the test could actually be quite accurate. If the test correctly identifies 100% of real cases, then…

Would a test that reported 100% positive similarly be "quite accurate"? It would catch all true positives, right?

Re: When they warn of rare disorders, these prenatal tests are usually wrong

#67

Post by mcguire » Sun, Jan 02, 2022, 9:56 PM UTC

halpert wrote: post #51

adjkant wrote: post #48

Earlier quoted context omitted.

While the author may not be well versed or focusing on the stats side, you're missing the human side here I think. > the tests are inaccurate, when in reality the tests are accurate If the test make someone consider terminating a pregnancy or even considering it, that's a lot of pain. So for that human, the test is failing its purpose potentially, depending on the value calculation of terminating a viable pregnancy v…

Or maybe you’re missing the human side of having a child born with a serious genetic defect?

Is it better to terminate 85 pregnancies which do not have a serious defect in order to catch 15 which do? At what point is it not better to terminate 100% of pregnancies?

Re: When they warn of rare disorders, these prenatal tests are usually wrong

#68

Post by jasonhansel » Sun, Jan 02, 2022, 10:00 PM UTC

IMHO, some of those criticizing the article for failing to understand statistics are missing the point.

The point is that people who get a "positive" result on these tests are often put through terrifying levels of anxiety when there is no actual problem; this anxiety is often exacerbated because they aren't informed of the false positive rate. This clearly has a harmful emotional effect on people, and explaining the false positives in Bayesian terms, or reframing it in terms of sensitivity and specificity, doesn't undo that damage.

That potential harm needs to be explained to patients, and it needs to be weighed carefully against the potential benefits of the test (as is done for PSA tests for prostate cancer, which also have a high false positive rate). Given that potential for harm, it's not unreasonable to ask that these tests be more tightly regulated.

To quote the OP:

> In interviews, 14 patients who got false positives said the experience was agonizing. They recalled frantically researching conditions they’d never heard of, followed by sleepless nights and days hiding their bulging bellies from friends. Eight said they never received any information about the possibility of a false positive, and five recalled that their doctor treated the test results as definitive.

(Edit: clarified)

Re: When they warn of rare disorders, these prenatal tests are usually wrong

#69

Post by subpixel » Sun, Jan 02, 2022, 10:04 PM UTC

siganakis wrote: post #26

My wife and I went through this a couple of years ago, with a 10 week NIPT calling a rare trisomy (chr 9), which is always fatal within a few weeks of birth. It was absolute hell. The key problem here is the waiting and uncertainty. You have the NIPT at 10w, but you can’t have the amniocentesis until several weeks later. When that came back fine, there were questions about whether it was a “mosaic” meaning only a sma…

Our experience was kicked off by a troublesome ultrasound and then confirmed by amniocentesis.

The tragedy of receiving news like this is probably fathomable, but I think it may be hard to grasp the emotional and intellectual agony of deciding whether to terminate a pregnancy based on a set of probabilities.

It breaks my heart to think that parents face this decision with erroneous data.

Re: When they warn of rare disorders, these prenatal tests are usually wrong

#70

Post by midjji » Sun, Jan 02, 2022, 10:12 PM UTC

jasonhansel wrote: post #68

IMHO, some of those criticizing the article for failing to understand statistics are missing the point. The point is that people who get a "positive" result on these tests are often put through terrifying levels of anxiety when there is no actual problem; this anxiety is often exacerbated because they aren't informed of the false positive rate. This clearly has a harmful emotional effect on people, and explaining the…

If you get a positive for a horrid cancer with a 90 percent false positives you should be afraid. Its lunacy for tests to be regulated beyond requiring rough false positive false negative rates, and if anything smacks of "I dont understand statistics and therefore have to protect my children from understanding statistics." The article is most likely written by some anti abortion idiot.