Potential false-positive rate among the 'asymptomatic infected individuals' • Hacker News Board

Re: Potential false-positive rate among the 'asymptomatic infected individuals'

#71

Post by eveningcoffee » Fri, Mar 13, 2020, 10:35 PM UTC

Scarblac wrote: post #15

bluGill wrote: post #7

Earlier quoted context omitted.

I'm starting to hear stories that the Chinese people demanded more quarantine than was needed. The problem got bad enough that the government decided to comply as it wouldn't hurt much more than the amount of quarantine was needed and might even help.

I really wonder how we're going to look back at this in a few years. On the one hand, no matter what measures politics takes here (the Netherlands), every political party is clamouring for more. That suggests to me that we'll probably end up doing too much. On the other hand, the scenes from Hubei and Italy are horrific, we're going to get them here as well, it could be far worse, we absolutely must act now. On the t…

Can you look up demographic of the infected people in Netherlands?

It can be that you have somehow managed to avoid infecting older generation. I have longer discussion about this here http://www.savespain.eu/italy-vs-germany

Re: Potential false-positive rate among the 'asymptomatic infected individuals'

#72

Post by naveen99 » Sat, Mar 14, 2020, 11:59 AM UTC

SkyPuncher wrote: post #59

cultus wrote: post #18

One fallacy that seems universal with healthcare folks is they think the false positive rate is the chance that a given positive result is erroneous. If an illness is rare, a positive result in a test with a 1% error rate might have an overwhelming probability of being a false positive. This is why prior probabilities need to be taken into account in making decisions.

With all due respect, this comment shows a lack of understanding about how health professionals assess the quality of diagnostics. My wife is a doctor (and I've learned a lot from her). In med school, they're specifically taught to evaluate diagnostics on their specificity and sensitivity - which essentially covers false positives and false negatives. If you hear a doctor talk about the "accuracy" of a test, it's lik…

I get bored by more esoteric statistics terms in epidemiology, but accuracy has a simple enough mathematical formula: https://www.lexjansen.com/nesug/nesug10/hl/hl07.pdf

(True positives + True Negatives) / number of all tested

Similar concept comes up in measuring accuracy of computerized image segmentation, where you ignore the true negatives

true positive / (true positive + false positive + false negative)

where it is called intersection over Union (IOU).

I can’t ever remember the names, and just rebuild whatever metric I care about in terms of true vs false and positive vs negative.

Applying all this to the real world is tough because of the over fitting problem. Even if you got the test to be 100% accurate in your tested population, it doesn’t mean it won’t be wrong on the next person it tests. Generalization is hard. So doctors have to guess based on their understanding of the tested and untested population and the sensitivity and specificity of the test. You can go meta and give the doctor a sensitivity and specificity also.

Re: Potential false-positive rate among the 'asymptomatic infected individuals'

#73

Post by ineedasername » Sat, Mar 14, 2020, 1:47 PM UTC

AnthonyMouse wrote: post #58

ineedasername wrote: post #37

Earlier quoted context omitted.

The same would be true of things like the Flu then too. Which means the true mortality rate of Flu is much lower, so even if COVID-19 is lower than current estimates for the same reason, it is still an order of magnitude higher than something like the Flu.

It's possible to get more accurate numbers for the flu because it's so prevalent, which makes it possible to do random sampling of the general population. If you did random sampling of the general population for this coronavirus right now, ~100% of the positive results would be erroneous because probably less than 0.01% of the population actually has it.

That's an excellent point! Thank you!

Re: Potential false-positive rate among the 'asymptomatic infected individuals'

#74

Post by cultus » Sat, Mar 14, 2020, 3:51 PM UTC

SkyPuncher wrote: post #59

cultus wrote: post #18

One fallacy that seems universal with healthcare folks is they think the false positive rate is the chance that a given positive result is erroneous. If an illness is rare, a positive result in a test with a 1% error rate might have an overwhelming probability of being a false positive. This is why prior probabilities need to be taken into account in making decisions.

With all due respect, this comment shows a lack of understanding about how health professionals assess the quality of diagnostics. My wife is a doctor (and I've learned a lot from her). In med school, they're specifically taught to evaluate diagnostics on their specificity and sensitivity - which essentially covers false positives and false negatives. If you hear a doctor talk about the "accuracy" of a test, it's lik…

The specificity (true positive) and sensitivity (true negative) do not solve the problem I am describing.

If something is rare, it has a low base rate. That even means a test with excellent specificity and sensitivity could still be wrong most of the time.

Decisions on test accuracy simply cannot be made coherently when ignoring the base rate. To make an intuitive example, suppose that one in a thousand people have a disease. A test for the disease has 90% specificity and 100% sensitivity. It will always correctly give a positive result if the person has the disease, and has a 99% chance that a given positive test is valid. Pretty good, much better than most tests.

Now suppose that 1/1000 of people have the disease. A person with a positive result has a 1% chance of not having the disease. If everyone is tested, then 1/1000 people will get true positive results. But, (999/1000 * 0.01) ~ 1% of people will get false positives.

Thus, a given person with a positive result has nearly a 10x chance of it being erroneous compared to it being accurate! As I said, the frequentist techniques that you describe and are taught in medical schools do not help with this.

Yet this is endemic in medicine. This sort of thing is why in a recent meta-study of 54 landmark cancer trials, only six could be replicated. That is frankly terrifying.