What Is the Medical Test Paradox, and How Can We Redesign Bayes' Rule?

TL;DR

The medical test paradox occurs because a highly accurate test can still have low predictive value when the disease is rare. Among 1,000 women with 1% breast cancer prevalence, 9 true positives and 89 false positives mean a positive result indicates only about a 1 in 11 chance of cancer. The Bayes-factor approach clarifies how test evidence updates prior probability, so read on to understand the calculation and its limits.

Transcript

Some of you may have heard this paradoxical fact about medical tests. It's very commonly used to introduce the topic of Bayes' rule in probability. The paradox is that you could take a test which is highly accurate, in the sense that it gives correct results to a large majority of the people taking it. And yet, under the right circumstances, when a... Read More

Key Insights

• An accurate test is not necessarily highly predictive because its positive predictive value depends on disease prevalence as well as sensitivity and specificity. Even when more than 90% of all patients receive correct results, most positive results can be false positives in a low-prevalence population.
• Positive predictive value is the number of true positives divided by all positive results. In the example, 9 true positives and 89 false positives produce a value of 9 divided by 98, so a positive result indicates only about a 1 in 11 probability of cancer.
• Sensitivity is the proportion of people with the disease who receive a positive result. The example has 90% sensitivity because 9 of 10 women with cancer test positive, while the remaining woman receives a false negative, giving a false negative rate of 10%.
• Specificity is the proportion of people without the disease who receive a negative result. The example has about 91% specificity, while its false positive rate is 9%. These measures describe outcomes among healthy people, not the probability of disease after a positive result.
• Disease prevalence is the prior probability before the test result is known. When prevalence is only 1%, false positives from the much larger healthy group can greatly outnumber true positives, even when the test has sensitivity and specificity above 90%.
• A medical test updates a patient's probability of disease rather than directly determining it. In the example, the prior probability is 1 in 100, and the positive result raises it by almost an order of magnitude to approximately 1 in 11.
• The Bayes factor for a positive result is sensitivity divided by the false positive rate. With 90% sensitivity and a 9% false positive rate, the factor is 10, meaning a positive result is 10 times more likely with cancer than without it.
• The small-prior approximation multiplies the prior probability by the Bayes factor. A prior of 1 in 1,000 therefore becomes approximately 1 in 100 after a positive result. The shortcut cannot handle high priors reliably because it can incorrectly predict probabilities reaching 100% certainty.

Questions & Answers

Q: What is the medical test paradox?

The paradox is that a test giving correct results to more than 90% of patients can still make a positive result only weakly predictive. With 1% breast cancer prevalence, 9 true positives and 89 false positives mean that only 9 of 98 positive results represent cancer, or approximately 1 in 11.

Q: Why can an accurate medical test have low predictive value?

Predictive value depends on disease prevalence as well as test sensitivity and specificity. When only 1% of the population has the disease, a 9% false positive rate applied to the much larger healthy group produces far more false positives than true positives.

Q: What is positive predictive value in medical testing?

Positive predictive value is the number of true positives divided by the total number of positive results. In the example, it is 9 divided by 9 plus 89, so the positive predictive value is approximately 1 in 11.

Q: What is the difference between sensitivity and specificity?

Sensitivity measures how often the test returns a positive result among people with the disease, while specificity measures how often it returns a negative result among people without the disease. The example has 90% sensitivity and about 91% specificity, corresponding to a 10% false negative rate and a 9% false positive rate.

Q: How does disease prevalence change the meaning of a positive test?

Prevalence supplies the prior probability of disease before the test result is known. At 1% prevalence, only 10 of 1,000 women have cancer, so false positives from the 990 women without cancer can greatly outnumber true positives.

Q: What is the Bayes factor for a positive medical test?

The Bayes factor for a positive result is the test sensitivity divided by the false positive rate. With 90% sensitivity and a 9% false positive rate, the Bayes factor is 10, meaning a positive result is 10 times more likely with cancer than without it.

Q: How can you quickly estimate disease probability after a positive test?

When the prior probability is small, multiply it by the Bayes factor for a quick approximation. A prior of 1 in 1,000 and a Bayes factor of 10 produce an estimated updated probability of approximately 1 in 100.

Q: Why does the Bayes-factor shortcut fail for larger prior probabilities?

Multiplying the prior probability by the Bayes factor is only an approximation for small priors. With a 10% prior and a Bayes factor of 10, the shortcut would predict 100% certainty even though positive results would still include people without cancer.

Summary & Key Takeaways

• A screening example starts with 1,000 women, of whom 1% have breast cancer. The test produces 9 true positives, 1 false negative, 89 false positives, and 901 true negatives. Therefore, a woman with a positive result has a cancer probability of 9 divided by 98, approximately 1 in 11.

• The apparent paradox comes from confusing test performance with positive predictive value. Sensitivity measures correctness among people with the disease, while specificity measures correctness among those without it. Neither statistic alone answers whether a person with a positive result has the disease, because that conclusion also depends strongly on the disease prevalence.

• A positive result should be viewed as an update to existing chances. The Bayes factor for a positive result is sensitivity divided by the false positive rate. In the example, it equals 10. For a small prior, multiplying by this factor gives a useful mental approximation of the updated probability, though it fails for larger priors.