Joseph Blitzstein: "The Soul of Statistics" | Harvard Thinks Big 4

TL;DR

Conditioning is the soul of statistics: every probability is conditional on the information you actually observe. In WWII, Abraham Wald advised armoring bomber sections with no bullet holes, because the damaged planes still returned while the ones hit in undamaged spots never came back. Ignoring how data is sampled produces selection bias.

Transcript

SPEAKER: Professor Blitzstein is a Professor of The Practice of Statistics. He is known for teaching the popular class, Stat 110, introduction to probability, which holds over 300 students each fall. He also has over 200,000 subscribers to the class on iTunes U. His research interests focus on statistical inference for complex networks. Professor B... Read More

Key Insights

• Conditioning is the soul of statistics, meaning every probability is conditional on the information you have, and conditional probability tells you how to update beliefs based on what you observe.
• Selection bias occurs when the data you get to see is not what you actually care about, as with bomber planes: you only observe the ones that survived and returned, not the ones shot down.
• Abraham Wald advised putting armor on the parts of returning WWII bombers that showed little or no damage, reasoning that planes hit in those spots did not make it back.
• Whenever you have a data set, you should think about how it was sampled, because ignoring the sampling process can lead to very misleading answers.
• Lombard's 1835 longevity study found chocolate makers lived to 73.6 years and professors to 66.6, but had only nine chocolate makers, raising a sample size issue.
• The 1835 study's finding that students lived an average of 20.2 years is an obvious conditioning error, since 20 is simply a normal age to be a student.
• Censoring is a statistical problem because we only know how long people lived after they died, so survival analysis exists to handle the unknown future lifespans of living people.
• Regression toward the mean means extreme observations tend to move toward the average on remeasurement, as Galton found that tall fathers had tall but less extreme sons.

Questions & Answers

Q: What is the soul of statistics according to Joseph Blitzstein?

Blitzstein argues that conditioning is the soul of statistics. This means conditional probability, the idea that we are always given information and every probability is conditional on the information we have. Conditional probability tells us how to update our beliefs based on the information we are able to observe. He goes further, suggesting this idea is not just the soul of statistics but everything, because our very presence in any situation is conditional on a huge conditional probability calculation of the events that led there.

Q: How did Abraham Wald decide where to put armor on WWII bombers?

During World War II, too many British bombers were being shot down, and armor was heavy and expensive so it could not be placed everywhere. The obvious approach was to armor where returning planes showed heavy damage, but Wald advised the exact opposite: reinforce the parts with little or no damage. His reasoning was that the planes you observe are only the ones that survived. Planes hit in the undamaged-looking areas never came back, so those spots were the true vulnerabilities needing armor.

Q: What is selection bias in statistics?

Selection bias occurs when what you get to observe is not the same as what you actually care about. In the bomber example, you know how bullet holes are distributed on planes that returned, but you want to know about the planes that did not make it back. Linking those two things is a statistical problem. Whenever you have a data set, you should think about how it was sampled, because ignoring the sampling process can lead to very misleading answers, especially in areas like network analysis.

Q: Why was the 1835 longevity study's finding about students misleading?

Lombard's 1835 study found that the average longevity of students was 20.2 years, which sounds alarming. However, the problem is what you are conditioning on. A student being 20 years old is a completely normal age for a student, so the figure reflects the typical age of students rather than early death. Blitzstein uses this extreme, obvious case to show how biases can silently distort conclusions, since the biases are less visible in professions like chocolate makers or professors.

Q: What is regression toward the mean?

Regression toward the mean is the tendency for extreme measurements to move closer to the average when measured again. For example, students who score very high on the SAT tend to score lower on a retake, while those who scored very poorly tend to improve. Sir Francis Galton, Darwin's cousin, was among the first to clearly explain it by studying heights: very tall fathers had tall sons who were less tall, and very short fathers had taller sons, both regressing toward, but not all the way to, the mean.

Q: What is censoring in survival analysis?

Censoring is the statistical problem that arises because we only know how long people lived after they died. For deceased people we know their full lifespan, but for living people we do not yet know how much longer they will live. This creates a bias if not handled carefully. A large field of statistical survival analysis exists to deal with this uncertainty about how long currently living people will continue to live, ensuring conclusions about longevity are not distorted.

Q: How does regression toward the mean relate to reward and punishment?

Daniel Kahneman described a Eureka moment while teaching flight instructors that praise works better than punishment. An instructor objected that praising good maneuvers led to worse next attempts, while screaming at bad ones led to improvement. Kahneman realized this reflects regression toward the mean: we reward good performance and punish bad performance, and because extremes naturally move toward the average, it appears we are punished for rewarding others and rewarded for punishing them, purely as a statistical artifact.

Q: Why should you think about how a data set was sampled?

Because the data you observe is filtered by the sampling process, and ignoring it can produce misleading answers. Blitzstein notes that much work on networks ignores where the network came from and how it was sampled, which can lead to very misleading results if you are not careful about what you are conditioning on. Every probability is conditional on the information available, so understanding how the data was collected is essential to drawing valid statistical conclusions from it.

Summary

Professor Blitznein, a professor of the practice of statistics, gives a talk titled "The Soul of Statistics." He begins by sharing a motivating example about British bombers during World War II and how statistician Abraham Wald advised them to put armor where the planes showed little or no damage. This example illustrates selection bias in statistics. Professor Blitznein emphasizes that conditioning is the soul of statistics, which means that all probabilities are conditional and our beliefs should be updated based on the information we have. He discusses the importance of understanding sampling and the danger of ignoring it, using the example of networks. Additionally, he explores selection bias and censoring in a longevity study and regression towards the mean in test scores. He concludes by discussing the conditional golden rule, which guides his teaching philosophy.

Questions & Answers

Q: What was the motivating example Professor Blitznein shared regarding British bombers during World War II?

Professor Blitznein shared the example of British bombers being shot down by the Nazis and how statistician Abraham Wald advised them on where to put armor. Instead of putting armor where the planes sustained heavy damage, Wald suggested putting armor where there was little or no damage on the planes that returned. This example illustrates the concept of selection bias in statistics.

Q: What does Professor Blitznein mean when he says "conditioning is the soul of statistics"?

By saying "conditioning is the soul of statistics," Professor Blitznein emphasizes that all probabilities are conditional on the information that we have. Conditional probability allows us to update our beliefs based on the observed information. It is a fundamental aspect of statistical thinking and analysis.

Q: How does Professor Blitznein highlight the importance of sampling in statistical studies?

Professor Blitznein discusses the danger of ignoring sampling in statistical studies, using the example of networks. He mentions that many studies on networks focus on analyzing the structure without considering how the network was sampled. This omission can lead to misleading answers and interpretations if researchers are not cautious about what they are conditioning on.

Q: What example does Professor Blitznein provide regarding selection bias and censoring?

Professor Blitznein shares a longevity study conducted in 1835. He presents average longevities for different professions, such as chocolate makers, professors, clocksmiths, locksmiths, and students. He explains the selection bias in the data set, such as the small sample size of chocolate makers and the issue of conditioning on the age of students. He also mentions censoring, which refers to the fact that we only know the lifespan of individuals after they have died.

Q: How does Professor Blitznein explain regression towards the mean?

Professor Blitznein explains regression towards the mean using the example of test scores. He describes how students who initially perform exceptionally well or poorly tend to move closer to the mean when they retake the test. He also mentions Sir Francis Galton's study on heights, which showed that tall fathers have sons who are not as tall on average, while short fathers have sons who tend to be taller. Regression towards the mean is a concept that applies to various domains and helps to maintain stability in populations.

Q: What is the conditional golden rule mentioned by Professor Blitznein?

The conditional golden rule is Professor Blitznein's teaching philosophy. It is based on the principle of "do unto others as you would have done unto you," but with the condition that it applies only to individuals who share the same interests, background, and knowledge. Professor Blitznein aims to follow this rule in his teaching and believes that conditional probability, although challenging, is worth contemplating.

Takeaways

Professor Blitznein emphasizes the importance of conditioning in statistics, highlighting its role in updating our beliefs based on observed information. He discusses the dangers of selection bias, ignoring sampling, and the misconception of regression towards the mean. Additionally, he shares his teaching philosophy, which is based on the conditional golden rule. Understanding these concepts is crucial for accurate statistical analysis and interpretation.

Summary & Key Takeaways

• Joseph Blitzstein argues that conditioning is the soul of statistics: every probability is conditional on the information we have. Conditional probability tells us how to update beliefs based on observed data, and this idea underlies all statistical thinking about what we can and cannot see.

• Abraham Wald's WWII bomber example illustrates selection bias. Rather than armoring where returning planes showed bullet holes, Wald advised armoring the undamaged areas, because planes hit there never returned. The data we observe is filtered by survival, so we must reason about the planes we cannot see.

• Lombard's 1835 longevity study and regression toward the mean show conditioning errors and sampling issues. Students appearing to live only 20.2 years reflects their age, not mortality. Galton found tall fathers have sons who regress toward the mean height, a phenomenon Kahneman connected to reward and punishment.