Estimating Effects After Matching: Key Challenges and Solutions for Quasi Experiments

Estimating Effects After Matching: Key Challenges and Solutions for Quasi Experiments

Introduction:
Quasi experiments are a valuable tool in research, allowing us to estimate causal effects when randomized controlled trials are not feasible or ethical. However, they come with their own set of challenges. In this article, we will explore two primary methods for estimating effects after matching and discuss the key challenges faced in quasi experiments, specifically focusing on the experiences at Netflix. We will also propose solutions and actionable advice to overcome these challenges.

1. Estimating Effects After Matching:
   Matching is a common technique used in quasi experiments to create comparable treatment and control groups. It aims to balance observed variables between the groups, reducing the risk of confounding factors. Two methods that have shown promising results in matched samples are using cluster-robust standard errors (SEs) and the bootstrap.

1.1 Cluster-Robust SEs:
Cluster-robust SEs are robust to within-cluster correlation, which is often present in matched samples. This method appropriately accounts for the dependence among observations within the same cluster, avoiding biased standard error estimates. By considering the clustering structure, cluster-robust SEs provide more accurate estimates of the true sampling variability of the effect estimator.

1.2 The Bootstrap:
The bootstrap is another method commonly used in matched samples to estimate the sampling variability of the effect estimator. It involves repeatedly resampling the data with replacement and estimating the effect each time. This allows for the construction of confidence intervals and hypothesis testing. Matching with replacement or inverse probability weighting can be employed in the bootstrap procedure.

However, it is important to note that regular robust SEs can sometimes over- or under-estimate the true sampling variability of the effect estimator. Careful consideration should be given to selecting the appropriate method based on the characteristics of the data and the research question at hand.

2. Key Challenges with Quasi Experiments at Netflix:
   While quasi experiments have proven to be useful at Netflix, there are specific challenges that arise when implementing them. Two key challenges are discussed below:

2.1 Limited Variable Balancing:
One challenge faced in quasi experiments is the ability to simultaneously balance on a limited number of observed variables. It is often difficult to find identical geographic units on all dimensions, making it challenging to achieve perfect balance. This can introduce potential bias and confounding factors into the analysis.

2.2 Noisy Results with Small Sample Size:
Another challenge in quasi experiments is obtaining precise results with small sample sizes. Due to resource constraints or other factors, the sample size may be limited, leading to large confidence intervals and imprecise effect estimates. This can make it difficult to draw meaningful conclusions from the analysis.

To address these challenges, Netflix has employed the use of dynamic linear models (DLMs) in their quasi experiments. DLMs allow for the control of pre-treatment variables at both the individual and city level. For example, if members in Toronto watch more Netflix originals than members in other cities in Canada, controlling for pre-treatment Netflix originals viewing can capture within and between unit variation separately. This approach helps to mitigate confounding factors and reduce noise in the results.

3. Actionable Advice:
   Based on the challenges discussed and the experiences at Netflix, here are three actionable pieces of advice for conducting quasi experiments:

3.1 Prioritize Variable Balancing:
While it may not always be possible to achieve perfect balance on all observed variables, prioritize the selection of variables that are most likely to be confounding factors. Carefully consider the trade-off between including more variables for balance and the risk of overfitting the model.

3.2 Explore Stratification:
Stratification is a technique that can be used in matching to create subgroups with similar characteristics. This can help to achieve a better balance between treatment and control groups, reducing confounding factors and improving the precision of effect estimates.

3.3 Increase Sample Size:
Whenever feasible, strive to increase the sample size to improve the precision of the results. This can be achieved through collaborations, data pooling, or extending the duration of the study. Larger sample sizes allow for smaller confidence intervals and more reliable effect estimates.

Conclusion:
Quasi experiments are a valuable tool for estimating causal effects when randomized controlled trials are not possible. However, they come with their own set of challenges. By employing suitable methods for estimating effects after matching, such as cluster-robust SEs and the bootstrap, and considering the unique challenges faced in quasi experiments, researchers can enhance the validity and reliability of their findings. By prioritizing variable balancing, exploring stratification, and increasing sample sizes, actionable steps can be taken to overcome the key challenges faced in quasi experiments. Through these efforts, we can continue to advance our understanding of causal relationships in complex settings.