You will simulate a district where you can see every farmer’s two potential outcomes. Then you will find out what regression does and does not recover.
This notebook builds on the previous exercise on selection bias due to confounding.
The setting is a farming district where some farmers have installed a new irrigation pump. You will simulate data in which you know both potential outcomes for every farmer. With that data you will compute the naïve treated-versus-control comparison, the true Average Treatment Effect (ATE), the Average Treatment Effect on the Treated (ATT), the Average Treatment Effect on the Controls (ATC), and the exact size of the selection bias.
Next you will find out what regression does and does not recover. First you will run a model with the confounder added, then a model with an interaction term. At the end, you will use the same farmers to distinguish confidence intervals from prediction intervals.
Work through the cells in order: later cells use objects made by earlier ones.
Rubin and Holland caution readers about asking certain kinds of causal questions, such as questions about “the effect of being male vs. female.” Yet nearly everyone, Rubin and Holland included, agrees that sex-based discrimination is an important problem that should be studied with the best available empirical methods.
What do you think? How can statistical methods and rigorous causal models be used to study discrimination on the basis of sex, race, or similar attributes? What do you take away from Rubin and Holland?
Write a first answer now. You will come back to this question at the end of the notebook.
Left out: variation across seeds and failures of overlap. See the closing note.
One hundred farmers live in a district: 50 are rich and 50 are poor. A new irrigation pump is available. Rich farmers own more land and hold reliable water rights, so the pump raises their harvest by $6,000. Poor farmers farm small, rain-fed plots with no water rights, so the pump does nothing for their harvest.
Pumps are expensive. Most rich farmers bought one. Poor farmers got pumps through a subsidy lottery, which gave pumps to half of them.
The outcome is harvest value in thousands of dollars. Read every line of the setup cell below, which runs automatically. Can you explain what Y0, Y1 and Y_obs are, and why we can see both potential outcomes here?
Before running the next cell, write down a prediction.
Will the naïve difference in mean harvests be larger or smaller than the true average effect of the pump? Why?
Because this is a simulation, you have Y1 and Y0 for every farmer. Use Y1, Y0 and treat to calculate:
The naïve difference can be split exactly into two pieces:
naïve difference = ATT + [ mean of Y0 among treated − mean of Y0 among controls ]
The bracketed term is the selection bias. It compares the treated and control groups in the world where nobody has a pump. Compute it and check that the identity holds.
A natural fix is to control for wealth. Run the regression below and compare the coefficient on treat to the ATE, ATT and ATC you computed in Part 3.
Agree with your group on an answer to Q15 first. Then check it.
Regression gives each group a weight proportional to its size times the variance of treatment inside it. Few comparisons exist among the rich, nearly all of whom have a pump. The lottery made the poor group the most informative.
The name for this is a variance-weighted average of within-group effects. Weight on group g is ng · pg(1 − pg), divided by the sum over groups.
| Measure | Rich share |
|---|---|
| Everyone (ATE weights) | 0.50 |
| Treated (ATT weights) | 0.64 |
| Regression weights | 0.26 |
Each panel shows one wealth group. Dots are individual farmers and the dash is the group mean.
Two people ask you about poor farmers who have a pump.
Both questions get the same point prediction from fit_int, but they get very different intervals.
In this notebook, “rich” played a role much like sex or race in Q1. It is an attribute of the farmer that you did not manipulate, and you estimated effects within groups defined by it.
In this notebook, “what is the effect of the pump for poor farmers?” was a well-defined causal question. Is “what is the effect of being rich on harvest?” well-defined in the same way? Why or why not? How does this distinction help you think about studying discrimination? Revise or extend your Q1 answer.
Two extensions to try: give each student their own seed and compare the additive and interaction estimates; or change the rich split to 49/1 and rerun Parts 5–7 to see what weak overlap does to the standard errors.