Unbiasedness of the RCT estimator: one trial, many trials

What do you expect to happen?

A school tests a tutoring programme with a randomized trial. In this simulation we know every student's score with and without tutoring, so we know the true average effect. Commit to your guesses before you touch the controls.

In a single randomized trial, the estimated effect will… equal the true average effect miss the true effect, but the average of many trials will equal it miss the true effect, and so will the average of many trials

If each trial uses 4 times as many students, the spread of the estimates will… stay about the same shrink to about half shrink to about a quarter

Follow these steps

1Choose one answer for each of the two predictions.Does everyone in your group give the same answers? Talk until you agree, or note where you differ.
2Set Number of trials to 1 and press New seed five times.Does the single estimate (the triangle) ever sit exactly on the dashed true-effect line?
3Set Number of trials to 2000 with 20 students per trial.How far is the solid line (mean of the estimates) from the dashed line? Agree on one answer as a group.
4Keep 2000 trials. Move students per trial from 20 to 80, then to 320.Each time n is multiplied by 4, by what factor does the spread (SD of the estimates) change?
5Return n to 80. Raise Selection into treatment from 0 to 3.Do the solid and dashed lines still agree? What is different about how the treated group is chosen?
6Press Check my predictions (in the next section).Which prediction, if any, did you get wrong, and why?

Run the experiment again, and again

Controls

Students per trial (n): 20Half are treated, half are control.

Number of trials: 500Each trial is a fresh random sample and assignment.

Random seed: 1

Selection into treatment: 0.00 is a true lottery. Higher means higher-scoring students are more likely to be treated.

Display size

Estimates

True average effect: –
First trial's estimate: –
Mean of the estimates: –
SD of the estimates: –
SD predicted by theory: –

What just happened?

Each trial uses different students and a different random split, so each estimate carries its own random error. Across many trials those errors cancel. Averaged over repeated trials, the estimator gives the true effect. That is what it means for the estimator to be unbiased. It says nothing about how close any one trial is.

Name it

Unbiased estimator. The difference in mean outcomes between treated and control units, under random assignment, satisfies:

E[ τ̂ ] = ATE, where τ̂ = mean(Y | treated) − mean(Y | control)
Var( τ̂ ) = σ₁²/n₁ + σ₀²/n₀ → SD( τ̂ ) = √( 2(σ₁² + σ₀²) / n ) ∝ 1/√n

Quadrupling n halves the SD, so precision costs sample size at a diminishing rate. When treatment depends on baseline scores, as with the selection control, the errors no longer cancel, and the estimator is biased.

Now compute it yourself

The page above runs the code below. Fill the gaps marked NA, run each cell, and tick Done when you are satisfied. The sliders above re-run exercise 2.

Setup

1. How far is one trial from the truth?

Guess first: will the estimate be within 1 point of the true effect? Then compute the difference in mean outcomes between treated (d = 1) and control (d = 0).

2. Where do many trials average out?

The loop is written for you. Fill in the bias: the mean of the estimates minus the true effect. The sliders above drive this cell.

3. How does the spread shrink with n?

For each n, the cell simulates 300 trials and takes the SD of the estimates. Fill in the SD predicted by theory, using v1 and v0 from setup and the fact that each group has n/2 students.

Exit question

A school's trial of 40 students estimates a gain of +9 points, but the true effect is +5. Did the randomized trial fail? What would you tell the school about this one number, and what would change if the trial had 160 students?

Your answer (1–3 sentences)0 / 600 characters

Assumptions and what is left out

Parameter. 2000 students. Score without tutoring ~ Normal(50, 10). Individual effect ~ Normal(5, 4). True ATE about 5 points.
Distribution. Each trial draws n students with replacement and treats exactly n/2 by random shuffle. Var = v1/(n/2) + v0/(n/2), where v1 and v0 are the population variances of Y(1) and Y(0).
Simplification. No dropout, spillovers or noncompliance. Potential outcomes are all known, which is possible only in a simulation.
Simplification. The selection control ranks students on a noisy baseline score and treats the top half. At 0 it is a pure lottery.
Data. None: everything is simulated in the setup cell.
Interpretation. Unbiased means the average over repeated trials equals the ATE. It is not a claim about one trial.
Interpretation. The model answer uses the theoretical SD of this population: about 3.3 points at n = 40 and 1.6 at n = 160.
Interpretation. Every button writes only to shared state (New seed, Check my predictions, Compare with a model answer, Lock). Both predictions and the written answer are in state. Only the typed instructor code stays local.
Interpretation. The instructor lock is a soft classroom lock, not security. It freezes this page's controls and answers; the code editors are read-only only if the platform honours it.

Left out: confidence intervals, clustering, stratification and covariate adjustment.

One trial, many trials