Qualification: How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It
Post-treatment regrading can compromise a treatment-effect estimate. Montgomery, Nyhan, and Torres analyze post-treatment conditioning in experiments. They classify three recurring practices: controlling for variables measured after treatment, dropping observations under post-treatment criteria, and restricting samples by post-treatment variables. Their analytical demonstrations and reanalyses show that these choices can distort causal-effect estimates. The direction and size of the bias may not be bounded without strong assumptions. Random assignment does not preserve its identifying advantage once an analysis conditions on information created or selected after treatment. This evidence qualifies Kaal's claim. Regrading a diagnostic after execution can alter the information used to evaluate an observed effect. If the regrading changes classifications, exclusions, or conditioning criteria after treatment, the resulting estimate may not identify the treatment effect claimed. Montgomery, Nyhan, and Torres do not examine Kaal's diagnostic runs. They also do not establish that every post-execution correction creates bias. A correction specified independently of treatment may instead improve measurement. The comparison supports a methodological limit. A treatment-effect conclusion requires the post-execution regrading rule, its timing, and its relation to treatment assignment to be independently reconciled.
research-methodsconsensus-and-securityscholarly-growth-coveragescholarly-literaturecausal-inferenceexperimental-designpost-treatment-biasoutcome-measurementevidence-provenance