# An example demonstrating the problem with “net treatment effects”

In this previous post, we described the concept of principal stratification in causal inference. Principal stratification was proposed as a framework for how to think about causal effects in the presence of post-treatment variables. It was a response to a standard method at that time known as “net treatment effects adjusting for post-treatment variables”. In this post, we present two examples that demonstrate why net treatment effects were the wrong way to think about causal effects in this setting. These examples are taken from Frangakis & Rubin (2002) (Reference 1).

Background

• We index individuals by $i$.
• Each individual is either in control ($Z_i = 1$) or treatment ($Z_i = 2$).
• We have some outcome/response metric of interest. If the individual is in control (treatment resp.), the response metric is $Y_i(1)$ ($Y_i(2)$ resp.). We only get to observe one of them, which we denote by $Y_i^{obs} = Y_i(Z_i)$.
• We have a post-treatment variable that takes on value $S_i(1)$ if the individual is in control and is $S_i(2)$ if the individual is in treatment. We only get to observe one of them, which we done by $S_i^{obs} = S_i(Z_i)$.

Net treatment effects propose comparing the distributions

$\text{pr} \left\{ Y_i^{obs} \mid S_i^{obs} = s, Z_i = 1 \right\}$ and $\text{pr} \left\{ Y_i^{obs} \mid S_i^{obs} = s, Z_i = 2 \right\}$.

The resulting difference is called the “net treatment effect of assignment $Z$ adjusting for the post-treatment variable $S^{obs}$.

Example 1

Consider a scenario where we have 3 subgroups of patients (sicker, normal, healthier) based on their possible post-treatment values $(S_i(1), S_i(2))$. We assume here that the post-treatment variable can only take on two values: L for low and H for high. Assume also that there are the same number of individuals in each subgroup, and that we care about the average treatment effect.

From the 3rd and 4th columns in the table below, we see that the treatment has no effect on the response $Y$ for sicker and healthier patients, but for normal patients, the treatment increases the response from 30 to 50 for a treatment effect of +20.

Consider the net treatment effect of assignment $Z$ adjusting for $S^{obs}$ for $S^{obs} = L$. This compares the means

$\mathbb{E} \left[ Y_i^{obs} \mid S_i^{obs} = L, Z_i = 1 \right]$ and $\mathbb{E} \left[ Y_i^{obs} \mid S_i^{obs} = L, Z_i = 2 \right]$,

which is equivalent to comparing the blue box to the orange box. This comparison suggests that the treatment effect is $10 - 20 = -10$: a negative effect! This is clearly an incorrect conclusion, and it arises because the blue box includes normal patients while the orange box does not.

Example 2

The set-up for the second example is the same as the first, except that the potential outcome values in the 3rd and 4th columns have changed. In this example, every subgroup has a treatment effect of +10.

The net treatment effect compares either the top blue box with the top orange box, or the bottom blue box with the bottom orange box. Either way, the net treatment effect is zero! This is clearly an incorrect conclusion. Again, this incorrect conclusion arises because the groups being compared are not the same. For $S^{obs} = L$, we are comparing sicker patients with sicker and normal ones, while for $S^{obs} = H$, we are comparing normal and healthier patients to just healthier ones.

Principal stratification corrects for this by saying that we should only make comparisons between groups which have the same membership.

References:

1. Frangakis, C. E., and Rubin, D. B. (2002). Principal stratification in causal inference.