Skip to main content
All CollectionsFAQModeling FAQA/B Testing Analysis FAQ
How do I interpret the variance waterfall with A/B testing?
How do I interpret the variance waterfall with A/B testing?
Updated over 5 months ago

A/B testing analysis will show you a variance waterfall from the control group to the treated group with the following components:

  • Observed Control Group: this shows you the observed average outcome metric for the control group. This is the observed outcome if you do nothing, i.e., in the absence of treatment. It may not be comparable to your treated group depending on what factors (variables) changed between the control and treated groups.

  • Controlled Factors: this is the portion of the variance attributed to factors other than the treatment. These factors are captured through the independent variables you selected during model configuration, and only these independent variables are taken into account in the analysis.

  • Matched Control Group: this shows the average outcome for a control group that has been matched, i.e. adjusted to be apples-to-apples when compared to the treated group. This group is your control adjusting for all factors captured through your independent variables. This is the expected outcome if you do nothing, i.e., in the absence of treatment.

  • Treatment Effect: the treatment effect (in this case defined as the average treatment effect on the treated) is the estimate of the true impact of the treatment applied excluding the impact of the controlled factors.

  • Observed Treated Group: this shows you the observed average outcome metric for the treated group. The difference between this outcome and the outcome for the control group can be attributed to the treatment itself and any other changes, controlled or not, that may have occurred between the two. In many cases these two groups are not comparable due to selection bias and other factors.

Did this answer your question?