Causal Inference Part 1: Using Causal Inference to Understand Product Experiments

5 min readJan 9, 2023

Causal inference is the process of determining the extent to which changes in one variable cause changes in another. This is particularly useful in product experimentation, as it allows companies to identify the specific factors that drive changes in metrics such as user engagement or sales.

Introduction

In product development, it is often important to understand the causal relationships between different factors and outcomes. Causal inference is the process of determining the extent to which changes in one variable cause changes in another. This can be particularly useful in product experimentation, as it allows companies to identify the specific factors that drive changes in metrics such as user engagement or sales.

The role of randomized controlled experiments in causal inference

One way to establish causality in product experimentation is through the use of randomized controlled experiments (RCTs). In an RCT, participants are randomly assigned to either a treatment group or a control group. The treatment group receives the experimental intervention (e.g., a new feature), while the control group does not. By comparing the outcomes of the two groups, it is possible to determine the causal effect of the intervention.

However, RCTs have some limitations. It may not always be possible or ethical to randomize the assignment of treatments, especially in cases where the treatment has strong effects on individuals or society. Additionally, RCTs may not always be representative of real-world conditions, as they often take place in controlled environments.

Alternative methods of causal inference

Quasi-experimental designs

Quasi-experimental designs involve the manipulation of an independent variable, but without the randomization of treatment assignment. This means that the treatment group and the control group are not randomly selected but are instead determined by the researcher or the natural circumstances of the study.

Quasi-experiments can provide useful information about causality, but they are generally considered to be less reliable than RCTs because of the potential for selection bias.

Observational studies

Observational studies involve the observation of naturally occurring relationships between variables, but do not involve any manipulation. These types of studies are often used in fields such as sociology and psychology and can provide valuable insights into real-world phenomena. However, because there is no manipulation of variables, it is difficult to establish causality with observational studies alone.

Instrumental variables

Instrumental variables involve the use of a third variable that is correlated with the treatment and the outcome and can be used to identify the causal effect of the treatment.

For example, if a company wants to understand the effect of a new marketing campaign on sales, they could use the amount of money spent on the campaign as the instrumental variable. By comparing the sales data of different regions where different amounts of money were spent on the campaign, the company could identify the causal effect of the campaign on sales.

Instrumental variables can be particularly useful in cases where it is not possible to randomly assign treatments. However, they are not always easy to identify and can be subject to certain assumptions and limitations.

Challenges and considerations in applying causal inference to product experiments

Selection bias occurs when the treatment and control groups are not comparable, leading to inaccurate results. This can happen when the groups are not randomly selected, or when there are differences between the groups that are not accounted for. For example, if a company is testing a new feature on a group of users who are more likely to be engaged with the product, the results of the experiment may not be representative of the entire user base.
Confounding variables, also known as third variables, are variables that are correlated with both the treatment and the outcome. They can affect the accuracy of the results by confounding the relationship between the treatment and the outcome. For example, if a company is testing the effect of a new feature on user engagement, and the users who receive the feature are also more likely to use the product at a time when engagement is naturally higher, the effect of the feature on engagement may be confounded by the time of day.
External validity refers to the extent to which the findings of an experiment can be generalized to other populations or contexts. It is important to consider external validity when applying causal inference to product experiments, as the results may not necessarily be applicable to other groups or situations. For example, if a company conducts an experiment on a group of users who are highly engaged with the product, the results may not be generalizable to the entire user base.

Examples of using causal inference in product experimentation

Case study 1: Using instrumental variables to understand the effect of a new feature on user engagement

A company is considering the addition of a new feature to its product, and wants to understand the impact of the feature on user engagement. They decide to use instrumental variables to identify the causal effect of the feature.

To do this, they gather data on the adoption of the feature by users and the level of user engagement. They also gather data on a third variable that is correlated with the adoption of the feature, but not with engagement (for example, the amount of money spent on marketing the feature).

By comparing the engagement data of users who adopted the feature with different amounts of marketing spend, the company is able to identify the causal impact of the feature on engagement.

Case study 2: Quasi-experimental design to assess the impact of a price change on sales

A company is considering a change to the price of its product, and wants to understand the impact of the price change on sales. They decide to use a quasi-experimental design to assess the causal effect of the price change.

To do this, they randomly vary the price of the product in different regions and compare the sales data. By comparing the sales data of regions with different price points, the company is able to identify the causal effect of the price change on sales.

Conclusion

In summary, causal inference is a powerful tool for understanding the relationships between different factors and outcomes in product experimentation. Randomized controlled experiments, quasi-experimental designs, observational studies, and instrumental variables are all methods that can be used to identify causal relationships. However, it is important to consider challenges and considerations such as selection bias, confounding variables, and external validity when applying causal inference to product experiments.

There are many potential future directions for the use of causal inference in product experimentation. One possibility is the development of more advanced methods for identifying and controlling for confounding variables. Additionally, there may be opportunities to use machine learning and other advanced analytical techniques to better understand the relationships between variables in complex product experiments. Finally, further research on the external validity of causal inference results in product experimentation could help to better understand the generalizability of findings across different contexts and populations.

Connect with the Author

If you enjoyed this article and would like to stay connected, feel free to follow me on Medium and connect with me on LinkedIn. I’d love to continue the conversation and hear your thoughts on this topic.