Business and government leaders need to be able to analyse data and make causal claims. This skill is becoming increasingly important. This can be achieved by emphasizing the importance of experiments within organizations. Research in psychology and behavioral economics has shown that there are many common mistakes when it comes to looking at data. We are prone to look for evidence that supports our preconceived notions, and we ignore data that could contradict them. We overlook important details about how the data was created. It is easy to concentrate on the data you have in front of us, even if the most important data is not there. This can lead to mistakes or avoidable disasters for individuals, companies, and governments that make the decision.
All of us have been taught that correlation does NOT imply causation. Many business leaders, elected officials, media outlets, and others still use misleading correlations to make causal claims. These claims are often not properly investigated, amplified and misunderstood to guide decisions.
There are many examples. Take for example a recent study on cardiovascular disease that examined whether bathing can lower the risk of developing it. People who regularly took baths were less likely to suffer strokes or have cardiovascular disease. According to the authors, baths have a positive effect on cardiovascular disease and stroke prevention. It is difficult to determine if the relationship between baths and health can be causal without a controlled experiment or a natural experiment in which subjects were randomly selected. It is possible, for example, that regular bath-takers have less stress and more time to relax. This could explain why they have lower rates. These findings were still widely reported, with headlines such as "Taking a bath is more than relaxing." You could be getting good health benefits from it.
Research in psychology and behavioral economics has shown that there are many common mistakes when examining data. We are prone to look for evidence that supports our preconceptions and disregard data that could challenge them. We overlook important details about how the data was created. It is easy to concentrate on the data you have in front of your eyes, even if the most important data is missing. It can seem as though what you see is all that is there, according to Nobel Laureate Daniel Kahneman.
This can lead to mistakes or avoidable disasters for individuals, companies, and governments that make the decisions. We are constantly bombarded daily with data and facts. Business and government leaders need to be able to analyse data and make causal claims. This can be achieved by emphasizing the importance of experiments within organizations.
Unsupported Causal Claims Can Lead Organizations astray
The correlation between crime and police spending was examined in a 2020 Washington Post article. The article concluded that there is no national correlation between police spending and crime rates. The current level of crime is an important factor in police spending. This creates a chicken-and-egg scenario. Research has shown that increasing the number of police officers leads to a decrease in crime.
eBay spent approximately $50 million per year advertising on search engines in 2013. Consultants had found that sales were higher in areas where there were more ads. The company was urged by economists Tom Blake and Chris Nosko to reexamine the causal claim. They conducted new randomized controlled trials and analyzed natural experiments to find that the ads were almost a waste of money, contrary to what the marketing team believed. These ads were targeted at people who were likely to shop on eBay.
Targeted customers' pre-existing purchase intent was responsible for both the advertisement being shown and the purchase decision. eBay's marketing team mistakenly undervalued this factor and assumed that the observed correlation was due to advertisements causing purchase. They would have likely avoided this mistake if they had investigated other factors that could have caused the correlation.
Yelp overcame similar challenges in 2015. According to a consulting report, Yelp's advertising platform earned more business than companies that did not advertise on it. Here's the problem: Yelp users who get more business may be more inclined to advertise. We discussed the challenge with my former COO, and decided to conduct a large-scale experiment in which we distributed packages of ads to thousands of randomly selected businesses. This experiment was successful because we were able to determine the factors that drove the correlation. Yelp gained new insights into the effects of ads by discovering that Yelp ads had a positive impact on sales.
The Nobel Committee champions causal inference research
Over the past 40 years, the landscape of empirical economics changed dramatically. Economics has developed a range of skills that are focused on assessing causal relations. This work has been the subject of two out of three Nobel Prizes. Abhijit Bangerjee, Esther Duflo and Michael Kremer shared two years ago the Nobel Prize. They were recognized for their innovative approach to alleviating poverty worldwide. In this year's Nobel Prize, economists Josh Angrist and Guido Imbens won the Nobel Prize. Angrist called it the credibility revolution in economics. Angrist, Imbens and Card were praised by the committee for their methodological contributions in the analysis of causal relations and Card for his empirical contributions towards labour economics.
It was remarkable to see the development of the causal inference toolkit, and the Nobel winners' work is truly inspirational. To think more deeply about causal claims, you don't need to be an economist. It is worth taking the time to learn about the processes that are generating the data. Instead of assuming that correlation is causation (or a lack thereof), consider what other factors could be driving the correlation, and whether or not these may be biased towards the relationship you see. You might feel reassured that the relationship you are seeing is causal in some cases. You might not trust the finding in some cases.
Experiments can be useful if you're concerned that there might be no causality to the correlation. Experiments are a key part of decision-making processes at companies like Amazon and Booking.com. However, experiments are not always possible. These cases are when you need to think about other evidence and find out if it might help answer the question. You might be able to find a natural experiment that works in your case.
