Sloppy Use of Machine Learning is Causing a ‘Reproducibility Crisis’ in Science

Civil wars are one of the messiest and scariest of human affairs. A strand of political science research claiming to predict when a civil war will break out with more than 90 percent accuracy was discovered by a professor and his student.

Machine learning is a technique beloved by tech giants that underpins modern artificial intelligence. Applying it to data such as a country's gross domestic product and unemployment rate was said to beat more conventional statistical methods at predicting the outbreak of civil war.

Many of the results turned out to be mirages when the researchers looked more closely. Data from the past is fed into a machine learning program that learns from unseen data. In several papers, researchers failed to separate the pools of data used to train and test their code's performance, a mistake that results in a system being tested with data it has seen before.

In each of the cases, there was an error in the machine- learning process. When he and Narayanan fixed those errors, they found that modern artificial intelligence did not offer any advantages.

The experience prompted the pair to investigate whether machine learning was being misapplied in other fields, and to conclude that incorrect use of the technique is a problem in modern science.

Artificial intelligence can uncover patterns that are hard to discern using more conventional data analysis. Artificial intelligence has been used by researchers to predict and control fusion reactor structures.

The impact of artificial intelligence on scientific research has been less than ideal. When the pair surveyed areas of science where machine learning was applied, they found that there were many errors in studies that relied on machine learning.

Many researchers are rushing to use machine learning without a full understanding of its limitations. The tech industry has rushed to offer artificial intelligence tools to lure newcomers, often with the goal of promoting cloud platforms and services The idea that you can take a four-hour long online course and use machine learning in your research has become so overblown. People don't stop to think about where things could go wrong.

Scientists are betting on the use of artificial intelligence in research. A professor at MIT who is researching novel solar cells uses artificial intelligence extensively. Machine learning is a powerful tool that should not be abandoned. If scientists from different fields share best practices, errors can be fixed. He says that you don't need to be a machine-learning expert.