If Eve is Right—How do You Follow the Science?
By Carolyn Mathas
Researchers used the ‘robot scientist’ Eve and automated text analysis to semi-automate
reproducing research results. Reproducibility is a major challenge in science. Led by the
University of Cambridge, researchers analyzed more than 12,000 research papers on breast
cancer cell biology. Less than one third – 22 papers – were found to be reproducible. Reported in
the journal Royal Society Interface, this demonstrates that it is possible to use robotics and
artificial intelligence to help address the reproducibility. More than 70% of researchers have tried
and failed to reproduce another scientist’s experiments, and more than half have failed to
reproduce some of their own experiments: this is the reproducibility crisis.
Several years ago, robot scientist Eve was developed, a computer/robotic system that uses
techniques from artificial intelligence (AI) to carry out scientific experiments.
As part of a project funded by DARPA, researchers designed an experiment that uses a
combination of AI and robotics to help address the reproducibility crisis, by getting computers to
read scientific papers and understand them, getting Eve to attempt to reproduce the experiments.
Two different human teams used Eve and two breast cancer cell lines and attempted to reproduce
the 74 results. Statistically significant evidence for repeatability was found for 43 papers,
meaning that the results were replicable under identical conditions, and significant evidence for
reproducibility or robustness was found in 22 papers, meaning the results were replicable by
different scientists under similar conditions. In two cases, the automation made serendipitous Several years ago, robot scientist Eve was developed, a computer/robotic system that uses techniques from artificial intelligence (AI) to carry out scientific experiments.
