Jul 1, 2020 11:16 PM
https://www.chemistryworld.com/news/chem...93.article
INTRO: In October last year, a team of natural product chemists discovered a glitch in a widely used piece of NMR software. Buried deep inside the code was a simple file sorting issue, which on certain operating systems led to incorrect values being predicted for chemical shifts. The finding cast uncertainty over results published in more than 150 scientific papers over a five year period.
This is not the first time that an error in a piece of software code has cast a shadow over computational research, these sorts of issues are actually surprisingly common. In one famous case, a coding error was at the heart of a seven-year dispute between some of the world’s top theoretical chemists, who were trying to model the phases of supercooled water. And recently, an algorithm used in older versions of the popular molecular dynamics software Gromacs was found to introduce order of magnitude mistakes during simulations.
Ideally, code will be well documented and publicly available, allowing researchers to scrutinise scripts and locate problems. But this isn’t always the case – traditional publishing practices, as well as concerns around intellectual property, often mean that code is difficult or even impossible to access.
Even when source code is open for all to see, other factors can complicate matters. [See Black box in Machine Learning] Computer programs tend to rely on an array of other pieces of software, which are continually being updated and new versions introduced. This makes repeating the exact conditions that a computational study was originally performed under surprisingly difficult. These problems have become so widespread that a ‘reproducibility crisis’ is now a major concern among computational scientists... (MORE - details)
Is AI Facing A Reproducibility Crisis?
https://www.youtube-nocookie.com/embed/fNpWMlGL3Ys
INTRO: In October last year, a team of natural product chemists discovered a glitch in a widely used piece of NMR software. Buried deep inside the code was a simple file sorting issue, which on certain operating systems led to incorrect values being predicted for chemical shifts. The finding cast uncertainty over results published in more than 150 scientific papers over a five year period.
This is not the first time that an error in a piece of software code has cast a shadow over computational research, these sorts of issues are actually surprisingly common. In one famous case, a coding error was at the heart of a seven-year dispute between some of the world’s top theoretical chemists, who were trying to model the phases of supercooled water. And recently, an algorithm used in older versions of the popular molecular dynamics software Gromacs was found to introduce order of magnitude mistakes during simulations.
Ideally, code will be well documented and publicly available, allowing researchers to scrutinise scripts and locate problems. But this isn’t always the case – traditional publishing practices, as well as concerns around intellectual property, often mean that code is difficult or even impossible to access.
Even when source code is open for all to see, other factors can complicate matters. [See Black box in Machine Learning] Computer programs tend to rely on an array of other pieces of software, which are continually being updated and new versions introduced. This makes repeating the exact conditions that a computational study was originally performed under surprisingly difficult. These problems have become so widespread that a ‘reproducibility crisis’ is now a major concern among computational scientists... (MORE - details)
Is AI Facing A Reproducibility Crisis?