Scientific rigor means implementing the highest standards and best practices of the scientific method and applying those to one’s research. It is all about discovering the truth.
Scientific rigor involves minimizing bias in subject selection and data analysis. It is about determining the appropriate sample size for your study so that you have sufficient statistical power to be more confident about whether you are generating false positives or missing out on false negatives. It’s about conducting research that has a good chance of being replicated in your own lab and other laboratories.
The idea is to figure out under what range of conditions a particular outcome is generated — the wider the range, the higher the likelihood that the outcome reflects a scientific truth. If results can be replicated in one’s own lab, but cannot be reproduced in another lab, does not generalize well in other situations, or does not hold up over the test of time, the outcome might not be that important.
What’s at Stake
We want to know and like to think that our research is generalizable and robust. If the research is not conducted with scientific rigor, then generalizability and robustness are not going to happen.
And, if neuroscience research is not conducted with scientific rigor, then it’s a big waste of time, money, and energy in pursuing outcomes that are not real. We also risk the possibility of missing great insights because we didn’t conduct the research in a way that would allow us to detect those insights.
Some of the issues around blind analysis and randomization of subjects are particularly important for clinical applications because lives are going to depend on their outcomes.
We also want science to be objective and empirical. But we’ve known for a long time that when an experimenter knows what treatment group a patient, rat, or a mouse is in, then that knowledge can influence the way they look at the data or analyze the data. It’s not intentional, but it is experimenter bias. The same is true for patients. If they know they are in one group and not the other, then bias is introduced. It is therefore critical to know how large the placebo effect is. Ultimately we are after the truth. If people — either the experimenter or the subject — are not blind to their treatment group, that introduces bias that will cloud the truth.
Challenges
The challenges are mainly around the time it takes to be careful and rigorous in research. PIs are faced with many responsibilities ranging from teaching, completing administrative/committee work, administering grants, keeping the lab funded, and publishing papers. They are pushed and pulled in all directions. However, it takes time to ensure that there is scientific rigor, and implementing the best practices of scientific rigor will almost certainly increase the time to publication of one’s results.
My Lab’s Approach
Before any experiment is conducted, the student or postdoc has to write a plan of study so they think purposefully about their study in advance. The plan of study includes, among other points: rationale for the study, an answer to whether it is discovery science or hypothesis-driven (both have their place), sample size, determination of statistical power, detailed methods and procedures, and how data will be analyzed.
We also periodically read and discuss three papers that span 110 years of science. One is John Platt’s 1964 Science paper, “Strong Inference,” which talks about hypothesis testing and the importance of having alternative hypotheses and an experimental design intended to falsify (not support) your hypothesis. The second paper written in 1897 by Thomas Crowder Chamberlin discusses the method of multiple working hypotheses. More recently, a 2014 paper by Douglas Fudge, “50 Years of Platt’s Strong Inference,” revisits principles outlined in the 1964 Platt paper.
Students find the idea of designing experiments that will falsify your hypothesis, given a particular outcome, a challenging exercise — and so do I. We have a lot of fun doing this.
Forward Look
Through the Training Modules to Enhance Data Reproducibility grant (I am a co-PI), SfN seeks to raise awareness in the neuroscience community about the importance of scientific rigor and its application to our discipline. If we are going to make strides that stand the test of time, it is essential.
The next generation of neuroscientists needs to be aware of these important points. Of course, awareness will make the discipline stronger. We also need to provide necessary training to conduct research in a rigorous way.
In addition, the relationship between young investigators and trainees and their mentors is really important in learning about scientific rigor. It is my responsibility as a mentor and trainer to raise young neuroscientists in a way that makes scientific rigor and strong inference habits of mind and practice.
It is also important for young researchers just learning the ropes to seek training in this area. This can teach them to question both their own assumptions and those of others in their lab, and to do what they can as budding scientists to enhance the scientific rigor and credibility of their research.
About the Contributor
