Research Rigor is a Mindset — A Bioinformatician Bench Biologist’s Conception
- Featured in:
- Rigor and Reproducibility
“Rigor” is the vaguest sought-after quality of research. If you attach a license to your work, you can say it’s open. If you guide a collaborator through a protocol, your work is reproducible. But what do you need to do to encourage reviewers and colleagues to reach for the word “rigorous” when describing your work? Any brief answer I could give risks turning out narrowly precise or unhelpfully elusive. I’ll sidestep my personal definition and suggest that your own specific path to rigor can be achieved through the fostering of a mindset.
We have a variety of mindsets we bring to specific occasions. Leaving cognitive science to the cognitive scientists, I’m defining “mindset” as a collection of attitudes and behaviors that are situational, implicit, and over time become automatic. A vacation mindset, for example, prompts me to open my wallets to spend more money than usual and move the acceptable time for an espresso martini several hours earlier than 5 p.m. (for me at least). My recommendation for thinking about rigor — especially in the area of data analysis — is to rely on a mindset you likely possess: the experimenter’s mindset.
You may or may not regularly do experiments, but assuming you’ve taken a lab course, you probably have habits of mind associated with laboratory work. Entering a lab, you’d likely reach for gloves and goggles, expect a checklist or protocol, and have some intuitions about what outcomes are desired. The more you work in a lab, the more your confidence can grow, and the more you can diagnose and solve ordinary problems without much thought.
When teaching scientists to analyze data or adopt new computing skills, I often find that many good habits and mindsets get left behind. We wouldn’t work with biohazard materials without PPE, but people keep single-copy datasets on 10-year-old hard drives. I don’t think anyone would willingly mouth pipette clinical samples, but some people’s data analysis depends on manually editing Excel spreadsheets. Having great coding skills doesn’t confer infallibility. In less-than-rigorous moments, developers fail to achieve the skepticism of a true experimentalist. A result: machine learning models used to diagnosed skin cancers which used dermatologist’s skin markings as a correlate of disease.
Here are three ways that approaching data and computing problems with an experimenter’s mindset can place you on a path to being rigorous.
- Stay positive.
Positivity is a valuable asset for working towards rigor. This advice isn't suggesting naïve optimism (see my next recommendation). However, I think you need to be motivated and convinced that your ultimate goal is something that is achievable. To achieve rigor, you’ll need to define what the term means for your work, and then you can make progress with a specific goal in mind. Like any goal, you may not complete it on your first attempt, but even incremental progress may have value. What’s more thrilling than getting an experimental system to work after numerous attempts? I don’t think it hurts to attach similar feelings to achieving more rigorous outcomes. - Document and build understanding.
Armed with positivity, central to rigor is the idea that we need to answer questions from multiple angles and with the right tools to expose flaws in our reasoning, methodology, or data acquisition. I think many of us are inherently skeptical about our work. To make this mindset productive, rigor offers a path to purposefully articulate and document not only our work but our assumptions. We can make this approach concrete at multiple levels. No matter how often or how routine an experiment is in the lab, it should be documented in a notebook. The same is true when working with data and software. We need to do routine things like documenting the software (including versions and even builds) and having backups and checksums on our data. If you are just building your computational skills, you’ll need training to know how to do these things properly and build effective habits. At the next level of rigor, you also need to insist on understanding how the software you use works. What assumptions does this software depend on? How was the software tested and developed? How sensitive and/or specific is the software? All of these are easy places for errors and mistakes to creep in. Substitute software for statistical tools or really anything we depend on for our research. Often, you’ll need to seek out and rely on the expertise of others. At the very least, be intentional about the level of understanding you need to use tools you aren’t an expert with. - Work openly and safely.
Related to the above, there are two specific ways working with data can be made more open, and thereby safer (i.e., less prone to error). The most important skill is using the proper tools to document how analyses were done. With data and code, this often takes the form of working with version control (e.g., git/GitHub). These take time to master and there is help to learn. Without suggesting the perfect is the enemy of the good, it's OK to start out with notes on your data analysis and data management strategy. However, there are very good reasons why software and data management strategies used by the experts are the ones you should strongly consider adopting. As your skills grow, you’ll also need to learn how to write scripts and automate your analyses to reduce the chance for human error. Being “safe” with your data means being aware of the best practices, developing skills, and collaborating with your knowledgeable colleagues before making decisions. Being open about where you need advice is a fantastic mindset to have here that will save you time and aggravation later — although you may have to deal with some of that upfront. Openness can also be applied to your study design in general and with approaches like study preregistration, which allows us to document everything in advance. While this level of formality may not always be required, there are things we do in the research lab that can be a map to good habits of mind in working with data.
Think about research rigor like you would an experiment: failure will be involved, but you can always learn and improve.
About the Contributor
