This event took place on April 10, 2018.
Scientific rigor is fundamental to two key goals of the scientific endeavor: knowledge creation and the translation of discoveries into new therapies. Research must be well-designed and implemented, accessible, and reproducible to reap the benefits of new discoveries.
For this virtual conference, SfN has partnered with NIH and neuroscientists around the world to provide you — no matter your career stage — with training resources to enhance rigor in experimental design, data analysis, and reporting of your research findings. This virtual conference is provided open access to the field and is supported by Grant Number R25DA041326 from the National Institute on Drug Abuse (NIDA). The contents of this series are solely the responsibility of the Society for Neuroscience and do not necessarily reflect the official views of NIDA.
The insights, experiences, and practical approaches for enhancing the rigor and transparency of neuroscience research you will hear from experts in a breadth of disciplines will cover various topics, including:
- Differences between early stage discovery, hypothesis development, and formal hypothesis testing, and how to incorporate rigor at all stages.
- Experimental design and implementation to minimize bias.
- Considerations for data gathering and analysis, such as sample size planning, group compilation, statistical power, and the meaning of statistical significance.
- NIH training resources and requirements for improving rigor and reproducibility in biomedical research.
- Transparent publishing practices and their effect on enhancing rigor in neuroscience research.
- How the field can incentivize rigorous research.
Visit the Neuronline collection, Promoting Awareness and Knowledge to Enhance Scientific Rigor in Neuroscience, to watch six related webinars on demand and explore additional articles and videos.
Click on each session title to reveal its description.
Speakers: Li-Huei Tsai, Cheryl Sisk, Emanuel DiCicco-Bloom, Os Steward
Time: 9:00 a.m. – 9:25 a.m. EDT
Distinguished faculty will discuss how they implement elements of scientific rigor and transparency in their research and training programs for their students and postdocs. As working scientists, Li-Huei Tsai, Cheryl Sisk, Emanuel DiCicco-Bloom, and Os Steward will share how they transition from testing and validating protocols and collecting preliminary data for hypothesis development, to designing a hypothesis-testing experiment that fully incorporates issues of scientific rigor, such as:
- Defining necessary controls.
- Power analyses.
- Random assignment to groups.
- Data collection to minimize unintentional bias.
- Plans for data analysis.
Speakers: Brian Caffo, Os Steward
Time: 9:25 a.m. – 10:00 a.m. EDT
Brian Caffo will share his perspective as a biostatistician on incorporating rigor and transparency into a study design — regardless of the form it takes. He will define key breakdowns in the statistical analysis of data, describe how those breakdowns can undermine reproducibility and rigor, and present simple rules for navigating statistical analyses. After his presentation, Brian will be joined by Os Steward for a short question-and-answer session.
Speakers: Rita Balice-Gordon, Theresa Hernández
Time: 10:10 a.m. – 11:00 a.m. EDT
Many sources of bias can affect the different stages of the experimental process — from design and execution, through data analysis and interpretation. These biases can range from obvious ones — such as what can occur during selection of samples — to unconscious biases that may be more difficult to identify — such as decisions that may bias you towards your original hypothesis, rather than following the path of where your data lead. In this session, Rita Balice-Gordon, who works in the pharmaceutical industry, and Theresa Hernández, who studies complementary and alternative medicines, will share insight into how they identify and mitigate various sources of bias in their research programs, particularly when exploring new therapeutic approaches. Presentations will be followed by a short question-and-answer session.
Speakers: Meaghan Creed, Deanna Barch, Mary Harrington
Time: 11:10 a.m. – 12:25 p.m. EDT
Researchers from different neuroscience disciplines will address how they approach data collection and analysis with rigor and transparency and how they are integrating best practices for managing and analyzing data into their work. Meaghan Creed will focus on how she designs behavioral and electrophysiology experiments with her analyses in mind — from replicate groups to sample sizes. Next, based on her work in the human imaging field, Deanna Barch will cover best practices in hypothesis-driven and data-driven analytic approaches, including methods for addressing power and multiple comparisons. Finally, Mary Harrington will discuss her decision to share her complete data sets and analysis code with the scientific community through the Open Science Framework, and how she and her undergraduate students stay up-to-date on evolving open science research practices to enhance transparency. Individual presentations will be followed by a panel discussion and Q&A with all speakers.
Speakers: Kristine Willis, Shiva Singh
Time: 12:45 p.m. – 1:15 p.m. EDT
NIH continues to update and refine its requirements and resources for training graduate students and postdoctoral fellows in the areas of rigor and reproducibility. In this session, Kristine Willis from the National Institute of General Medical Sciences will share the latest updates regarding NIH requirements in research and training grant applications for enhancing reproducibility and rigor. She will also discuss training resources that support NIH’s ongoing effort to promote reproducible and rigorous research, quantitative and computational skills development, and the sociology and ethics of decision-making related to reproducibility and rigor.
Speakers: Marina Picciotto, Elena Porro, Malcolm Macleod
Time: 1:25 p.m. – 2:40 p.m. EDT
Transparent reporting of experimental methodologies and results is critical for enhancing rigor and increasing reproducibility of neuroscience research. In this session, Marina Picciotto, Elena Porro, and Malcolm Macleod will address recent changes in scientific publishing to enhance transparency. Speakers will discuss initiatives, such as revised reporting guidelines from JNeurosci, Cell Press’ STAR Methods, and the Nature Publishing Group’s author checklist, that can help standardize reporting of experimental details and statistical analyses. Emerging initiatives — such as preprints and registered reports and protocols — will also be discussed. Brief individual presentations will be followed by a panel discussion and Q&A with all speakers.
Speakers: Marcus Munafò, Kate Button
Time: 2:50 p.m. – 3:50 p.m. EDT
Researchers can enhance the rigor and reproducibility of neuroscience research by making their research protocols, data, and analyses more transparent and accessible. In this session, Marcus Munafò and Kate Button will focus on the actionable steps individual scientists — regardless of their field — can employ to integrate rigor and transparency into their research programs, emphasizing open science initiatives, such as pre-registration of protocols and open data practices. Individual presentations will be followed by a moderated Q&A discussion period, which will provide an opportunity to explore how various open science practices can be implemented in different settings.
Speakers: Os Steward, Emanuel DiCicco-Bloom, Cheryl Sisk
Time: 4:00 p.m. – 4:45 p.m. EDT
Rigorous conduct of neuroscience research is influenced by many factors beyond the actual conduct of science. These factors, often called “perverse incentives,” can include:
- Pressure on scientists to increase the breadth or flashiness of their research while minimizing caveats and sacrificing rigor.
- Pressure to rush research into publication to meet career or funding deadlines.
- Disincentives for publishing negative results or replication studies, and many more.
In this closing session, Os Steward, Emanuel DiCicco-Bloom, and Cheryl Sisk will engage conference attendees in a discussion about these and other factors that warrant collective action by the field. They will emphasize the shared responsibility all stakeholders —funding agencies, publishers, scientific academy, and others — play in recognizing these perverse incentives and rewarding efforts that enhance rigor and reproducibility.