“Data peeking without p-hacking? A guide to Sequential Analysis”
Hosted by the Cognitive, Computational and Systems Neuroscience Pathway (CCSN)
Agenda
- 2:30—3:00 pm (CDT): Presentation “Data peeking without p-hacking? A guide to Sequential Analysis”
- 3:30—4:30 pm (CDT): Q & A, prioritizing trainees
Abstract: When designing an experiment, it is essential to think about the sample size you
need to collect. However, due to uncertainty in the effect size you will observe, it is often difficult to perform an a-priori power analysis. It would often be much more convenient to collect data in small batches and repeatedly analyze the data, were it not for the fact that this is known as a ‘p-hacking’, and inflates the Type 1 error rate. In this workshop we will discuss statistical techniques to sequentially analyze data in small batches, or even after every participant, without inflating error rates. Such approaches are efficient and flexible ways to collect data when the expected effect size is uncertain. We will focus on sequential analyses technique developed in biostatistics. This workshop has a pragmatic, hand-on approach, and we will discuss how to incorporate feasibility constraints, stopping either after concluding the presence or the absence of a meaningful effect, and free software tools that exists to design and analyze sequential trials.
Brief bio: Daniel Lakens is an experimental psychologist working at the Human-Technology Interaction group at Eindhoven University of Technology. In addition to his empirical work in cognitive and social psychology, he works actively on improving research methods and statistical inferences, and has published on the importance of replication research, sequential analyses and equivalence testing, and frequentist statistics. He was involved in establishing dedicated grants for replication studies by the Dutch science funder NWO, and with Brian Nosek co-edited a special issue with the first Registered Reports in psychology in 2014. His lab is funded until 2022 by a VIDI grant on a project that aims to improve the reliability and efficiency of psychological science. He teaches about better research practices on Coursera, and received the LeamerRosenthal Prize for Open Social Science in 2017 for his course ‘Improving Your Statistical Inferences’ in which more than 50.000 learners have enrolled.
