Open Science Hypothesis Testing

The evaluation of hypotheses is a core feature of research in the behavioural, social, and biomedical sciences. Over the last decade, the replication crisis has drawn a lot of attention to high rates of false-positive results of hypothesis tests in the literature, caused by problems such as publication bias (selective publication of positive results), ‘questionable research practices’, and statistical misconceptions. This course will give participants a non-technical introduction to the theoretical basis of hypothesis evaluation from different perspectives (frequentist, Bayesian, information theoretic) and teach how to apply hypothesis evaluation appropriately to avoid fooling oneself and others.

The course is targeted at students and researchers who want to improve their understanding of how to evaluate theory-based hypotheses.

The first day of the course will cover classic null-hypothesis significance testing (NHST), common problems with NHST (misconceptions, questionable research practices, publication bias), open-science practices to avoid these problems, and how to draw more informative inferences with equivalence testing.

The second day will focus on hypothesis evaluation using model selection. Model selection provides an alternative to dichotomous decisions that are the default in NHST and allows more nuanced inferences. Two types will be discussed: Bayesian model selection (BMS) and information theoretic model selection, that is, model selection using information criteria. For both types, the focus will be on informative, theory-based hypotheses (as opposed to null hypothesis testing). The methods that will be covered include Bayes factors, posterior model probabilities, the AIC-type criteria called the GORIC and GORICA, and GORIC(A) weights.

The third day of the course will address informative hypothesis evaluation for the evaluation of replication studies, including both direct and conceptual replications (i.e., of multiple studies). Attention will be paid to (among others) hypothesis updating and combining evidence from multiple studies addressing the same research question (using both BMS and the GORICA).

Participants are requested to bring their own laptop computer. Software will be available online.

Among our Methodology and Statistics postgraduate courses, there is another course that deals with Bayes: A gentle introduction to Bayesian Estimation 
This course focuses on Bayesian statistics in terms of prediction (focusing on SEM) but does not cover hypothesis testing nor Bayes factors.
The course takes place in the week prior to our course and may be a nice prequel.