Beyond null-hypothesis testing

Description

Participants will learn to apply open-science principles and cutting-edge statistical techniques that ensure maximally informative analyses. Contemporary issues - such as publication bias, questionable research practices, the statistical evaluation of (non-null) hypotheses, and methods for evaluating the same research question with multiple (replication) studies - are also addressed. 

The course will be non-technical in nature and is targeted at PhD students and researchers who want to apply the presented approaches to their own data.

The evaluation of hypotheses is a core feature of research in the behavioural, social, and biomedical sciences. Over the last decade, the replication crisis has drawn a lot of attention to high rates of false-positive results of hypothesis tests in the literature, caused by problems such as publication bias (selective publication of positive results), ‘questionable research practices’, and statistical misconceptions. This course will give participants a non-technical introduction to the theoretical basis of hypothesis evaluation from different perspectives (frequentist, information theoretic, and Bayesian,) and teach how to apply hypothesis evaluation appropriately to avoid fooling oneself and others.

The course is targeted at students and researchers who want to improve their understanding of how to evaluate theory-based hypotheses.

The first day of the course will cover classic null-hypothesis significance testing (NHST), common problems with NHST (misconceptions, questionable research practices, publication bias), open-science practices to avoid these problems, and how to draw more informative inferences with equivalence testing.

The second day will focus on hypothesis evaluation using model selection. Model selection provides an alternative to dichotomous decisions that are the default in NHST and allows more nuanced inferences. Two types will be discussed: information theoretic model selection, that is, model selection using information criteria, and Bayesian model selection (BMS). For both types, the focus will be on informative, theory-based hypotheses (as opposed to null hypothesis testing). The methods that will be covered include the AIC-type criteria called the GORIC and GORICA, GORIC(A) weights, Bayes factors, and posterior model probabilities.

The third day of the course will address informative hypothesis evaluation for multiple (replication) studies, including both direct and conceptual replications. Attention will be paid to (among others) hypothesis updating and combining evidence from multiple studies addressing the same research question (using both GORICA and BMS).

Each day consists of lectures, including small hands-on sessions, and ends with a lab meeting, where there is also room to work on your own data.

Participants are requested to bring their own laptop. Software will be available online.

Lecturers

Dr. Rebecca Kuiper and  dr. Anne Scheel

Target audience

The course will be non-technical in nature and is targeted at students and researchers who want to use the approaches presented for the evaluation of their own data. 

The participants can come from a variety of fields, such as sociology, psychology, education, human development, marketing, business, biology, medicine, political science, and communication sciences.

For an overview of all our summer courses offered by the Department of Methodology and Statistics please click here.

We also offer tailor-made M&S courses and in-house M&S training. If you want to look at the possibilities, please contact Dr. Laurence Frank at pe.dsai@uu.nl. 

Aim of the course

After attending the course:

• you understand the epistemological basis of hypothesis testing as well as frequentist, Bayesian, and information-theoretic approaches to evaluate hypotheses statistically.
• you have obtained the practical skills necessary to evaluate hypotheses with these approaches, using open-science practices to conduct and report your research transparently and reduce the risk of error and bias.
• you learned how to evaluate replication studies.
• you learned how to combine the evidence from multiple diverse studies.

Study load

3 full days: 

Each day consists of lectures including small hands-on exercises and ends with a lab meeting (where there is also room to work on your own data).

Please note that there are no graded activities included in this course. Therefore, we are not able to provide students with a transcript of grades. You will obtain a certificate upon completion of this course.

Tags

• Hypothesis testing
• JASP
• R
• model selection
• Bayes
• information criteria
• equivalence testing
• Open Science
• preregistration
• reproducibility