More and more researchers rely upon Systematic Reviews: attempts to synthesize the state of the art in a particular scientific field. However, the scientific output of the world doubles every nine years. In this tsunami of new knowledge, there is not enough time to read everything – resulting in costly, abandoned, or error-prone work. Using the latest methods from the field of Artificial Intelligence (AI), you can reduce the number of papers to screen up to 95%(!). This summer school course introduces you to the open-source software ASReview to help you speed up your systematic review.
Performing a systematic review is a very rigorous process, which is increasingly intensive due to the ever-growing number of scientific publications to review. Nevertheless, systematic reviews are pivotal for scholars, clinicians, policymakers, journalists, and, ultimately, the general public. Developing a search strategy for a systematic review is an iterative process to balance recall, precision, and quality. That is, including as many potentially relevant and – ideally – high-quality studies as possible (recall or sensitivity) while simultaneously limiting the total number of studies to screen (precision or specificity). In light of the time-consuming and costly process of conducting rigorous systematic reviews with the constant growth of scientific publications, reports, guidelines, and other data sources, recent advances in natural language processing (NLP), text mining, and machine learning have produced new algorithms that can accurately mimic human endeavour in systematic review activity, faster and more cheaply.
Within this course, every day consists of both lectures and do-it-yourself computer labs. The talks will be provided by a multidisciplinary team of experts from different fields: statistics, systematic reviewing, data science, open science, bibliometrics and transparent software engineering. For the computer sessions, we have a team ready to help you.
On the first day of the course, we compare the classical manual-based pipeline of performing a systematic review using the PRISMA steps with the AI-aided approach using screening prioritization. We assume participants are familiar with PRISMA. If not, you are requested to read the information on the PRISMA-website before the start of the summer school: http://prisma-statement.org/. In the afternoon, we will work with the open-source software ASReview (www.asreview.nl) by using example datasets to experience the benefits of using active learning. Make sure to have installation rights on your pc!
The second day will be dedicated to systematically obtaining the perfect dataset. The basics of searching online databases will be discussed using examples and demos: composing a search query and getting the highest quality of data (e.g., complete abstracts). Because of the use of active learning, the size of the dataset can be different compared to a classical systematic review. How does this affect your search? Is there still a need to search multiple databases? How do you process these large datasets? A much larger dataset can be screened with the same effort.
The third day will be devoted to an in-depth explanation of the different feature extraction techniques (TF-IDF, word2vec, sBert), classifiers (e.g., Naive Bayes, SVM, neural nets), the query strategies (certainty, uncertainty, random sampling) and balancing strategies to deal with the highly sparse relevant papers in the dataset. Although this part of the course is technical, we consider it essential to better understand how AI works if you want to use AI-aided tools (and to be able to answer questions from your supervisors, reviewers, peers, and friends).
The fourth day is about Open Science.
In the morning, we will discuss FAIR data for AI-aided systematic reviews. That is, although sharing the search query and data is part of the PRISMA checklist, sharing the complete (meta)data underlying a systematic review, including all labeling decisions, is not standard. Therefore, we will discuss a data-sharing protocol, including the importance of persistent identifiers (DOIs), abstract retrieval, and trusted repositories. Moreover, it is not enough to make the search query and the meta-data FAIR (Findable, Accessible, Interoperable, and Reusable). The AI also makes decisions throughout the process, which should be made FAIR as well. Therefore, all settings of the AI and every iteration of the model have to be stored and made human-readable. We will explain this process and demonstrate how this can be done.
In the afternoon, we organize an interactive session on the future of Open Science.
The fifth day consists of Q&A sessions and consultations.
Laura Hofstee, Rens van de Schoot , Daniel Oberski, Jan de Boer, Felix Weijdema, Jelle Teijema.
Participants from various fields — including psychology, education, human development, public health, prevention science, sociology, marketing, business, biology, medicine, political science, and communication — will benefit from the course.
It is helpful if you have a concrete plan for carrying out a systematic review to start working with your own data immediately.
For an overview of all our summer school courses offered by the Department of Methodology and Statistics please click here.
Aim of the course
After engaging in the course lectures and discussions, and completing the hands-on practice activities, participants will be able to carry out their AI-aided systematic review using active learning.
Please note that there are no graded activities included in this course. Therefore, we are not able to provide students with a transcript of grades. You will obtain a certificate upon completion of this course.
The tuition fee for PhD students from the Faculty of Social and Behavioural Sciences from Utrecht University will be funded by the Graduate School of Social and Behavioural Sciences.
There are no scholarships available for this course.
We also offer tailormade M&S courses and in-house M&S training. If you want to look the possibilities, please contact us at firstname.lastname@example.org
Team M&S Summer School | E: email@example.com