eRisk 2018:

Early risk prediction on the Internet


CLEF 2018 Workshop

Avignon, 10-14 September 2018

Find Out More

CLEF eRisk 2018:

Early risk prediction on the Internet


eRisk explores the evaluation methodology, effectiveness metrics and practical applications (particularly those related to health and safety) of early risk detection on the Internet. Early detection technologies can be employed in different areas, particularly those related to health and safety. For instance, early alerts could be sent when a predator starts interacting with a child for sexual purposes, or when a potential offender starts publishing antisocial threats on a blog, forum or social network. Our main goal is to pioneer a new interdisciplinary research area that would be potentially applicable to a wide variety of situations and to many different personal profiles. Examples include potential paedophiles, stalkers, individuals that could fall into the hands of criminal organisations, people with suicidal inclinations, or people susceptible to depression.

Participate


This is the second year that this lab runs and the lab has two main tasks:

Task 1: Early Detection of Signs of Depression

This is a continuation of the eRisk 2017 pilot task.

The challenge consists in performing a task on early risk detection of depression. The challenge consists of sequentially processing pieces of evidence and detect early traces of depression as soon as possible. The task is mainly concerned about evaluating Text Mining solutions and, thus, it concentrates on texts written in Social Media. Texts should be processed in the order they were created. In this way, systems that effectively perform this task could be applied to sequentially monitor user interactions in blogs, social networks, or other types of online media.

The test collection for this task has the same format as the collection described in [Losada & Crestani 2016]. The source of data is also the same used for eRisk 2017. It is a collection of writings (posts or comments) from a set of Social Media users. There are two categories of users, depressed and non- depressed, and, for each user, the collection contains a sequence of writings (in chronological order). For each user, his collection of writings has been divided into 10 chunks. The first chunk contains the oldest 10% of the messages, the second chunk contains the second oldest 10%, and so forth.

The task is organized into two different stages:

  • Training stage. Initially, the teams that participate in this task will have access to a training stage where we will release the whole history of writings for a set of training users (we will provide all chunks of all training users), and we will indicate what users have explicitly mentioned that they have been diagnosed with depression. The participants can therefore tune their systems with the training data. In 2018, the training data for Task 1 is composed of all 2017 users (2017 training users + 2017 test users)
  • Test stage. The test stage will consist of 10 sequential releases of data (i.e. done at different dates). The first release will consist of the 1st chunk of data (oldest writings of all test users), the second release will consist of the 2nd chunk of data (second oldest writings of all test users), and so forth. After each release, the participants will have a few days to process the data and, before the next release, each participating system has to choose between two options: a) emitting a decision on the user (i.e. depressed or non-depressed), or b) seeing more chunks. This choice has to be made for each user in the collection. If the system emits a decision, its decision will be final and it will be evaluated based on the correctness of the system's decision and the number of chunks required to make the decision (using a metric for which the fewer writings required to make the alert, the better). If the system does not emit a decision then it will have access to the next chunk of data, that is it will have access to more writings, but it will have a penalty for a “later emission”).

Evaluation: The evaluation will take into account not only the correctness of the system's output (i.e. whether or not the user is depressed) but also the delay taken to emit its decision. To meet this aim, we will consider the ERDE metric proposed in [Losada & Crestani 2016].

The proceedings of the lab will be published in the online CEUR-WS Proceedings and on the conference website.

To have access to the collection all participants have to fill, sign and send a user agreement form (follow the instructions provided here). Once you have submitted the signed copyright form, you can proceed to register for the lab at CLEF 2018 Labs Registration site

Important Dates

Task 2: Early Detection of Signs of Anorexia

This is a new task in 2018. The format, source of data and overall organization of the task are equivalent to those used for Task 1.

The challenge consists in performing a task on early risk detection of anorexia. The challenge consists of sequentially processing pieces of evidence and detect early traces of anorexia as soon as possible. The task is mainly concerned about evaluating Text Mining solutions and, thus, it concentrates on texts written in Social Media. Texts should be processed in the order they were created. In this way, systems that effectively perform this task could be applied to sequentially monitor user interactions in blogs, social networks, or other types of online media.

The test collection for this task has the same format as the collection described in [Losada & Crestani 2016]. The source of data is also the same. It is a collection of writings (posts or comments) from a set of Social Media users. There are two categories of users: those that have been diagnosed with anorexia, and a control group (non-anorexia). For each user, the collection contains a sequence of writings (in chronological order). For each user, his collection of writings has been divided into 10 chunks. The first chunk contains the oldest 10% of the messages, the second chunk contains the second oldest 10%, and so forth.

The task is organized into two different stages:

  • Training stage. Initially, the teams that participate in this task will have access to a training stage where we will release the whole history of writings for a set of training users (we will provide all chunks of all training users), and we will indicate what users have explicitly mentioned that they have been diagnosed with anorexia. The participants can therefore tune their systems with the training data.
  • Test stage. The test stage will consist of 10 sequential releases of data (i.e. done at different dates). The first release will consist of the 1st chunk of data (oldest writings of all test users), the second release will consist of the 2nd chunk of data (second oldest writings of all test users), and so forth. After each release, the participants will have a few days to process the data and, before the next release, each participating system has to choose between two options: a) emitting a decision on the user (i.e. anorexia or non-anorexia), or b) seeing more chunks. This choice has to be made for each user in the collection. If the system emits a decision, its decision will be final and it will be evaluated based on the correctness of the system's decision and the number of chunks required to make the decision (using a metric for which the fewer writings required to make the alert, the better). If the system does not emit a decision then it will have access to the next chunk of data, that is it will have access to more writings, but it will have a penalty for a “later emission”).

Evaluation: The evaluation will take into account not only the correctness of the system's output (i.e. whether or not the user has anorexia) but also the delay taken to emit its decision. To meet this aim, we will consider the ERDE metric proposed in [Losada & Crestani 2016].

The proceedings of the lab will be published in the online CEUR-WS Proceedings and on the conference website.

To have access to the collection all participants have to fill, sign and send a user agreement form (follow the instructions provided here). Once you have submitted the signed copyright form, you can proceed to register for the lab at CLEF 2018 Labs Registration site

Important Dates
08 NOV
  • Registration for lab opens
  • 8/11/2017

30 NOV
  • Release of the training data
  • 30/11/2017

06 FEB
  • Release of 1st chunk of test data
  • 06/02/2018

  • Release of 2nd chunk of test data
  • 13/02/2018

  • Release of 3rd chunk of test data
  • 20/02/2018

  • Release of 4th chunk of test data
  • 27/02/2018

  • Release of 5th chunk of test data
  • 06/03/2018

  • Release of 6th chunk of test data
  • 13/03/2018

  • Release of 7th chunk of test data
  • 20/03/2018

  • Release of 8th chunk of test data
  • 27/03/2018

  • Release of 9th chunk of test data
  • 03/04/2018

  • Release of 10th chunk of test data
  • 10/04/2018

  • Release of evaluation results to all participants
  • 24/04/2018

31 MAY
  • Task participant papers due
  • 31/05/2018

15 JUN
  • Notification of acceptance
  • 15/06/2018

29 JUN
  • Camera ready. Task participant papers
  • 29/06/2018

Organizers


More information


+34 881 816 451

CLEF 2018 Conference & CLEF initiative: