CoNLL Shared Task 2019: Call for Proposals

 

SIGNLL (ACL's Special Interest Group on Natural Language Learning) invites proposals for the CoNLL Shared Task 2019.

Background

Since 1999, CoNLL (the Conference on Computational Natural Language Learning) has included a shared task in which training and test data are provided by the organizers, which allows participating systems to be evaluated and compared in a systematic way. Descriptions of the participating systems and an evaluation of their performance are presented both at the conference and in the proceedings.

Previous shared tasks

Previous shared tasks have been devoted to:

  • Chunking (1999, 2000)
  • Clause identifications (2001)
  • Named entity recognition (2002, 2003)
  • Semantic role labeling (2004, 2005)
  • Dependency parsing (2006, 2007)
  • Joint parsing of syntactic and semantic dependencies (2008, 2009)
  • Hedge detection (2010)
  • Co-reference resolution (2011, 2012)
  • Grammatical error correction (2013, 2014)
  • Universal morphological reinflection (2017, 2018)
  • Multilingual parsing from raw text to universal dependencies (2017, 2018)

Submission guidelines

For CoNLL-2019 proposals are invited for any shared task involving natural language learning, including not only natural language processing tasks and applications, but also the modeling of human language learning and processing. In recent years, a number of CoNLL shared tasks have used TIRA as a platform for evaluation purposes.

Proposals should contain the following information:
  • Names and affiliations of the members of the organizing committee (including a designated contact person)
  • Description of the shared task, including its relevance for the study of computational natural language learning
  • Description of the process for selecting, acquiring and preparing the data for training and evaluation, including conditions of availability
  • Description of the procedure for evaluating the performance of participating systems, including the availability of evaluation software (and whether TIRA will be used)
  • Any other relevant information
A tentative timeline for the CoNLL Shared Task 2019 is as follows:
  • January 14, 2019: deadline for proposals
  • January 24, 2019: confirmation of acceptance
  • March 1, 2019: first version of task website online
  • March 1, 2019: first call for papers out
  • April 15, 2019: all trial datasets (plus documentation and scorer) available
  • May 1, 2019: test data available
  • July 1, 2019: test phase ends
  • July 10, 2019: submission of system description papers

Proposals should make clear that a similar schedule can be implemented, especially for the preparation of resources (trial/training/development/test data sets).

Proposals should be 2 to 3 pages long, in PDF format.(We encourage proposers to include appendices with examples that help understanding the data and type of evaluation)

The deadline to submit a proposal is January 14, 2019. Proposals should be submitted by e-mail to proposals@signll.org, from the email of the contact person, with the subject “CoNLL-2019 Shared Task Proposal”.

Proposals will be reviewed by the SIGNLL steering committee. Proposers might be contacted during the reviewing period to provide further information. Potential proposers should feel free to indicate the intention to submit by emailing proposals@signll.org. We are also happy to answer your questions about the process, or to solicit early feedback for a proposal.