Dialog State Tracking Challenge 2 & 3

News

DSTC 3 is now underway, see the announcement for more details and a brief introduction to the challenge. The labelled seed data is now available for download (dstc3_seed.tar.gz).

The results for the DSTC 2 have been released, there were 9 teams participating with 31 entries total. The results may be downloaded in the section to the right. The Featured Metrics are also presented in a table: DSTC 2 Featured Metrics.

The Dialog State Tracking Challenge (DSTC) is a research challenge focused on improving the state of the art in tracking the state of spoken dialog systems. State tracking, sometimes called belief tracking, refers to accurately estimating the user's goal as a dialog progresses. Accurate state tracking is desirable because it provides robustness to errors in speech recognition, and helps reduce ambiguity inherent in language within a temporal process like dialog.

The first DSTC was a success, with 9 teams participating and a total of 27 entries. The data, publications and other materials are available on the DSTC 1 website. DSTC 2 then introduced more complicated and dynamic dialog states, which may change through the dialog, in a new domain (restaurant information). DSTC 3 presents the challenge of adapting to a new domain with a small amount of seed data, and lots of data in a similar but smaller domain.

Until recently, many state tracking models and approaches had been shared, but direct comparisons were impossible. A shared research task like this facilitates direct comparisons among state tracking models, helping to advance the state-of-the art.

Organized by

University of Cambridge Department of Engineering Microsoft Research

Endorsed by

SIGdial

Challenge Overview

In this challenge, participants are given labelled corpora of dialogs to develop state tracking algorithms. The trackers will then be evaluated on a common set of held-out dialogs which are released, un-labelled, during a one week period.

The corpus was collected using Amazon Mechanical Turk, and consists of dialogs in two domains: restaurant information, and tourist information. Tourist information subsumes restaurant information, and includes bars, cafés etc. as well as multiple new slots. There will be two rounds of evaluation using this data:

  • DSTC 2 will release a large number of training dialogs related to restaurant search. Compared to DSTC (which was in the bus timetables domain), DSTC 2 will introduce changing user goals, tracking 'requested slots' as well as the new restaurants domain. Results from DSTC 2 will be presented at SIGDIAL 2014.
  • DSTC 3 will address the problem of adapation to a new domain -- tourist information. DSTC 3 will release a small amount of labelled data in the tourist information domain; participants will use this data plus the restaurant data from DSTC 2 for training.

Dialogs used for training are fully labelled; user transcriptions, user dialog-act semantics and dialog state are all annotated. (This corpus therefore is also suitable for studies in Spoken Language Understanding.)

After each round of evaluation, the labelled test sets will be released, along with the output of the trackers entered into the challenge.

For more detailed information, please see the handbook.

Mailing List

To join the mailing list, send an email to listserv@lists.research.microsoft.com with 'subscribe DSTC' in the body of the message (without quotes). Joining the list is encouraged for those with an interest in the challenge, and is a necessity for those participating.

Post to the list using the address: dstc@lists.research.microsoft.com.

Important Dates

DSTC 2

  • 7th October 2013 Labelled restaurant information train and development set released
  • 20th January 2014 Unlabelled restaurant information test set released
  • 27th January 2014 Tracker output on restaurant information test set due
  • 3rd February 2014 Results on restaurant information test set given to participants
  • 5th March 2014 Approximate SIGdial deadline
  • June 2014 Results presented at SIGDIAL Conference

DSTC 3

  • 4th April 2014 Labelled tourist information seed set released
  • 9th June 2014 Unlabelled tourist information test set released
  • 16th June 2014 Tracker output on tourist information test set due
  • 23rd June 2014 Results on tourist information test set given to participants
  • [venue for reporting results to be confirmed]