Shared Tasks 🛠️

Our AQuA shared tasks allow us to coordinate our work on quality assessment methods, understand what contributions already exist, build on the work of others, and objectively compare results. After working with translation experts to gather and annotate data for our various tasks, we publish these as shared tasks on this site along with instructions for utilizing the data, submitting methods to be evaluated/validated, and a leaderboard of previously submitted methods. This site will serve as the hub of our research activities and a starting point for developing assessment methods.

Each of these shared tasks are similar in that they all provide:

  • A general description of the task
  • Common, gold standard evaluation and validation data sets
  • A mechanism for submitting the results achieved with new methods
  • Common, centrally managed evaluation scripts to process submissions
  • A leaderboard showing how the various submitted methods compare to each other on a level playing field