Project proposal for the 2017/2018 MPhil course in Advanced Computer Science.

Note: Please discuss with me before applying for this project. I am available for meetings either before Nov 13th or after November 17th.

Language Modelling for Neural Error Detection

Proposer: Marek Rei
Supervisor: Marek Rei

Description

Automated error detection requires systems that are able to identify various types of errors in texts written by language learners. This includes basic errors, such as spelling or punctuation mistakes, but also more complex errors such as word order, verb agreement, incorrect idioms, etc.
For example:
_ _ X X _ _ _ _ _ _ We can invite also people who are not members . _ _ X _ _ _ _ _ My friend eats two ice creams yesterday .

Recent work has shown that neural bidirectional LSTM models perform well at this task (Rei and Yannakoudakis (2016), Rei et al. (2016)). However, these models are trained using hand-annotated datasets, which are much smaller than available unlabeled corpora.

Rei (2017) showed that using a language modelling objective on the same labelled data improved performance for error detection, among other sequence labelling tasks. Peters et al. (2017) proposed a method for explicitly including unlabelled data, by training a separate language model and integrating features from its hidden state into the sequence labeller. They showed performance improvements on the CoNLL 2000 chunking datatset and CoNLL 2003 named entity recognition dataset.

This project will extend the latter approach to the task of error detection in learner texts. Since language models are designed to detect unusual word sequences and require only unlabelled data for training, their features can potentially be quite useful for the error detection model.

The existing error detection system, written in Theano, is available online:
https://github.com/marekrei/sequence-labeler
A Tensorflow version of this code will also be available soon. The project can start from this codebase and modify it to include the necessary extensions.

For evaluating performance on error detection, the manually labelled FCE (Yannakoudakis et al., 2011) and CoNLL14 (Ng et al., 2014) datasets are available.

References:

Compositional Sequence Labeling Models for Error Detection in Learner Writing
Marek Rei and Helen Yannakoudakis. 2016.

Attending to characters in neural sequence labeling models
Marek Rei, Sampo Pyysalo and Gamal K.O. Crichton. 2016.

Semi-supervised Multitask Learning for Sequence Labeling
Marek Rei. 2017.

Semi-supervised sequence tagging with bidirectional language models
Matthew E. Peters, Waleed Ammar, Chandra Bhagavatula, Russell Power. 2017

Automatic Annotation and Evaluation of Error Types for Grammatical Error Correction
Christopher Bryant Mariano Felice Ted Briscoe. 2017

A New Dataset and Method for Automatically Grading ESOL Texts
Helen Yannakoudakis, Ted Briscoe, Ben Medlock. 2011

The CoNLL-2014 Shared Task on Grammatical Error Correction
Ng et al. 2014