Heafield, Kenneth

Renals, Stephen

Currey, Anna

European Research Council

2019-09-24T09:43:03Z

2019-09-24T09:43:03Z

2019-11-23

The goal of neural machine translation (NMT) is to build an end-to-end system that automatically translates sentences from the source language to the target language. Neural machine translation has become the dominant paradigm in machine translation in recent years, showing strong improvements over prior statistical methods in many scenarios. However, neural machine translation relies heavily on parallel corpora for training; even for two languages with abundant monolingual resources (or with a large number of speakers), such parallel corpora may be scarce. Thus, it is important to develop methods for leveraging additional types of data in NMT training. This thesis explores ways of augmenting the parallel training data of neural machine translation with non-parallel sources of data. We concentrate on two main types of additional data: monolingual corpora and structural annotations. First, we propose a method for adding target-language monolingual data into neural machine translation in which the monolingual data is converted to parallel data through copying. Thus, the NMT system is trained on two tasks: translation from source language to target language, and autoencoding the target language. We show that this model achieves improvements in BLEU score for low- and medium-resource setups. Second, we consider the task of zero-resource NMT, where no source ↔ target parallel training data is available, but parallel data with a pivot language is abundant. We improve these models by adding a monolingual corpus in the pivot language, translating this corpus into both the source and the target language to create a pseudo-parallel source-target corpus. In the second half of this thesis, we turn our attention to syntax, introducing methods for adding syntactic annotation of the source language into neural machine translation. In particular, our multi-source model, which leverages an additional encoder to inject syntax into the NMT model, results in strong improvements over non-syntactic NMT for a high-resource translation case, while remaining robust to unparsed inputs. We also introduce a multi-task model that augments the transformer architecture with syntax; this model improves translation across several language pairs. Finally, we consider the case where no syntactic annotations are available (such as when translating from very low-resource languages). We introduce an unsupervised hierarchical encoder that induces a tree structure over the source sentences based solely on the downstream task of translation. Although the resulting hierarchies do not resemble traditional syntax, the model shows large improvements in BLEU for low-resource NMT.

http://hdl.handle.net/1842/36195

en

The University of Edinburgh

Currey, A. and Heafield, K. (2018). Multi-source syntactic neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2961–2966. Association for Computational Linguistics.

Currey, A. and Heafield, K. (2018). Unsupervised source hierarchies for low-resource neural machine translation. In Proceedings of the Workshop on the Relevance of Linguistic Structure in Neural Architectures for NLP, pages 6–12. Association for Computational Linguistics.

Currey, A. and Heafield, K. (2019). Incorporating source syntax into transformer-based neural machine translation. In Proceedings of the Fourth Conference on Machine Translation. Association for Computational Linguistics.

Currey, A., Miceli Barone, A. V., and Heafield, K. (2017). Copied monolingual data improves low-resource neural machine translation. In Proceedings of the Second Conference on Machine Translation, pages 148–156. Association for Computational Linguistics.

Sennrich, R., Birch, A., Currey, A., Germann, U., Haddow, B., Heafield, K., Miceli Barone, A. V., and Williams, P. (2017). The University of Edinburgh’s neural MT systems for WMT17. In Proceedings of the Second Conference on Machine Translation, pages 389–399. Association for Computational Linguistics.

machine translation

automatically translating

pivot language

monolingual corpora

structural annotations

neural machine translation

Moving beyond parallel data for neural machine translation

Thesis or Dissertation

Doctoral

PhD Doctor of Philosophy