Lopez, Adam

Heafield, Kenneth

Bogoychev, Nikolay Veselinov

2019-07-26T09:07:52Z

2019-07-26T09:07:52Z

2019-07-01

Parallel systems have been widely adopted in the field of machine translation, because the raw computational power they offer is well suited to this computationally intensive task. However programming for parallel hardware is not trivial as it requires redesign of the existing algorithms. In my thesis I design efficient algorithms for machine translation on parallel hardware. I identify memory accesses as the biggest bottleneck to processing speed and propose novel algorithms that minimize them. I present three distinct case studies in which minimizing memory access substantially improves speed: Starting with statistical machine translation, I design a phrase table that makes decoding ten times faster on a multi-threaded CPU. Next, I design a GPU-based n-gram language model that is twice as fast per £ as a highly optimized CPU implementation. Turning to neural machine translation, I design new stochastic gradient descent techniques that make end-to-end training twice as fast. The work in this thesis has been incorporated in two popular machine translation toolkits: Moses and Marian.

http://hdl.handle.net/1842/35886

en

The University of Edinburgh

Bogoychev, N. and Hoang, H. (2016). Fast and highly parallelizable phrase table for statistical machine translation. In Proceedings of the First Conference on Machine Translation, pages 102–109, Berlin, Germany. Association for Computational Linguistics.

Bogoychev, N., Junczys-Dowmunt, M., Heafield, K., and Aji, A. F. (2018). Accelerating asynchronous stochastic gradient descent for neural machine translation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium.

Bogoychev, N. and Lopez, A. (2016). N-gram language models for massively parallel devices. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers.

Haddow, B., Bogoychev, N., Emelin, D., Germann, U., Grundkiewicz, R., Heafield, K., Miceli Barone, A. V., and Sennrich, R. (2018). The University of Edinburgh‘s Submissions to the WMT18 News Translation Task. In WMT 2018, Brussels, Belgium. Association for Computational Linguistics.

Hoang, H., Bogoychev, N., Schwartz, L., and Junczys-Dowmunt, M. (2016). Fast, scalable phrase-based SMT decoding. CoRR, abs/1610.04265.

Junczys-Dowmunt, M., Grundkiewicz, R., Dwojak, T., Hoang, H., Heafield, K., Neckermann, T., Seide, F., Germann, U., Fikri Aji, A., Bogoychev, N., Martins, A. F. T., and Birch, A. (2018). Marian: Fast neural machine translation in C++. In Proceedings of ACL 2018, System Demonstrations, pages 116–121, Melbourne, Australia. Association for Computational Linguistics.

machine translation systems

memory speed

machine translation

efficient algorithms

processing speed

phrase tables

GPU-based n-gram language model

optimized CPU implementation

Moses

Marian

Fast machine translation on parallel and massively parallel hardware

Thesis or Dissertation

Doctoral

PhD Doctor of Philosophy