End-to-end Lip-reading: A Preliminary Study

Masters Thesis

Thapa, K. (2023). End-to-end Lip-reading: A Preliminary Study. Masters Thesis London South Bank University School of Engineering https://doi.org/10.18744/lsbu.92zq5
AuthorsThapa, K.
TypeMasters Thesis

Deep lip-reading is the combination of the domains of computer vision and natural language processing. It uses deep neural networks to extract speech from silent videos. Most works in lip-reading use a multi staged training approach due to the complex nature of the task. A single stage, end-to-end, unified training approach, which is an ideal of machine learning, is also the goal in lip-reading. However, pure end-to-end systems have not yet been able to perform as good as non-end-to-end systems. Some exceptions to this are the very recent Temporal Convolutional Network (TCN) based architectures. This work lays out preliminary study of deep lip-reading, with a special focus on various end-to-end approaches. The research aims to test whether a purely end-to-end approach is justifiable for a task as complex as deep lip-reading. To achieve this, the meaning of pure end-to-end is first defined and several lip-reading systems that follow the definition are analysed. The system that most closely matches the definition is then adapted for pure end-to-end experiments. Four main contributions have been made: i) An analysis of 9 different end-to-end deep lip-reading systems, ii) Creation and public release of a pipeline1 to adapt sentence level Lipreading Sentences in the Wild 3 (LRS3) dataset into word level, iii) Pure end-to-end training of a TCN based network and evaluation on LRS3 word-level dataset as a proof of concept, iv) a public online portal2 to analyse visemes and experiment live end-to-end lip-reading inference. The study is able to verify that pure end-to-end is a sensible approach and an achievable goal for deep machine lip-reading.

PublisherLondon South Bank University
Digital Object Identifier (DOI)https://doi.org/10.18744/lsbu.92zq5
File Access Level
Publication dates
Print04 Jan 2023
Publication process dates
Deposited10 Jan 2023
Permalink -


Download files

License: CC BY 4.0
File access level: Open

  • 347
    total views
  • 257
    total downloads
  • 21
    views this month
  • 19
    downloads this month

Export as