End-to-end Lip-reading: A Preliminary Study

Masters Thesis


Thapa, K. (2023). End-to-end Lip-reading: A Preliminary Study. Masters Thesis London South Bank University School of Engineering https://doi.org/10.18744/lsbu.92zq5
AuthorsThapa, K.
TypeMasters Thesis
Abstract

Deep lip-reading is the combination of the domains of computer vision and natural language processing. It uses deep neural networks to extract speech from silent videos. Most works in lip-reading use a multi staged training approach due to the complex nature of the task. A single stage, end-to-end, unified training approach, which is an ideal of machine learning, is also the goal in lip-reading. However, pure end-to-end systems have not yet been able to perform as good as non-end-to-end systems. Some exceptions to this are the very recent Temporal Convolutional Network (TCN) based architectures. This work lays out preliminary study of deep lip-reading, with a special focus on various end-to-end approaches. The research aims to test whether a purely end-to-end approach is justifiable for a task as complex as deep lip-reading. To achieve this, the meaning of pure end-to-end is first defined and several lip-reading systems that follow the definition are analysed. The system that most closely matches the definition is then adapted for pure end-to-end experiments. Four main contributions have been made: i) An analysis of 9 different end-to-end deep lip-reading systems, ii) Creation and public release of a pipeline1 to adapt sentence level Lipreading Sentences in the Wild 3 (LRS3) dataset into word level, iii) Pure end-to-end training of a TCN based network and evaluation on LRS3 word-level dataset as a proof of concept, iv) a public online portal2 to analyse visemes and experiment live end-to-end lip-reading inference. The study is able to verify that pure end-to-end is a sensible approach and an achievable goal for deep machine lip-reading.

Year2023
PublisherLondon South Bank University
Digital Object Identifier (DOI)https://doi.org/10.18744/lsbu.92zq5
File
License
File Access Level
Open
Publication dates
Print04 Jan 2023
Publication process dates
Deposited10 Jan 2023
Permalink -

https://openresearch.lsbu.ac.uk/item/92zq5

Download files


File
MRes_Thesis_Revised.pdf
License: CC BY 4.0
File access level: Open

  • 12
    total views
  • 7
    total downloads
  • 12
    views this month
  • 7
    downloads this month

Export as