Lip Reading Sentences Using Deep Learning with Only Visual Cues
Journal article
Fenghour, S., Chen, D., Guo, K. and Xiao, P. (2020). Lip Reading Sentences Using Deep Learning with Only Visual Cues. IEEE Access. https://doi.org/10.1109/ACCESS.2020.3040906
| Authors | Fenghour, S., Chen, D., Guo, K. and Xiao, P. |
|---|---|
| Abstract | In this paper, a neural network-based lip reading system is proposed. The system is lexicon-free and uses purely visual cues. With only a limited number of visemes as classes to recognise, the system is designed to lip read sentences covering a wide range of vocabulary and to recognise words that may not be included in system training. The system has been testified on the challenging BBC Lip Reading Sentences 2 (LRS2) benchmark dataset. Experiments with videos of varying illumination have shown that the proposed model has a good robustness to varying levels of lighting. Compared with the state-of-the-art works in lip reading sentences, the system has achieved a significantly improved performance with 15% lower word error rate. The main contributions of this paper are: 1) The classification of visemes in continuous speech using a specially designed transformer with a unique topology; 2) The use of visemes as a classification schema for lip reading sentences; and 3) The conversion of visemes to words using perplexity analysis. All the contributions serve to enhance the accuracy of lip reading sentences. The paper also provides an essential survey of the research area. |
| Keywords | Deep learning; Lip reading; Neural networks; Perplexity analysis; Speech recognition |
| Year | 2020 |
| Journal | IEEE Access |
| Publisher | Institute of Electrical and Electronics Engineers (IEEE) |
| ISSN | 2169-3536 |
| Digital Object Identifier (DOI) | https://doi.org/10.1109/ACCESS.2020.3040906 |
| Publication dates | |
| 26 Nov 2020 | |
| Publication process dates | |
| Accepted | 19 Nov 2020 |
| Deposited | 21 Nov 2020 |
| Publisher's version | License File Access Level Open |
| Accepted author manuscript | License File Access Level Controlled |
| Additional information | Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. |
Permalink -
https://openresearch.lsbu.ac.uk/item/8v4xv
Download files
404
total views496
total downloads4
views this month9
downloads this month
Export as
Related outputs
Photothermal Radiometry Data Analysis by Using Machine Learning
Xiao, P. and Chen, D. (2024). Photothermal Radiometry Data Analysis by Using Machine Learning. Sensors. 24 (10), p. 3015. https://doi.org/10.3390/s24103015Novel Parameter-Free and Parametric Same Degree Distribution-based Dimensionality Reduction Algorithms for Trustworthy Data Structure Preserving
Hajderanj, L., Chen, D. and Dudley-Mcevoy, S. (2023). Novel Parameter-Free and Parametric Same Degree Distribution-based Dimensionality Reduction Algorithms for Trustworthy Data Structure Preserving. Information Sciences. 661, p. 120030. https://doi.org/10.1016/j.ins.2023.120030Skin Capacitive Image Stitching and Occlusion Measurements
Ciortea, L. I., Chen, D. and Xiao, P. (2023). Skin Capacitive Image Stitching and Occlusion Measurements. Cosmetics. 10 (1), p. 32. https://doi.org/10.3390/cosmetics10010032Understanding Cancer Patients with Diagnostically Influential Factors using High Dimensional Data Embedding
Syed, A. S., Hajderanj, L., Guo, K. and Chen, D. (2022). Understanding Cancer Patients with Diagnostically Influential Factors using High Dimensional Data Embedding. in: Imoize, A. L., Hemanth, D. J., Do, D.-T. and Sur, S. N. (ed.) Explainable Artificial Intelligence in Medical Decision Support Systems The Institution of Engineering and Technology (IET).UAV target tracking method based on deep reinforcement learning
Zhang, H., He, P., zhang, M., Chen, D., Neretin, E. and Li, B. (2022). UAV target tracking method based on deep reinforcement learning. 2022 International Conference on Cyber-physical Social Intelligence. Nanjing, China 21 - 24 Oct 2022Developing Phoneme-based Lip-reading Sentences System for Silent Speech Recognition
El Bialy, R., Chen, D., Fenghour, S., Hussein, W., Xiao, P., Karam, O. H. and Li, B. (2022). Developing Phoneme-based Lip-reading Sentences System for Silent Speech Recognition. CAAI Transactions on Intelligence Technology. 8 (1), pp. 128-139. https://doi.org/10.1049/cit2.12131An effective context-focused hierarchical mechanism for task-oriented dialogue response generation
Zhao, M., Wang, L., Jiang, Z., Li, R., Lu, X., Hu, Z. and Chen, D. (2022). An effective context-focused hierarchical mechanism for task-oriented dialogue response generation. Computational Intelligence. 38 (5), pp. 1831-1858. https://doi.org/10.1111/coin.12544The Effect of Sun Tan Lotion on Skin By Using Skin TEWL and Skin Water Content Measurements
Xiao, P. and Chen, D. (2022). The Effect of Sun Tan Lotion on Skin By Using Skin TEWL and Skin Water Content Measurements. MDPI Sensors. 22. https://doi.org/10.3390/s22093595Few-shot Object Recognition based on Three-Way Decision and Active Learning
Li, B., Luo, S., Wang, J., Tian, L. and Chen, D. (2022). Few-shot Object Recognition based on Three-Way Decision and Active Learning. Visual Computer . 37.An Effective Conversion of Visemes to Words for High-Performance Automatic Lipreading.
Fenghour, S., Chen, D., Guo, K., Li, B. and Xiao, P. (2021). An Effective Conversion of Visemes to Words for High-Performance Automatic Lipreading. Sensors. 21 (23). https://doi.org/s21237890UAV visual flight control method based on deep reinforcement learning
Bai, S., Li, B., Gan, Z. and Chen, D. (2021). UAV visual flight control method based on deep reinforcement learning. 2021 International Conference on Cyber-Physical Social Intelligence (ICCSI). https://doi.org/10.1109/iccsi53130.2021.9736242UAV flight control method based on deep reinforcement learning
Bai, S., Li, B., Gan, Z. and Chen, D. (2021). UAV flight control method based on deep reinforcement learning. 2021 International Conference on Cyber-Physical Social Intelligence (ICCSI). Beijing, China 18 Dec 2021 - 20 Mar 2022 Institute of Electrical and Electronics Engineers (IEEE). https://doi.org/10.1109/ICCSI53130.2021.9736242Skin Characterizations by Using Contact Capacitive Imaging and High-Resolution Ultrasound Imaging with Machine Learning Algorithms
Chirikhina, E., Chirikhin, A., Dewsbury-Ennis, S., Bianconi, F. and Xiao, P. (2021). Skin Characterizations by Using Contact Capacitive Imaging and High-Resolution Ultrasound Imaging with Machine Learning Algorithms. Applied Sciences. 11 (18), p. e8714. https://doi.org/10.3390/app11188714ME‐MADDPG: An efficient learning‐based motion planning method for multiple agents in complex environments
Wan, K., Wu, D., Li, B., Gao, X., Hu, Z. and Chen, D. (2021). ME‐MADDPG: An efficient learning‐based motion planning method for multiple agents in complex environments. International Journal of Intelligent Systems. 37 (3), pp. 2393-2427. https://doi.org/10.1002/int.22778Learning the structure of Bayesian networks with ancestral and/or heuristic partition
Tan, X., Gao, X., Wang, Z., Han, H., Liu, X. and Chen, D. (2021). Learning the structure of Bayesian networks with ancestral and/or heuristic partition. Information Sciences. https://doi.org/10.1016/j.ins.2021.10.052Deep Learning-based Automated Lip-Reading: A Survey
Fenghour, S., Chen, D., Guo, K., Li, B. and Xiao, P. (2021). Deep Learning-based Automated Lip-Reading: A Survey. IEEE Access. https://doi.org/10.1109/ACCESS.2021.3107946Deep Learning Causal Attributions of Breast Cancer
Chen, D., Hajderanj, L., Mallet, S., Camenen, P., Li, B., Hao, R. and Zhao, E. (2021). Deep Learning Causal Attributions of Breast Cancer. in: Arai, K. (ed.) Intelligent Computing, Proceedings of the 2021 Computing Conference, Lecture Notes in Networks and Systems, Vol 285, Intelligent Computing (A. Kohei, Editor) Springer.The Impact of Supervised Manifold Learning on Structure Preserving and Classification Error: A Theoretical Study
Hajderanj, L., Chen, D. and Weheliye, I. (2021). The Impact of Supervised Manifold Learning on Structure Preserving and Classification Error: A Theoretical Study. IEEE Access. 9. https://doi.org/10.1109/ACCESS.2021.3066259Enhancing Transformer-based language models with Commonsense Representations for Knowledge-driven Machine Comprehension
Li, R., Jiang, Z., Wang, L., Lu, X., Zhao, M. and Chen, D. (2021). Enhancing Transformer-based language models with Commonsense Representations for Knowledge-driven Machine Comprehension. Knowledge-Based Systems. 220, p. 106936. https://doi.org/10.1016/j.knosys.2021.106936The Development of a Skin Image Analysis Tool by Using Machine Learning Algorithms
Xiao, P., Zhang, Xu, Pan, Wei, Ou, Xiang, Bontozoglou, C., Chirikhina, E. and Chen, D. (2020). The Development of a Skin Image Analysis Tool by Using Machine Learning Algorithms. Cosmetics. 7 (3), p. e67. https://doi.org/10.3390/cosmetics7030067Three‐way decision of target threat decision making based on adaptive threshold algorithms
Li, B., Tian, Li., Han, Y. and Chen, D. (2020). Three‐way decision of target threat decision making based on adaptive threshold algorithms. The Journal of Engineering. 2020 (13), pp. 293-297. https://doi.org/10.1049/joe.2019.1202Effectiveness analysis of ship formation air defence based on deep belief network
Li, B., Luo, H., Wang, Y. and Chen, D. (2020). Effectiveness analysis of ship formation air defence based on deep belief network. The Journal of Engineering. 2020 (13), pp. 394-398. https://doi.org/10.1049/joe.2019.1201Maneuvering target tracking of UAV based on MN-DDPG and transfer learning
Li, B., Yang, Z.P., Chen, D.Q., Liang, S.Y. and Ma, H. (2020). Maneuvering target tracking of UAV based on MN-DDPG and transfer learning. Defence Technology. https://doi.org/10.1016/j.dt.2020.11.014Deep Learning Causal Attributions of Breast Cancer
Chen, D, Hajderanj, L, Mallet, S, Camenen, P, Li, B, Ren, H and Zhao, E (2020). Deep Learning Causal Attributions of Breast Cancer. Computing 2021. London 15 - 16 Jul 2021 The Science and Information (SAI) Organization. https://doi.org/10.1007/978-3-030-80129-8_10UAV Maneuvering Target Tracking in Uncertain Environments based on Deep Reinforcement Learning and Meta-learning
Li, B., Gan, Z., Chen, D. and Aleksandrovich, D.S. (2020). UAV Maneuvering Target Tracking in Uncertain Environments based on Deep Reinforcement Learning and Meta-learning. Remote Sensing. 12 (22), p. 3789. https://doi.org/10.3390/rs12223789Single- and Multi-Distribution Dimensionality Reduction Approaches for a Better Data Structure Capturing
Hajderanj, L., Chen, D., Grisan, E. and Dudley-McEvoy, S (2020). Single- and Multi-Distribution Dimensionality Reduction Approaches for a Better Data Structure Capturing. IEEE Access. 8, pp. 207141 - 207155. https://doi.org/10.1109/ACCESS.2020.3038460Learning Bayesian Networks based on Order Graph with Ancestral Constraints
Wang, Z., Gao, X., Tian, X., Yang, Y. and Chen, D. (2020). Learning Bayesian Networks based on Order Graph with Ancestral Constraints. Knowledge-Based Systems. https://doi.org/10.1016/j.knosys.2020.106515In Vivo Assessment of Water Content, Trans-Epidermial Water Loss and Thickness in Human Facial Skin
Chirikhina, E., Chirikhin, A., Xiao, P., Dewsbury-Ennis, S. and Bianconi, F. (2020). In Vivo Assessment of Water Content, Trans-Epidermial Water Loss and Thickness in Human Facial Skin. Applied Sciences. 10 (17), p. e6139. https://doi.org/10.3390/app10176139An Adaptive Task Scheduling Method for Networked UAV Combat Cloud System Based on Virtual Machine and Task Migration
Li, B., Liang, S., Tian, L., Chen, D. and Zhang, M. (2020). An Adaptive Task Scheduling Method for Networked UAV Combat Cloud System Based on Virtual Machine and Task Migration. Mathematical Problems in Engineering. p. 5391479. https://doi.org/10.1155/2020/5391479An adaptive dwell time scheduling model for phased array radar based on three-way decision
Li, B., Tian, L., Chen, D. and Liang, S. (2020). An adaptive dwell time scheduling model for phased array radar based on three-way decision. Journal of Systems Engineering and Electronics. pp. 500-509. https://doi.org/10.23919/JSEE.2020.000030In Vivo Skin Characterizations by Using Opto-Thermal Depth-Resolved Detection Spectra
Xu Zhang, Bontozoglou, C. and Xiao, P. (2019). In Vivo Skin Characterizations by Using Opto-Thermal Depth-Resolved Detection Spectra. Cosmetics. 6 (3), p. 54. https://doi.org/10.3390/cosmetics6030054Advances in Radiometry Research
Xiao, P. (2019). Advances in Radiometry Research. Nova Science Publishers, Inc..Practical Java Programming for IoT, AI, and Blockchain
Xiao, P/ (2019). Practical Java Programming for IoT, AI, and Blockchain. Wiley.Intelligent Aircraft Maneuvering Decision Based on CNN
Li, B., Liang, S., Tian, L. and Chen, D. (2019). Intelligent Aircraft Maneuvering Decision Based on CNN. Proceedings of the 3rd International Conference on Computer Science and Application Engineering. (138). https://doi.org/10.1145/3331453.3362046Intelligent Attitude Control of Aircraft Based on LSTM
Li, B, Gao, P, Li, X and Chen, D (2019). Intelligent Attitude Control of Aircraft Based on LSTM. 3rd International Conference on Artificial Intelligence Applications and Technologies. Beijing, China 01 - 03 Aug 2019 IOP Publishing. https://doi.org/10.1088/1757-899X/646/1/012013Applications of Capacitive Imaging in Human Skin Texture and Hair Analysis
Bontozoglou, C. and Xiao, P. (2019). Applications of Capacitive Imaging in Human Skin Texture and Hair Analysis . Applied Sciences. 10 (1), p. 256. https://doi.org//10.3390/app10010256A Task Scheduling Algorithm for Phased Array Radar Based on Dynamic Three-way Decision
Li, B., Tian, L., Chen, D. and Han, Y. (2019). A Task Scheduling Algorithm for Phased Array Radar Based on Dynamic Three-way Decision. Sensors. 20 (1). https://doi.org/10.3390/s20010153Intelligent Flight Control of Combat Aircraft Based on Autoencoder
Li, B., Gao, P., Liang, S. and Chen, D. (2019). Intelligent Flight Control of Combat Aircraft Based on Autoencoder. 2019 The 4th International Conference on Robotics, Control and Automation. GuangZhou 26 - 28 Jul 2019 https://doi.org/10.1145/3351180.3351210Skin Capacitive Imaging Analysis Using Deep Learning GoogLeNet
Zhang, X., Pan, W., Bontozoglou, C., Chirikhina, E., Chen, D. and Xiao, P. (2019). Skin Capacitive Imaging Analysis Using Deep Learning GoogLeNet. Computing Conference 2020. London, UK 16 - 17 Jul 2019 Springer.FRS: A Simple Knowledge Graph Embedding Model for Entity Prediction
Wang, L.F., Lu, X., Jiang, Z., Zhang, Z., Li, R., Zhao, M. and Chen, D. (2019). FRS: A Simple Knowledge Graph Embedding Model for Entity Prediction. Mathematical Biosciences and Engineering. 16 (6), pp. 7789-7807. https://doi.org/10.3934/mbe.2019391Predicting Customer Profitability Dynamically over Time: An Experimental Comparative Study
Chen, D., Guo, K. and Li, B. (2019). Predicting Customer Profitability Dynamically over Time: An Experimental Comparative Study. 24th Iberoamerican Congress on Pattern Recognition (CIARP 2019). Havana, Cuba 28 - 31 Oct 2019 https://doi.org/10.1007/978-3-030-33904-3_16In Vivo Human Hair Hydration Measurements by Using Opto-Thermal Radiometry
Bontozoglou, C, Zhang, X, Patel, A, Lane, ME and Xiao, P (2019). In Vivo Human Hair Hydration Measurements by Using Opto-Thermal Radiometry. International Journal of Thermophysics. 40 (2). https://doi.org/10.1007/s10765-018-2477-xLearning Bayesian Networks using the Constrained Maximum a Posteriori Probability Method
Yang, Y, Gao, X, Guo, Z and Chen, D (2019). Learning Bayesian Networks using the Constrained Maximum a Posteriori Probability Method. Pattern Recognition. 91, pp. 123-134. https://doi.org/10.1016/j.patcog.2019.02.006Micro-relief analysis with skin capacitive imaging
Bontozoglou, C, Zhang, X and Xiao, P (2019). Micro-relief analysis with skin capacitive imaging. Skin Research and Technology. 25 (2), pp. 165-170. https://doi.org/10.1111/srt.12628Epsilon Interactive Virtual User Manual (VUM)
Al Hashimi, O. and Xiao, P (2019). Epsilon Interactive Virtual User Manual (VUM). 2018 International Conference on Computing, Electronics and Communications Engineering, ICCECE 2018. Southend-on-Sea 16 - 17 Aug 2018 pp. 138-143 https://doi.org/10.1109/iCCECOME.2018.8658872Learning Bayesian network parameters via minimax algorithm
Gao, X, Gao, G, Ren, H, Chen, D and He, C (2019). Learning Bayesian network parameters via minimax algorithm. International Journal of Approximate Reasoning. 108, pp. 62-75. https://doi.org/10.1016/j.ijar.2019.03.001Improving Prediction Accuracy of Breast Cancer Survivability and Diabetes Diagnosis via RBF Networks trained with EKF models
Adegoke, V, Chen, D and Banissi, E (2019). Improving Prediction Accuracy of Breast Cancer Survivability and Diabetes Diagnosis via RBF Networks trained with EKF models. International Journal of Computer Information Systems and Industrial Management. 11, pp. 82-100.A New Supervised t-SNE with Dissimilarity Measure for Effective Data Visualization and Classification
Hajderanj, L, Weheliye, I and Chen, D (2019). A New Supervised t-SNE with Dissimilarity Measure for Effective Data Visualization and Classification. 2019 8th International Conference on Software and Information Engineering. Cairo 09 - 12 Apr 2019Recurrent Neural Networks for Decoding Lip Read Speech
Fenghour, S, Chen, D and Xiao, P (2019). Recurrent Neural Networks for Decoding Lip Read Speech. 2019 8th International Conference on Software and Information Engineering (ICSIE 2019). Cairo 09 - 12 Apr 2019Decoder-Encoder LSTM for Lip Reading
Fenghour, S., Chen, D. and Xiao, P. (2019). Decoder-Encoder LSTM for Lip Reading. Proceedings of the 2019 8th International Conference on Software and Information Engineering. https://doi.org/10.1145/3328833.3328845Enhancing Ensemble Prediction Accuracy of Breast Cancer Survivability and Diabetes Diagnostic using optimized EKF-RBFN trained prototypes, The 10th International Conference on Soft Computing and Pattern Recognition
Adegoke, V, Chen, D, Banissi, E and Barikzai, S (2019). Enhancing Ensemble Prediction Accuracy of Breast Cancer Survivability and Diabetes Diagnostic using optimized EKF-RBFN trained prototypes, The 10th International Conference on Soft Computing and Pattern Recognition. The 10th International Conference on Soft Computing and Pattern Recognition. Porto, Portugal 13 - 15 Dec 2018Distributed deep networks based on Bagging-Down SGD algorithm
Qin, C, Gao, X and Chen, D (2019). Distributed deep networks based on Bagging-Down SGD algorithm. Xi Tong Gong Cheng Yu Dian Zi Ji Shu/Systems Engineering and Electronics. 41 (5), pp. 1021-1027. https://doi.org/10.3969/j.issn.1001-506X.2019.05.13Towards automated cost analysis, benchmarking and estimating in construction: a machine learning approach
Chen, D, Hajderanj, L and Fiske, J (2019). Towards automated cost analysis, benchmarking and estimating in construction: a machine learning approach. 13th Multi Conference on Computer Science and Information Systems (MCCSIS). Porto, Portugal 16 - 18 Jul 2019Design of a voice control 6DoF grasping robotic arm based on ultrasonic sensor, computer vision and Alexa voice assistance
Wang, Z, Chen, D and Xiao, P (2019). Design of a voice control 6DoF grasping robotic arm based on ultrasonic sensor, computer vision and Alexa voice assistance. International Conference on Information Technology in Medicine and Education. Qingdao, China 23 - 25 Aug 2019 Institute of Electrical and Electronics Engineers (IEEE). https://doi.org/10.1109/ITME.2019.00150Capacitive Imaging for Skin Characterizations and Solvent Penetration Measurements
Zhang, X., Bontozoglou, C., Chirikhina, E., Lane, M. and Xiao, P. (2018). Capacitive Imaging for Skin Characterizations and Solvent Penetration Measurements . MDPI Cosmetics . 5 (3), p. 52. https://doi.org/10.3390/cosmetics5030052Developing a web based interactive 3D virtual environment for novel skin measurement instruments
Xiao, P and Al Hashimi, O. (2018). Developing a web based interactive 3D virtual environment for novel skin measurement instruments. 2018 Advances in Science and Engineering Technology International Conferences (ASET). Dubai, Sharjah, Abu Dhabi, United Arab Emirates 06 - 07 Feb 2018 London South Bank University. https://doi.org/10.1109/ICASET.2018.8376823In-Vivo Skin Capacitive Image Classification Using AlexNet Convolution Neural Network
Zhang, X, Pan, W and Xiao, P (2018). In-Vivo Skin Capacitive Image Classification Using AlexNet Convolution Neural Network. 2018 3rd International Conference on Image, Vision and Computing (ICIVC 2018). Chongqing, China 27 - 29 Jun 2018 Institute of Electrical and Electronics Engineers (IEEE). pp. 439-443 https://doi.org/10.1109/ICIVC.2018.8492860
Educational Network Bandwidth Analysis and Prediction
Oumar, O A, Dyllon, S and Xiao, P (2018). Educational Network Bandwidth Analysis and Prediction. the 14th International Conference on Machine Learning and Data Mining (MLDM'2018) July 14-19, 2018. New York, USA 14 - 19 Jul 2018Visual analytics in the public sector: An analysis on diversities and similarities of London’s wards
Chen, D, Sanz, BM and Zhao, E (2018). Visual analytics in the public sector: An analysis on diversities and similarities of London’s wards. International Conference on Big Data Analytics, Data Mining and Computational Intelligence 2018 (BigDaCI 2018). Madrid, Spain 18 - 20 Jul 2018 Bigdaci.Educational Bandwidth Traffic Prediction using Non-Linear Autoregressive Neural Networks
Oumar, O A, Dyllon, S, Xiao, P and Hong, T (2018). Educational Bandwidth Traffic Prediction using Non-Linear Autoregressive Neural Networks. The 21st International Conference on Climbing and Walking Robots and the Support Technologies for Mobile Machines - CLAWAR 2018. Panama 10 - 12 Sep 2018Contour Mapping for Speaker-Independent Lip Reading System
Fenghour, S, Chen, D and Xiao, P (2018). Contour Mapping for Speaker-Independent Lip Reading System. The 11th International Conference on Machine Vision (ICMV 2018). Munich, Germany 01 - 03 Nov 2018