MM DialogueGAT- A Fusion Graph Attention Network for Emotion Recognition using Multi-model System

Journal article


Fu, R., Gai, X., Al-Absi, A.A., Abdulhakim Al-Absi, M., Alam, M., Li, Y., Jiang, M. and Wang, X. (2024). MM DialogueGAT- A Fusion Graph Attention Network for Emotion Recognition using Multi-model System. IEEE Access. https://doi.org/10.1109/access.2024.3350156
AuthorsFu, R., Gai, X., Al-Absi, A.A., Abdulhakim Al-Absi, M., Alam, M., Li, Y., Jiang, M. and Wang, X.
Abstract

Emotion recognition is an important part of human-computer interaction and human communication information is multi-model. Despite advancements in emotion recognition models, certain challenges persist. The first problem pertains to the predominant focus in existing research on mining the interaction information between modes and the context information in the dialogue process but neglects to mine the role information between multi-model states and context information in the dialogue process. The second problem is in the context information of the dialogue where the information is not completely transmitted in a temporal structure. Aiming at these two problems, we propose a multi-model fusion dialogue graph attention network (MM DialogueGAT). To solve the problem 1, the bidirectional GRU mechanism is used to extract the information from each model. In the multi-model information fusion problem, different model configurations and different combinations use the cross-model multi-head attention mechanism to establish a multi-head attention layer. Text, video and audio information are used as the main and auxiliary modes for information fusion. To solve the problem 2, in the temporal context information extraction problem, the GAT graph structure is used to capture the context information in the mode. The results show that our model achieves good results using the IMEOCAP datasets.

Year2024
JournalIEEE Access
PublisherInstitute of Electrical and Electronics Engineers (IEEE)
ISSN2169-3536
Digital Object Identifier (DOI)https://doi.org/10.1109/access.2024.3350156
Web address (URL)https://doi.org/10.1109/ACCESS.2024.3350156
Publication dates
Online05 Jan 2024
Publication process dates
Deposited08 Feb 2024
Accepted author manuscript
License
File Access Level
Open
Permalink -

https://openresearch.lsbu.ac.uk/item/9650q

Download files

  • 24
    total views
  • 15
    total downloads
  • 1
    views this month
  • 1
    downloads this month

Export as

Related outputs

Multi-directional gated recurrent unit and convolutional neural network for load and energy forecasting: A novel hybridization
Abid, F., Alam, M., Alamri, F.S. and Siddique, I. (2023). Multi-directional gated recurrent unit and convolutional neural network for load and energy forecasting: A novel hybridization. AIMS Mathematics. 8 (9), pp. 19993-20017. https://doi.org/10.3934/math.20231019
A Double-Layer Blockchain Based Trust Management Model for Secure Internet of Vehicles
Ruan, W., Liu, J., Chen, Y., Islam, S.M. N. and Alam, M. (2023). A Double-Layer Blockchain Based Trust Management Model for Secure Internet of Vehicles. Sensors. 23 (10), p. 4699. https://doi.org/10.3390/s23104699
Aiden: Association-Learning-Based Attack Identification on the Edge of V2X Communication Networks
Alam, M., Chen, Y. and Mumtaz, S. (2022). Aiden: Association-Learning-Based Attack Identification on the Edge of V2X Communication Networks. IEEE Transactions on Green Communications and Networking. https://doi.org/10.1109/TGCN.2022.3188674
Variational Inference for a Recommendation System in IoT Networks Based on Stein’s Identity
Liu, J., Chen, Y., Islam, Sardar M. N. and Alam, M. (2022). Variational Inference for a Recommendation System in IoT Networks Based on Stein’s Identity. Applied Sciences. 12 (4), p. e1816. https://doi.org/10.3390/app12041816
Reliability analysis of the internet of things using Space Fault Network
Shasha L., Tiejun C. and Alam, M. (2020). Reliability analysis of the internet of things using Space Fault Network. Alexandria Engineering Journal . 60 (1), pp. 1259-1270. https://doi.org/10.1016/j.aej.2020.10.049
A Survey on Fault Tolerance Techniques for Wireless Vehicular Networks
Almeida, J., Rufino, J, Alam, M. and Ferreira, J. (2019). A Survey on Fault Tolerance Techniques for Wireless Vehicular Networks. Electronics. 8 (11), p. 1358. https://doi.org/10.3390/electronics8111358