Enhancing Transformer-based language models with Commonsense Representations for Knowledge-driven Machine Comprehension
Journal article
Li, R., Jiang, Z., Wang, L., Lu, X., Zhao, M. and Chen, D. (2021). Enhancing Transformer-based language models with Commonsense Representations for Knowledge-driven Machine Comprehension. Knowledge-Based Systems. 220, p. 106936. https://doi.org/10.1016/j.knosys.2021.106936
Authors | Li, R., Jiang, Z., Wang, L., Lu, X., Zhao, M. and Chen, D. |
---|---|
Abstract | Compared to the traditional machine reading comprehension (MRC) with limitation to the information in a passage, knowledge-driven MRC tasks aim to enable models to answer the question according to text and related commonsense knowledge. Although pre-trained Transformer-based language models (TrLMs) such as BERT and Roberta, have shown powerful perfor-mance in MRC, external knowledge such as unspoken commonsense and world knowledge still can not be used and explained explicitly. In this work, we present three simple yet effective injection methods integrated into the structure of TrLMs to fine-tune downstream knowledge-driven MRC tasks with off-the-shelf commonsense representations. Moreover, we introduce a mask mechanism for a token-level multi-hop relationship searching to fil-ter external knowledge. Experimental results indicate that the incremental TrLMs have significantly outperformed the baseline systems by 1%-4.1% on DREAM and CosmosQA, two prevalent knowledge-driven datasets. Further analysis shows the effectiveness of the proposed methods and the robustness of the incremental model in the case of an incomplete training set. |
Keywords | Machine Reading Comprehension; Transformer; Commonsense |
Year | 2021 |
Journal | Knowledge-Based Systems |
Journal citation | 220, p. 106936 |
Publisher | Elsevier |
ISSN | 0950-7051 |
Digital Object Identifier (DOI) | https://doi.org/10.1016/j.knosys.2021.106936 |
Publication dates | |
Online | 06 Mar 2021 |
Publication process dates | |
Accepted | 04 Mar 2021 |
Deposited | 18 Mar 2021 |
Accepted author manuscript | License File Access Level Open |
https://openresearch.lsbu.ac.uk/item/8w41z
Download files
151
total views160
total downloads0
views this month5
downloads this month