The Library
GiBERT : enhancing BERT with linguistic information using a lightweight gated injection method
Tools
Peinelt, Nicole, Rei, Marek and Liakata, Maria (2021) GiBERT : enhancing BERT with linguistic information using a lightweight gated injection method. In: 2021 Conference on Empirical Methods in Natural Language Processing, Punta Cana, Dominican Republic, 7-11 Nov 2021. Published in: Findings of the Association for Computational Linguistics: EMNLP 2021 pp. 2322-2336. doi:10.18653/v1/2021.findings-emnlp.200
|
PDF
WRAP-GiBERT-enhancing-BERT-with-linguistic-information-using-a-lightweight-gated-injection-method-Peinelt-22.pdf - Published Version - Requires a PDF viewer. Available under License Creative Commons Attribution 4.0. Download (461Kb) | Preview |
Official URL: http://dx.doi.org/10.18653/v1/2021.findings-emnlp....
Abstract
Large pre-trained language models such as BERT have been the driving force behind recent improvements across many NLP tasks. However, BERT is only trained to predict missing words – either through masking or next sentence prediction – and has no knowledge of lexical, syntactic or semantic information beyond what it picks up through unsupervised pre-training. We propose a novel method to explicitly inject linguistic information in the form of word embeddings into any layer of a pre-trained BERT. When injecting counter-fitted and dependency-based embeddings, the performance improvements on multiple semantic similarity datasets indicate that such information is beneficial and currently missing from the original model. Our qualitative analysis shows that counter-fitted embedding injection is particularly beneficial, with notable improvements on examples that require synonym resolution.
Item Type: | Conference Item (Paper) | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Subjects: | Q Science > QA Mathematics > QA76 Electronic computers. Computer science. Computer software | ||||||||||||
Divisions: | Faculty of Science, Engineering and Medicine > Science > Computer Science | ||||||||||||
Library of Congress Subject Headings (LCSH): | Natural language processing (Computer science), Computational linguistics, Machine learning, Neural networks (Computer science) | ||||||||||||
Journal or Publication Title: | Findings of the Association for Computational Linguistics: EMNLP 2021 | ||||||||||||
Publisher: | ACL | ||||||||||||
Book Title: | Findings of the Association for Computational Linguistics: EMNLP 2021 | ||||||||||||
Official Date: | 28 December 2021 | ||||||||||||
Dates: |
|
||||||||||||
Page Range: | pp. 2322-2336 | ||||||||||||
DOI: | 10.18653/v1/2021.findings-emnlp.200 | ||||||||||||
Status: | Peer Reviewed | ||||||||||||
Publication Status: | Published | ||||||||||||
Access rights to Published version: | Open Access (Creative Commons) | ||||||||||||
Copyright Holders: | Association for Computational Linguistics | ||||||||||||
Date of first compliant deposit: | 31 May 2022 | ||||||||||||
Date of first compliant Open Access: | 1 June 2022 | ||||||||||||
RIOXX Funder/Project Grant: |
|
||||||||||||
Conference Paper Type: | Paper | ||||||||||||
Title of Event: | 2021 Conference on Empirical Methods in Natural Language Processing | ||||||||||||
Type of Event: | Conference | ||||||||||||
Location of Event: | Punta Cana, Dominican Republic | ||||||||||||
Date(s) of Event: | 7-11 Nov 2021 | ||||||||||||
Related URLs: |
Request changes or add full text files to a record
Repository staff actions (login required)
View Item |
Downloads
Downloads per month over past year