The Library
Toward automated factchecking : developing an annotation schema and benchmark for consistent automated claim detection
Tools
Konstantinovskiy, Lev, Price, Oliver, Babakar, Mevan and Zubiaga, Arkaitz (2021) Toward automated factchecking : developing an annotation schema and benchmark for consistent automated claim detection. Digital Threats: Research and Practice, 2 (2). pp. 1-16. doi:10.1145/3412869 ISSN 2692-1626.
Research output not available from this repository.
Request-a-Copy directly from author or use local Library Get it For Me service.
Official URL: http://dx.doi.org/10.1145/3412869
Abstract
In an effort to assist factcheckers in the process of factchecking, we tackle the claim detection task, one of the necessary stages prior to determining the veracity of a claim. It consists of identifying the set of sentences, out of a long text, deemed capable of being factchecked. This article is a collaborative work between Full Fact, an independent factchecking charity, and academic partners. Leveraging the expertise of professional factcheckers, we develop an annotation schema and a benchmark for automated claim detection that is more consistent across time, topics, and annotators than are previous approaches. Our annotation schema has been used to crowdsource the annotation of a dataset with sentences from UK political TV shows. We introduce an approach based on universal sentence representations to perform the classification, achieving an F1 score of 0.83, with over 5% relative improvement over the state-of-the-art methods ClaimBuster and ClaimRank. The system was deployed in production and received positive user feedback.
Item Type: | Journal Article | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|
Divisions: | Faculty of Science, Engineering and Medicine > Science > Computer Science | ||||||||||
Journal or Publication Title: | Digital Threats: Research and Practice | ||||||||||
Publisher: | ACM | ||||||||||
ISSN: | 2692-1626 | ||||||||||
Official Date: | June 2021 | ||||||||||
Dates: |
|
||||||||||
Volume: | 2 | ||||||||||
Number: | 2 | ||||||||||
Page Range: | pp. 1-16 | ||||||||||
DOI: | 10.1145/3412869 | ||||||||||
Status: | Peer Reviewed | ||||||||||
Publication Status: | Published | ||||||||||
Access rights to Published version: | Restricted or Subscription Access | ||||||||||
Copyright Holders: | ACM | ||||||||||
RIOXX Funder/Project Grant: |
|
Request changes or add full text files to a record
Repository staff actions (login required)
View Item |