Publication Details
Claim-Dissector: An Interpretable Fact-Checking System with Joint Re-ranking and Veracity Prediction
Motlíček Petr, doc. Ing., Ph.D. (DCGM)
Smrž Pavel, doc. RNDr., Ph.D. (DCGM)
fact-checking, open-domain fact-checking, claim, claim-dissector, dissector,
fine-grained retrieval, coarse-grained supervision, interpretability,
interpretable retrieval, evidence-grounded prediction, verification, fact
verification, veracity assessement
We present Claim-Dissector: a novel latent variable model for fact-checking and
analysis, which given a claim and a set of retrieved evidence jointly learns to
identify: (i) the relevant evidences to the given claim (ii) the veracity of the
claim. We propose to disentangle the per-evidence relevance probability and its
contribution to the final veracity probability in an interpretable way - the
final veracity probability is proportional to a linear ensemble of per-evidence
relevance probabilities. In this way, the individual contributions of evidences
towards the final predicted probability can be identified. In per-evidence
relevance probability, our model can further distinguish whether each relevant
evidence is supporting (S) or refuting (R) the claim. This allows to quantify how
much the S/R probability contributes to final verdict or to detect disagreeing
evidence. Despite its interpretable nature, our system achieves results
competetive with state-of-the-art on the FEVER dataset, as compared to typical
two-stage system pipelines, while using significantly fewer parameters.
Furthermore, our analysis shows that our model can learn fine-grained relevance
cues while using coarse-grained supervision and we demonstrate it in 2 ways. (i)
We show that our model can achieve competitive sentence recall while using only
paragraph-level relevance supervision. (ii) Traversing towards the finest
granularity of relevance, we show that our model is capable of identifying
relevance at the token level. To do this, we present a new benchmark TLR-FEVER
focusing on token-level interpretability - humans annotate tokens in relevant
evidences they considered essential when making their judgment. Then we measure
how similar are these annotations to the tokens our model is focusing on.
@inproceedings{BUT185594,
author="Martin {Fajčík} and Petr {Motlíček} and Pavel {Smrž}",
title="Claim-Dissector: An Interpretable Fact-Checking System with Joint Re-ranking and Veracity Prediction",
booktitle="Findings of the Association for Computational Linguistics: ACL 2023",
year="2023",
series="ACL",
volume="2023",
pages="10184--10205",
publisher="Association for Computational Linguistics",
address="Toronto",
doi="10.18653/v1/2023.findings-acl.647",
isbn="978-1-959429-62-3",
url="https://aclanthology.org/2023.findings-acl.647/"
}