|
--- |
|
language: |
|
- en |
|
thumbnail: https://cogcomp.seas.upenn.edu/images/logo.png |
|
tags: |
|
- text-classification |
|
- bart |
|
- xsum |
|
license: cc-by-sa-4.0 |
|
datasets: |
|
- xsum |
|
widget: |
|
- text: "<s> Ban Ki-moon was elected for a second term in 2007. </s></s> Ban Ki-Moon was re-elected for a second term by the UN General Assembly, unopposed and unanimously, on 21 June 2011." |
|
- text: "<s> Ban Ki-moon was elected for a second term in 2011. </s></s> Ban Ki-Moon was re-elected for a second term by the UN General Assembly, unopposed and unanimously, on 21 June 2011." |
|
|
|
--- |
|
|
|
# bart-faithful-summary-detector |
|
|
|
## Model description |
|
|
|
A BART (base) model trained to classify whether a summary is *faithful* to the original article. See our [paper in NAACL'21](https://www.seas.upenn.edu/~sihaoc/static/pdf/CZSR21.pdf) for details. |
|
|
|
## Usage |
|
Concatenate a summary and a source document as input (note that the summary needs to be the **first** sentence). |
|
|
|
Here's an example usage (with PyTorch) |
|
```python |
|
from transformers import AutoTokenizer, AutoModelForSequenceClassification |
|
|
|
tokenizer = AutoTokenizer.from_pretrained("CogComp/bart-faithful-summary-detector") |
|
model = AutoModelForSequenceClassification.from_pretrained("CogComp/bart-faithful-summary-detector") |
|
|
|
article = "Ban Ki-Moon was re-elected for a second term by the UN General Assembly, unopposed and unanimously, on 21 June 2011." |
|
|
|
bad_summary = "Ban Ki-moon was elected for a second term in 2007." |
|
good_summary = "Ban Ki-moon was elected for a second term in 2011." |
|
|
|
bad_pair = tokenizer(text=bad_summary, text_pair=article, return_tensors='pt') |
|
good_pair = tokenizer(text=good_summary, text_pair=article, return_tensors='pt') |
|
|
|
bad_score = model(**bad_pair) |
|
good_score = model(**good_pair) |
|
|
|
print(good_score[0][:, 1] > bad_score[0][:, 1]) # True, label mapping: "0" -> "Hallucinated" "1" -> "Faithful" |
|
``` |
|
|
|
|
|
|
|
|
|
### BibTeX entry and citation info |
|
|
|
```bibtex |
|
@inproceedings{CZSR21, |
|
author = {Sihao Chen and Fan Zhang and Kazoo Sone and Dan Roth}, |
|
title = {{Improving Faithfulness in Abstractive Summarization with Contrast Candidate Generation and Selection}}, |
|
booktitle = {NAACL}, |
|
year = {2021} |
|
} |
|
``` |
