README.md
5.2 KB · 176 lines · markdown Raw
1 ---
2 language: en
3 tags:
4 - summarization
5 model-index:
6 - name: google/pegasus-xsum
7 results:
8 - task:
9 type: summarization
10 name: Summarization
11 dataset:
12 name: samsum
13 type: samsum
14 config: samsum
15 split: train
16 metrics:
17 - name: ROUGE-1
18 type: rouge
19 value: 21.8096
20 verified: true
21 - name: ROUGE-2
22 type: rouge
23 value: 4.2525
24 verified: true
25 - name: ROUGE-L
26 type: rouge
27 value: 17.4469
28 verified: true
29 - name: ROUGE-LSUM
30 type: rouge
31 value: 18.8907
32 verified: true
33 - name: loss
34 type: loss
35 value: 3.0317161083221436
36 verified: true
37 - name: gen_len
38 type: gen_len
39 value: 20.3122
40 verified: true
41 - task:
42 type: summarization
43 name: Summarization
44 dataset:
45 name: xsum
46 type: xsum
47 config: default
48 split: test
49 metrics:
50 - name: ROUGE-1
51 type: rouge
52 value: 46.8623
53 verified: true
54 - name: ROUGE-2
55 type: rouge
56 value: 24.4533
57 verified: true
58 - name: ROUGE-L
59 type: rouge
60 value: 39.0548
61 verified: true
62 - name: ROUGE-LSUM
63 type: rouge
64 value: 39.0994
65 verified: true
66 - name: loss
67 type: loss
68 value: 1.5717021226882935
69 verified: true
70 - name: gen_len
71 type: gen_len
72 value: 22.8821
73 verified: true
74 - task:
75 type: summarization
76 name: Summarization
77 dataset:
78 name: cnn_dailymail
79 type: cnn_dailymail
80 config: 3.0.0
81 split: test
82 metrics:
83 - name: ROUGE-1
84 type: rouge
85 value: 22.2062
86 verified: true
87 - name: ROUGE-2
88 type: rouge
89 value: 7.6701
90 verified: true
91 - name: ROUGE-L
92 type: rouge
93 value: 15.4046
94 verified: true
95 - name: ROUGE-LSUM
96 type: rouge
97 value: 19.2182
98 verified: true
99 - name: loss
100 type: loss
101 value: 2.681241273880005
102 verified: true
103 - name: gen_len
104 type: gen_len
105 value: 25.0234
106 verified: true
107 ---
108
109 ### Pegasus Models
110 See Docs: [here](https://huggingface.co/transformers/master/model_doc/pegasus.html)
111
112 Original TF 1 code [here](https://github.com/google-research/pegasus)
113
114 Authors: Jingqing Zhang, Yao Zhao, Mohammad Saleh and Peter J. Liu on Dec 18, 2019
115
116 Maintained by: [@sshleifer](https://twitter.com/sam_shleifer)
117
118 Task: Summarization
119
120 The following is copied from the authors' README.
121
122 # Mixed & Stochastic Checkpoints
123
124 We train a pegasus model with sampled gap sentence ratios on both C4 and HugeNews, and stochastically sample important sentences. The updated the results are reported in this table.
125
126 | dataset | C4 | HugeNews | Mixed & Stochastic|
127 | ---- | ---- | ---- | ----|
128 | xsum | 45.20/22.06/36.99 | 47.21/24.56/39.25 | 47.60/24.83/39.64|
129 | cnn_dailymail | 43.90/21.20/40.76 | 44.17/21.47/41.11 | 44.16/21.56/41.30|
130 | newsroom | 45.07/33.39/41.28 | 45.15/33.51/41.33 | 45.98/34.20/42.18|
131 | multi_news | 46.74/17.95/24.26 | 47.52/18.72/24.91 | 47.65/18.75/24.95|
132 | gigaword | 38.75/19.96/36.14 | 39.12/19.86/36.24 | 39.65/20.47/36.76|
133 | wikihow | 43.07/19.70/34.79 | 41.35/18.51/33.42 | 46.39/22.12/38.41 *|
134 | reddit_tifu | 26.54/8.94/21.64 | 26.63/9.01/21.60 | 27.99/9.81/22.94|
135 | big_patent | 53.63/33.16/42.25 | 53.41/32.89/42.07 | 52.29/33.08/41.66 *|
136 | arxiv | 44.70/17.27/25.80 | 44.67/17.18/25.73 | 44.21/16.95/25.67|
137 | pubmed | 45.49/19.90/27.69 | 45.09/19.56/27.42 | 45.97/20.15/28.25|
138 | aeslc | 37.69/21.85/36.84 | 37.40/21.22/36.45 | 37.68/21.25/36.51|
139 | billsum | 57.20/39.56/45.80 | 57.31/40.19/45.82 | 59.67/41.58/47.59|
140
141 The "Mixed & Stochastic" model has the following changes:
142 - trained on both C4 and HugeNews (dataset mixture is weighted by their number of examples).
143 - trained for 1.5M instead of 500k (we observe slower convergence on pretraining perplexity).
144 - the model uniformly sample a gap sentence ratio between 15% and 45%.
145 - importance sentences are sampled using a 20% uniform noise to importance scores.
146 - the sentencepiece tokenizer is updated to be able to encode newline character.
147
148
149 (*) the numbers of wikihow and big_patent datasets are not comparable because of change in tokenization and data:
150 - wikihow dataset contains newline characters which is useful for paragraph segmentation, the C4 and HugeNews model's sentencepiece tokenizer doesn't encode newline and loose this information.
151 - we update the BigPatent dataset to preserve casing, some format cleanings are also changed, please refer to change in TFDS.
152
153
154 The "Mixed & Stochastic" model has the following changes (from pegasus-large in the paper):
155
156
157 trained on both C4 and HugeNews (dataset mixture is weighted by their number of examples).
158 trained for 1.5M instead of 500k (we observe slower convergence on pretraining perplexity).
159 the model uniformly sample a gap sentence ratio between 15% and 45%.
160 importance sentences are sampled using a 20% uniform noise to importance scores.
161 the sentencepiece tokenizer is updated to be able to encode newline character.
162
163
164 Citation
165 ```
166
167
168 @misc{zhang2019pegasus,
169 title={PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization},
170 author={Jingqing Zhang and Yao Zhao and Mohammad Saleh and Peter J. Liu},
171 year={2019},
172 eprint={1912.08777},
173 archivePrefix={arXiv},
174 primaryClass={cs.CL}
175 }
176 ```