History Reuse and Bag-of-Words Loss for Long Summary Generation

文献类型： 外文期刊

第一作者： Liu, Qing

作者： Liu, Qing;Chen, Lei;Yuan, Yuan;Liu, Qing;Wu, Huarui

作者机构：

关键词： History; Decoding; Training; Predictive models; Postal services; Computational modeling; Vocabulary; Abstractive summarization; long summaries; history reuse; bag-of-words; word order deviation

期刊名称：IEEE-ACM TRANSACTIONS ON AUDIO SPEECH AND LANGUAGE PROCESSING （ 影响因子：3.919； 五年影响因子：3.906 ）

ISSN： 2329-9290

年卷期： 2021 年 29 卷

页码：

收录情况： SCI

摘要： Recurrent Neural Network (RNN) based abstractive text summarization models have made great progress over the past few years, largely triggered by the encoder-decoder architecture. However, there has been little work improving the generation of relatively long summaries. In this paper, we concentrate on two prominent problems in long summary generation. First, although significant efforts have been made to assist the encoder in handling long sequences, the decoder struggles with long sequences owing to the limited storage capacity of RNN. We propose a simple and effective approach called history reuse, which first mines critical information from the history summary sequence and then transmits the information to the decoder. Second, since encoder-decoder models are typically trained to produce exactly the same summary as the target summary, certain word order deviations between the predicted summary and target summary are excessively punished. Accordingly, we introduce a fully differentiable loss called bag-of-words (BoW) loss, which takes advantage of the feature of BoW discarding word order information in texts, and computes the difference between the two summaries at the BoW space. Experiments on two benchmark datasets, CNN/Daily Mail and Pubmed, demonstrate that our methods significantly improve the baseline.

分类号：

  • 相关文献
作者其他论文 更多>>

微信小程序