Chinese named entity recognition for agricultural diseases based on entity-related visual prompts injection

文献类型： 外文期刊

第一作者： Zhang, Chenshuo

作者： Zhang, Chenshuo;Wang, Chunshan;Liang, Fangfang;Zhang, Chenshuo;Wu, Huarui;Wang, Chunshan;Chen, Cheng;Zhu, Huaji;Wu, Huarui;Chen, Cheng;Zhu, Huaji;Liang, Fangfang;Zhang, Lijie;Wang, Chunshan

作者机构：

关键词： Agricultural diseases; Visual prompts; Multimodal named entity recognition; Pre-trained language model

期刊名称：COMPUTERS AND ELECTRONICS IN AGRICULTURE （ 影响因子：8.9； 五年影响因子：9.3 ）

ISSN： 0168-1699

年卷期： 2024 年 227 卷

页码：

收录情况： SCI

摘要： Named entity recognition is a crucial step in information extraction for agricultural diseases. However, most existing works only utilize word embedding models to generate contextual semantic features, which are limited by textual contextual dependencies and suffer from the problem of missing contextual semantic features. There is a semantic correlation between the visual information in agricultural disease images and the target entities in agricultural disease texts, but this visual information has not been fully utilized among the existing works. To solve the aforementioned issues, we propose a Chinese named entity recognition model for agricultural diseases based on entity-related visual prompts injection. First, we introduce an improved object detector with Shape-IoU to obtain local images related to the target entities. Second, we propose a visual feature extraction structure based on multi-scale feature fusion to extract visual features from the local images and encode them as visual prompts. Third, we propose a novel visual-guided attention mechanism to achieve multimodal information fusion within the pretrained language model. Furthermore, the first multimodal named entity recognition (MNER) dataset for agricultural diseases named Disease7000 was also collected and annotated, which contains 7,000 image-text pairs, 4 entity types, and 31,656 samples. Experimental results show that our proposed model achieved the best F1 scores of 89.17% and 75.06% on Disease7000 and the publicly available Twitter2015 dataset, respectively, demonstrating the effectiveness and generalizability of the model. Additionally, the ablation study validate the effectiveness of each module in many aspects.

分类号：

  • 相关文献
作者其他论文 更多>>

微信小程序