Entity Recognition Method for Judicial Documents Based on BERT Model

doi:10.12068/j.issn.1005-3026.2020.10.003

Abstract

Abstract: Using manual analysis of case files， it is easy to cause the problem of case entity omission and low efficiency of feature extraction. Therefore， the bidirectional encoder representation from transformers pre-training model based on the traditional long short-term memory networks and conditional random fields was used to fine tune the model parameters on the manually labeled corpus for entity recognition. And then the semantic coding output from the previous layer was decoded by the long short-term memory networks and conditional random fields to complete entity extraction. The pre-training model has the advantages of huge parameters， powerful feature extraction ability and multi-dimensional semantic representation of entities， which can effectively improve the effect of entity extraction. The experimental results showed that the proposed model can achieve more than 89% entity extraction accuracy， which is significantly better than the traditional recurrent neural network and convolutional neural network model.

Key words: deep learning, pre-training model, bidirectional long short-term memory, conditional random field, named entity recognition

CLC Number:

TP391

CHEN Jian， HE Tao， WEN Ying-you， MA Lin-tao. Entity Recognition Method for Judicial Documents Based on BERT Model[J]. Journal of Northeastern University Natural Science, 2020, 41(10): 1382-1387.

References

[1]Sundar N G，Sunny A T.An efficient information extraction model for personal named entity［J］.International Journal of Computer Trends & Technology，2013，4(3):119-128.
[2]Li D，Huang L，Ji H，et al.Biomedical event extraction based on knowledge-driven tree-LSTM［C］//Proceedings of NAACL- HLT 2019.Minneapolis，2019:1421-1430.
[3]甘丽新，万常选，刘德喜，等.基于句法语义特征的中文实体关系抽［J］.计算机研究与发展，2016，53(2):284-302.(Gan Li-xin，Wan Chang-xuan，Liu De-xi，et al.Extraction of Chinese entity relations based on syntactic and semantic features［J］.Journal of Computer Research and Development，2016，53(2):284-302.)
[4]Nikoulina V，Sandor A，Dymetman M.Hybrid adaptation of named entity recognition systems for statistical machine translation purposes［J］.Journal of Radiation Research，2011，53(2):1-16.
[5]王鑫，邹磊，王朝坤，等.知识图谱数据管理研究综述［J］.软件学报，2019，30(7):2139-2174.(Wang Xin，Zou Lei，Wang Chao-kun，et al.A survey of knowledge map data management［J］.Journal of Software，2019，30(7):2139-2174.)
[6]俞鸿魁，张华平，刘群，等.基于层叠隐马尔可夫模型的中文命名实体识别［J］.通信学报，2006，27(2):87-94.(Yu Hong-kui，Zhang Hua-ping，Liu Qun，et al.Chinese named entity recognition based on cascading hidden Markov model［J］.Journal on Communications，2006，27(2):87-94.)
[7]McCallum A，Freitag D，Pereira F C N.Maximum entropy Markov models for information extraction and segmentation［C］//Proceedings of the Seventeenth International Conference on Machine Learning.Sydney，2000:591-598.
[8]周俊生，戴新宇，尹存燕，等.基于层叠条件随机场模型的中文机构名自动识别［J］.电子学报，2006，34(5) :804-809.(Zhou Jun-sheng，Dai Xin-yu，Yin Cun-yan，et al.Automatic recognition of Chinese organization name based on cascading conditional random field model［J］.Acta Electronica Sinica，2006，34(5):804-809.)
[9]Ertekin S，Bottou L.Nonconvex online support vector machines［J］.Transactions on Pattern Analysis and Machine Intelligence，2011，33(2):368-381.
[10]Feng Y H，Hong Y U，Sun G，et al.Named entity recognition method based on BLSTM［J］.Computer Science，2018，45(2):261-268.
[11]Dong C，Zhang J，Zong C，et al.Character-based LSTM-CRF with radical-level features for Chinese named entity recognition［C］//Natural Language Processing and Chinese Computing 2016.Kunming，2016:239-250.
[12]Ma X，Hovy E.End-to-end sequence labeling via bi-directional LSTM-CNNs-CRF［C］//Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics.Berlin，2016:1064-1074.
[13]Strubell E，Verga P，Belanger D，et al.Fast and accurate entity recognition with iterated dilated convolutions［C］//Conference on Empirical Methods in Natural Language Processing.Copenhagen，2017:2670-2680.
[14]Vaswani A，Shazeer N，Parmar N，et al.Attention is all you need［C］//Proceedings of the 31st International Conference on Neural Information Processing Systems.New York，2017:6000-6010.
[15]Zhang Y，Yang J.Chinese NER using lattice LSTM［C］//Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics.Stroudsburg，2018:1554-1564.
[16]Yang Y S，Zhang M S，Chen W L，et al.Adversarial learning for Chinese NER from crowd annotations［C］//Proceedings of the 32th AAAI Conference on Artificial Intelligence.New Orleans，2018:3216-3222.
[17]Devlin J，Chang M W，Lee K，et al.BERT:pre-training of deep bidirectional transformers for language understanding［C］//Proceedings of the 2019 Conference of the Association for Computational Linguistics.Stroudsburg，2019:4171-4186.
[18]Matthew E P，Mark N，Mohit I，et al.Deep contextualized word representations［C］//Proceedings of the 2019 Annual Conference of the North American Chapter of the Association for Computational Linguistics.New Orleans，2018:3253-3268.
[19]Bengio Y，Lamblin P，Popovici D，et al.Greedy layer-wise training of deep network［J］.Advances in Neural Information Processing System，2007，6(19):153-162.
[20]Li J Q，Zhao S H，Yang J J.WCP-RNN:a novel RNN-based approach for Bio-NER in Chinese EMRs［J］.Journal of Supercomputing，2018，1(16):1-18.
[15]关守平，房少纯.一种新型的区间-粒子群优化算法［J］.东北大学学报(自然科学版)，2012，33(10):1381-1384.(Guan Shou-ping，Fang Shao-chun.A new interval particle swarm optimization algorithm［J］.Journal of Northeastern University(Natural Science)，2012，33(10):1381-1384.)