Understanding spatial language in radiology: Representation framework, annotation, and spatial relation extraction from chest X-ray reports using deep learning

Birchall, 2015, Spatial ability in radiologists: a necessary prerequisite?, Br. J. Radiol., 88, 6, 10.1259/bjr.20140511 Corry, 2011, The future of recruitment and selection in radiology. Is there a role for assessment of basic visuospatial skills?, Clin. Radiol., 66, 481, 10.1016/j.crad.2010.12.003 P. Kordjamshidi, M.V. Otterlo, M.-F. Moens, Spatial Role Labeling: Task Definition and Annotation Scheme, in: Proceedings of the Language Resources & Evaluation Conference, 2010, pp. 413–420. Kordjamshidi, 2017, Spatial Language Understanding with Multimodal Graphs using Declarative Learning based Programming, 33 Kordjamshidi, 2015, Structured learning for spatial information extraction from biomedical text: Bacteria biotopes, BMC Bioinform., 16, 1, 10.1186/s12859-015-0542-z Roberts, 2015, Automatic Extraction and Post-coordination of Spatial Relations in Consumer Language, 1083 Demner-Fushman, 2016, Preparing a collection of radiology examinations for distribution and retrieval, J. Am. Med. Inform. Assoc., 23, 304, 10.1093/jamia/ocv080 N. Ramrakhiyani, G. Palshikar, V. Varma, A Simple Neural Approach to Spatial Role Labelling, in: Advances in Information Retrieval, 2019, pp. 102–108. doi:10.1007/978-3-030-15719-7_13. P. Shi, J. Lin, Simple BERT Models for Relation Extraction and Semantic Role LabelingarXiv:1904.05255. Si, 2019, Enhancing clinical concept extraction with contextual embeddings, J. Am. Med. Inform. Assoc., 1 Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, R. Soricut, ALBERT: A Lite BERT for Self-supervised Learning of Language Representations (2019) 1–16arXiv:1909.11942. Lee, 2019, BioBERT: a pre-trained biomedical language representation model for biomedical text mining, Bioinformatics, 1 K. Huang, J. Altosaar, R. Ranganath, ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission (2019) 1–19arXiv:1904.05342. Kordjamshidi, 2015, Global machine learning for spatial ontology population, J. Web Semant., 30, 3, 10.1016/j.websem.2014.06.001 Roberts, 2012, A machine learning approach for identifying anatomical locations of actionable findings in radiology reports, 779 Rink, 2013, Extracting actionable findings of appendicitis from radiology reports using natural language processing, 221 C. Zhang, X. Zhang, W. Jiang, Q. Shen, S. Zhang, Rule-based extraction of spatial relations in natural language text, in: 2009 International Conference on Computational Intelligence and Software Engineering, no. 40971231, IEEE, 2009, pp. 1–4. doi:10.1109/CISE.2009.5363900. J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, in: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Association for Computational Linguistics, Minneapolis, Minnesota, 2019, pp. 4171–4186. doi:10.18653/v1/N19-1423. K. Clark, U. Khandelwal, O. Levy, C.D. Manning, What Does BERT Look at? An Analysis of BERT’s Attention, in: Proceedings of the Second BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, Association for Computational Linguistics, 2019, pp. 276–286. doi:10.18653/v1/W19-4828. Jawahar, 2019, What Does BERT Learn about the Structure of Language?, 3651 Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. Salakhutdinov, Q.V. Le, XLNet: Generalized Autoregressive Pretraining for Language UnderstandingarXiv:1906.08237. Hayward, 1995, Spatial language and spatial representation, Cognition, 55, 39, 10.1016/0010-0277(94)00643-Y Mani, 2010, SpatialML: annotation scheme, resources, and evaluation, Lang. Resources Eval., 44, 263, 10.1007/s10579-010-9121-0 J. Pustejovsky, J.L. Moszkowicz, Integrating Motion Predicate Classes with Spatial and Temporal Annotations, in: Coling 2008: Companion Volume: Posters, Coling 2008 Organizing Committee, 2008, pp. 95–98. URL <https://www.aclweb.org/anthology/C08-2024>. J. Fasola, M.J. Mataric, Using semantic fields to model dynamic spatial relations in a robot architecture for natural language instruction of service robots, in: 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2013, pp. 143–150. https://doi.org/10.1109/IROS.2013.6696345. A. Chang, M. Savva, C.D. Manning, Learning Spatial Knowledge for Text to 3D Scene Generation, in: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Association for Computational Linguistics, 2014, pp. 2028–2038. https://doi.org/10.3115/v1/D14-1217. E. Kergosien, H. Alatrista-Salas, M. Gaio, F.N. Güttler, M. Roche, M. Teisseire, When textual information becomes spatial information compatible with satellite images, in: 2015 7th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management (IC3K), Vol. 01, 2015, pp. 301–306. Collell, 2018, Learning representations specialized in spatial knowledge: leveraging language and vision, Trans. Assoc. Comput. Linguist., 6, 133, 10.1162/tacl_a_00010 M. Ulinski, B. Coyne, J. Hirschberg, SpatialNet: A Declarative Resource for Spatial Relations, in: Proceedings of the Combined Workshop on Spatial Language Understanding (SpLU) and Grounded Communication for Robotics (RoboNLP), Association for Computational Linguistics, 2019, pp. 61–70. https://doi.org/10.18653/v1/W19-1607. Wang, 2017, ChestX-ray8: Hospital-scale chest X-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases, 3462 J. Irvin, P. Rajpurkar, M. Ko, Y. Yu, S. Ciurea-Ilcus, C. Chute, H. Marklund, B. Haghgoo, R. Ball, K. Shpanskaya, J. Seekins, D.A. Mong, S.S. Halabi, J.K. Sandberg, R. Jones, D.B. Larson, C.P. Langlotz, B.N. Patel, M.P. Lungren, A.Y. Ng, CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert ComparisonarXiv:1901.07031. Wang, 2019, Enhanced Diagnosis of Pneumothorax with an Improved Real-time Augmentation for Imbalanced Chest X-rays Data Based on DCNN, IEEE/ACM Trans. Comput. Biol. Bioinf., 14, 1 Pesce, 2019, Learning to detect chest radiographs containing lung nodules using visual attention networks, Med. Image Anal., 53, 26, 10.1016/j.media.2018.12.007 Hassanpour, 2016, Information extraction from multi-institutional radiology reports, Artif. Intell. Med., 66, 29, 10.1016/j.artmed.2015.09.007 S. Cornegruta, R. Bakewell, S. Withey, G. Montana, Modelling Radiological Language with Bidirectional Long Short-Term Memory NetworksarXiv:1609.08409. A. Bustos, A. Pertusa, J.-M. Salinas, M. de la Iglesia-Vayá, PadChest: A large chest x-ray image dataset with multi-label annotated reportsarXiv:1901.07441. Hassanpour, 2017, Characterization of change and significance for clinical findings in radiology reports through natural language processing, J. Digit. Imaging, 30, 314, 10.1007/s10278-016-9931-8 Annarumma, 2019, Automated triaging of adult chest radiographs with deep artificial neural networks, Radiology, 291, 196, 10.1148/radiol.2018180921 C. Friedman, P.O. Alderson, J.H. Austin, J.J. Cimino, S.B. Johnson, A general natural-language text processor for clinical radiology, Journal of the American Medical Informatics Association: JAMIA 1 (2) (1994 Mar-Apr) 161–174. doi:10.1136/jamia.1994.95236146. Friedman, 1995, Architectural requirements for a multipurpose natural language processor in the clinical environment, 347 Friedman, 2004, Automated encoding of clinical documents based on natural language processing, J. Am. Med. Inform. Assoc.: JAMIA, 11, 392, 10.1197/jamia.M1552 Sevenster, 2012, Automatically correlating clinical findings and body locations in radiology reports using MedLEE, J. Digit. Imaging, 25, 240, 10.1007/s10278-011-9411-0 Yim, 2016, Tumor information extraction in radiology reports for hepatocellular carcinoma patients, 455 Steinkamp, 2019, Toward complete structured information extraction from radiology reports using machine learning, J. Digit. Imaging, 32, 554, 10.1007/s10278-019-00234-y Alex, 2019, Text mining brain imaging reports, J. Biomed. Semant., 10, 23, 10.1186/s13326-019-0211-7 Wang, 2018, TieNet: Text-Image Embedding Network for Common Thorax Disease Classification and Reporting in Chest X-Rays, 9049 Peng, 2018, NegBio: a high-performance tool for negation and uncertainty detection in radiology reports, 188 Z.A. Daniels, D.N. Metaxas, Exploiting Visual and Report-Based Information for Chest X-Ray Analysis by Jointly Learning Visual Classifiers and Topic Models, in: IEEE 16th International Symposium on Biomedical Imaging (ISBI), 2019. Zech, 2018, Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: A cross-sectional study, PLoS Med., 15, 1, 10.1371/journal.pmed.1002683 Shin, 2016, Learning to Read Chest X-Rays: Recurrent Neural Cascade Model for Automated Image Annotation, 2497 X. Huang, Y. Fang, M. Lu, Y. Yao, M. Li, An Annotation Model on End-to-End Chest Radiology Reports, in: IEEE Access, Vol. 7, IEEE, 2019. doi:10.1109/access.2019.2917922. Leaman, 2015, Challenges in clinical natural language processing for automated disorder normalization, J. Biomed. Inform., 57, 28, 10.1016/j.jbi.2015.07.010 Aronson, 2010, An overview of MetaMap: historical perspective and recent advances, J. Am. Med. Inform. Assoc., 17, 229, 10.1136/jamia.2009.002733 S. Candemir, S. Rajaraman, G. Thoma, S. Antani, Deep learning for grading cardiomegaly severity in chest x-rays: An investigation, in: IEEE Life Sciences Conference (LSC), IEEE, 2018, pp. 109–113. doi:10.1109/LSC.2018.8572113. G. Lample, M. Ballesteros, S. Subramanian, K. Kawakami, C. Dyer, Neural Architectures for Named Entity Recognition, in: Proceedings of NAACL-HLT, 2016, pp. 260–270. arXiv:1603.01360. M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G.S. Corrado, A. Davis, J. Dean, M. Devin, et al., Tensorflow: Large-scale machine learning on heterogeneous distributed systems, arXiv preprint arXiv:1603.04467. Li, 2017, A neural joint model for entity and relation extraction from biomedical text, BMC Bioinform., 18, 198, 10.1186/s12859-017-1609-9 M. Miwa, M. Bansal, End-to-End Relation Extraction using LSTMs on Sequences and Tree Structures, in: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, 2016, pp. 1105–1116. arXiv:arXiv:1601.00770v3, doi:10.18653/v1/P16-1105. Langlotz, 2006, RadLex: a new method for indexing online educational materials, Radiographics, 26, 1595, 10.1148/rg.266065168 R.K. Srivastava, K. Greff, J. Schmidhuber, Highway NetWorks: Training Very Deep Networks, in: NIPS’15 Proceedings of the 28th International Conference on Neural Information Processing Systems, 2015, pp. 2377–2385. arXiv:1507.06228.