Extended reality for procedural planning and guidance in structural heart disease – a review of the state-of-the-art

Steinberg DH, Staubach S, Franke J, Sievert H (2010) Defining structural heart disease in the adult patient: current scope, inherent challenges and future directions. Eur Heart J Supplements 12(supplE):E2–E9 https://doi.org/10.1093/eurheartj/suq012

Milgram P, Kishono FA (1994) Taxonomy of mixed reality visual displays. IEICE Trans Inform Systems; E77–D(12):1321–1329. https://doi.org/10.1.1.102.4646

Brigham TJ (2017) Reality check: basics of augmented, virtual, and mixed reality. Med Ref Serv Q 36(2):171–178. https://doi.org/10.1080/02763869.2017.1293987

Bando T, Iijima A, Yano S (2012) Visual fatigue caused by stereoscopic images and the search for the requirement to prevent them: a review. Displays 33(2):76–83. https://doi.org/10.1016/J.DISPLA.2011.09.001

Li K (2019) “Understanding Waveguides: the Key Technology for Augmented Reality Near-eye Display (Part I).” Virtual Reality Pop [Available from: https://virtualrealitypop.com/understanding-waveguide-the-key-technology-for-augmented-reality-near-eye-display-part-i-2b16b61f4bae

Gelman SA, Rotschild C, Bruckheimer E, Kaufman A, Dickman D (2020) METHOD AND SYSTEM FOR DISPLAYING HOLOGRAPHIC IMAGES WITHIN A REAL OBJECT.US Patent App.17/015,126.

Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD et al (2021) The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. PLoS Med 18(3):e1003583. https://doi.org/10.1371/journal.pmed.1003583

Gehrsitz P, Rompel O, Schober M, Cesnjevar R, Purbojo A, Uder M et al (2021) Cinematic Rendering in Mixed-Reality Holograms: A New 3D Preoperative Planning Tool in Pediatric Heart Surgery. Frontiers in cardiovascular medicine. Front Cardiovasc Med;8:633611. https://doi.org/10.3389/fcvm.2021.633611

Ye W, Zhang X, Li T, Luo C, Yang L (2021) Mixed-reality hologram for diagnosis and surgical planning of double outlet of the right ventricle: a pilot study. Clin Radiol;76(3):237.e1-.e7. https://doi.org/10.1016/j.crad.2020.10.017

Lu JC, Ensing GJ, Ohye RG, Romano JC, Sassalos P, Owens ST et al (2020) Stereoscopic Three-Dimensional Visualization for Congenital Heart Surgery Planning: Surgeons’ Perspectives. J Am Soc Echocardiogr;33(6):775–7. https://doi.org/10.1016/j.echo.2020.02.003

Pushparajah K, Chu KYK, Deng S, Wheeler G, Gomez A, Kabir S et al (2021) Virtual reality three-dimensional echocardiographic imaging for planning surgical atrioventricular valve repair. JTCVS Tech;7:269-77. https://doi.org/10.1016/j.xjtc.2021.02.044

Milano EG, Kostolny M, Pajaziti E, Marek J, Regan W, Caputo M et al (2021) Enhanced 3D visualization for planning biventricular repair of double outlet right ventricle: a pilot study on the advantages of virtual reality. Eur Heart J - Digit Health 2(4):667–675. https://doi.org/10.1093/ehjdh/ztab087

Vettukattil J, Baliulis G, Hillman N, Haw M (2020) Interactive visualization enables biventricular repair in highly complex congenital heart disease (CHD). Chest;157(6):A19. https://doi.org/10.1016/j.chest.2020.05.023

Pather N, Birbara N, Sobhanmanesh S, Otton J (2018) Virtual reality for surgical training and preoperative planning. FASEB J 32(S1):lb534–lb. https://doi.org/10.1096/fasebj.2018.32.1_supplement.lb534

Cen J, Liufu R, Wen S, Qiu H, Liu X, Chen X et al (2021) Three-Dimensional Printing, virtual reality and mixed reality for Pulmonary Atresia: Early Surgical Outcomes evaluation. Heart lung and circulation. Heart Lung Circ;30(2):296–302. https://doi.org/10.1016/j.hlc.2020.03.017

Tedoriya T, Okano R, Miyauchi T, Fukuzumi M, Gatakte Y (2019) Virtual reality image evaluation for aortic valve leaflet reconstruction. Structural Heart;3:80. https://doi.org/10.1080/24748706.2019.1589818

Sadeghi AH, Bakhuis W, Van Schaagen F, Oei FBS, Bekkers JA, Maat APWM et al (2020) Immersive 3D virtual reality imaging in planning minimally invasive and complex adult cardiac surgery. Eur Heart J - Digit Health 1(1):62–70. https://doi.org/10.1093/ehjdh/ztaa011

Chan F, Aguirre S, Martin E, Ma M, Hanley F (2021) Intra-operative augmented reality guidance for complex pulmonary artery repair, a step toward precision surgery. J Am Coll Cardiol;77:3239. https://doi.org/10.1016/S0735-1097(21)04593-9

Milano EG, Pajaziti E, Sauvage E, Cook A, Schievano S, Mortensen KH et al (2019) Taking surgery out of reality. Circ Cardiovasc Imaging 12(7):e009297. https://doi.org/10.1161/CIRCIMAGING.119.009297

Ong CS, Krishnan A, Huang CY, Spevak P, Vricella L, Hibino N et al (2018) Role of virtual reality in congenital heart disease. Congenit Heart Dis 13(3):357–361. https://doi.org/10.1111/chd.12587

Kumar RP, Pelanis E, Bugge R, Brun H, Palomar R, Aghayan DL, et al. (2020). Use of mixed reality for surgery planning: Assessment and development workflow. J Biomed Inform;112:100077. https://doi.org/10.1016/j.yjbinx.2020.100077

Ayerbe VMC, Morales MLV, Rojas CJL, Cortés MLA (2020) Visualization of 3D models through virtual reality in the planning of congenital cardiothoracic anomalies correction: an initial experience. World J Pediatr Congenital Heart Surg 11(5):627–629. https://doi.org/10.1177/2150135120923618

Perens G, Yoshida T, Finn JP (2021) 4-Dimensional virtual reality model of tetralogy of Fallot. Prog Pediatr Cardiol;64:101453. https://doi.org/10.1016/j.ppedcard.2021.101453

Qiu H, Huang M, Cen J, Chen J, Liu X, Li X et al (2021) VR and 3D Printing for Pre-op Planning of Left Ventricular Myxoma in a child. Ann Thorac Surg;113(6):e457-e60. https://doi.org/10.1016/j.athoracsur.2021.07.094

Nam HH, Herz C, Lasso A, Drouin S, Posada A, Morray B et al (2020) Simulation of Transcatheter Atrial and Ventricular Septal Defect Device Closure Within Three-Dimensional Echocardiography-Derived Heart Models on Screen and in Virtual Reality. J Am Soc Echocardiogr;33(5):641-4.e2. https://doi.org/10.1016/j.echo.2020.01.011

Tedoriya T, Miyauchi T, Okano R, Fukuzumi M, Tamura Y, Gatate Y (2020) Novel Technique for Aortic Valve Reconstruction with Three Same-Sized Autologous Pericardial Leaflets - Useful Application of 3D Hologram Evaluation in Order to Enhance Reproducibility. Structural Heart;4:28. https://doi.org/10.1080/24748706.2020.1715149

Dutcher J, Schmidt W, Dahl P, Humbert J (2020) Echopixel 3d ct vs. Transesophageal echocardiography for pre-procedural planning of watchman implantation. J Am Coll Cardiol;75:1206. https://doi.org/10.1016/S0735-1097%2820%2931833-7

Sinha K, Vowels TJ, Kothapalli PR, Barker CM, Atkins MD, von Wyler MC et al (2019) A novel 3-Dimensional imaging platform for device sizing in transcatheter aortic valve implantation. J Am Coll Cardiol Intv;12:S51. https://doi.org/10.1016/j.jcin.2019.01.177

Sadeghi AH, Ooms JF, Bakhuis W, Taverne YJHJ, Van Mieghem NM, Bogers AJJC (2021) Immersive Virtual Reality Heart Models for Planning of Transcatheter Paravalvular Leak Closure: A Feasibility Study. JACC Cardiovasc Interv;14(16):1854-6. https://doi.org/10.1016/j.jcin.2021.06.018

Tandon A, Burkhardt BEU, Batsis M, Zellers TM, Velasco Forte MN, Valverde I, Cardiovascular et al (2019) Sinus Venosus Defects: Anatomic Variants and Transcatheter Closure Feasibility Using Virtual Reality Planning. JACC Cardiovasc Imaging;12(5):921-4. https://doi.org/10.1016/j.jcmg.2018.10.013

Butera G, Sturla F, Pluchinotta FR, Caimi A, Carminati M Holographic Augmented Reality and 3D Printing for Advanced Planning of Sinus Venosus ASD/Partial Anomalous Pulmonary Venous Return Percutaneous Management. JACC Cardiovasc Interv;12(14):1389-91. https://doi.org/10.1016/j.jcin.2019.03.020

Aregullin EO, Mohammad Nijres B, Al-Khatib Y, Vettukattil J (2021) Transcatheter Fontan completion using novel balloon and stent system. Catheter Cardiovasc Interv 97(4):679–684. https://doi.org/10.1002/ccd.29463

Pasqualin G, Sturla F, D’Aiello AF, Chessa M (2021) Mixed reality navigation of a systemic venous baffle obstruction: unravelling the percutaneous approach in atrial switch operation. Eur Heart J 42(41):4284. https://doi.org/10.1093/eurheartj/ehaa961

Ghosh RM, Mascio CE, Rome JJ, Jolley MA, Whitehead KK (2021) Use of virtual reality for Hybrid Closure of multiple ventricular septal defects. JACC: Case Reports 3(14):1579–1583. https://doi.org/10.1016/j.jaccas.2021.07.033

Pappalardo O, Pasquali M, Rossini G, Maltagliati A, Italiano G, Fusini L et al (2021) A platform for real-3d visualization and planning of left atrial appendage occlusion through mixed reality. Eur Heart J Cardiovasc Imaging;22:S1. https://doi.org/10.1093/ehjci/jeaa356.247

Vigil C, Cianciulli A, Nam HH, Abid A, Herz C, Lasso A et al (2021) Modeling Tool for Rapid virtual planning of the Intracardiac Baffle in double-outlet right ventricle. Ann Thorac Surg;111(6):2078-83. https://doi.org/10.1016/j.athoracsur.2021.02.058

James RC, Monsky WL, Jorgensen NW, Seslar SP, James RC, Monsky WL et al (2020) Virtual-reality guided Versus Fluoroscopy-Guided Transseptal puncture in a Cardiac Phantom. J Invasive Cardiol;32(2):76-81

Currie ME, McLeod J, Patel R, Kiaii B, Peters TM. (2016). Augmented reality system for ultrasound guidance of transcatheter aortic valve implantation. Innovations (Phila);11(1):31-9. https://doi.org/10.1097/IMI.0000000000000235

Guiraudon GM, Jones DL, Bainbridge D, Linte C, Pace D, Moore J et al (2010) Augmented reality image guidance during off-pump mitral valve replacement through the guiraudon universal cardiac introducer. Innovations (Phila);5(6):430-8. https://doi.org/10.1097/IMI.0b013e31820278ef

Linte CA, Moore J, Wedlake C, Bainbridge D, Guiraudon GM, Jones DL et al (2009) Inside the beating heart: An in vivo feasibility study on fusing pre- and intra-operative imaging for minimally invasive therapy. Int J Comput Assist Radiol Surg;4(2):113-23. https://doi.org/10.1007/s11548-008-0278-6

Vasilyev NV, Novotny PM, Martinez JF, Loyola H, Salgo IS, Howe RD et al (2008) Stereoscopic vision display technology in real-time three-dimensional echocardiography-guided intracardiac beating-heart surgery. J Thorac Cardiovasc Surg;135(6):1334-41. https://doi.org/10.1016/j.jtcvs.2007.12.045

Moore JT, Chu MWA, Kiaii B, Bainbridge D, Guiraudon G, Wedlake C et al (2013) A navigation platform for guidance of beating heart transapical mitral valve repair. IEEE Trans Biomed Eng;60(4):1034-40. https://doi.org/10.1109/TBME.2012.2222405

Sadri S, Loeb G, Grinshpoon A, Elvezio C, Velagapudi P, Ng VG et al (2018) Augmented reality guidance for cerebral embolic protection (CEP) with the sentinel device during transcatheter aortic valve replacement (TAVR): first-in-human study. Circulation;138:Abstract 1209. https://doi.org/10.1161/circ.138.suppl_1.12019

Kasprzak JD, Witowski J, Pawlowski J, Peruga JZ, Złahoda-Huzior A (2019) Percutaneous patent ductus arteriosus closure using intraprocedural mixed reality visualization of 3D computed tomography angiography data: first-in-man experience. Eur Heart J - Cardiovasc Imaging 20(7):839. https://doi.org/10.1093/ehjci/jez008

Rymuza B, Grodecki K, Kamiński J, Scisło P, Huczek Z (2017) Holographic imaging during transcatheter aortic valve implantation procedure in bicuspid aortic valve stenosis. Kardiologia Polska (Polish Heart Journal) 75(10):1056. https://doi.org/10.5603/KP.2017.0195

Witowski J, Darocha S, Kownacki Ł, Pietrasik A, Pietura R, Banaszkiewicz M et al (2019) Augmented reality and three-dimensional printing in percutaneous interventions on pulmonary arteries. Quant Imaging Med Surg;9(1):3?29. https://doi.org/10.21037/qims.2018.09.08

Kasprzak JD, Pawlowski J, Peruga JZ, Kaminski J, Lipiec P (2020) First-in-man experience with real-time holographic mixed reality display of three-dimensional echocardiography during structural intervention: balloon mitral commissurotomy. Eur Heart J 41(6):801. https://doi.org/10.1093/eurheartj/ehz127

Zbronski K, Rymuza B, Scislo P, Kochman J, Huczek Z (2018) Augmented reality in left atrial appendage occlusion. Kardiol Pol;76:212. https://doi.org/10.5603/KP.2018.0017

van den Bosch AE, Koning AHJ, Meijboom FJ, McGhie JS, Simoons ML, van der Spek PJ et al (2005) Dynamic 3D echocardiography in virtual reality. Cardiovasc Ultrasound;3:37. https://doi.org/10.1186/1476-7120-3-37

Bol Raap G, Kappetein AP, Bogers AJJC, Koning AHJ, Van Der Spek PJ, Scohy TV et al (2007) Virtual reality 3D echocardiography in the assessment of tricuspid valve function after surgical closure of ventricular septal defect. Cardiovasc Ultrasound;5:8. https://doi.org/10.1186/1476-7120-5-8

Beitnes JO, Klaeboe LG, Karlsen JS, Urheim S (2015) Mitral valve analysis using a novel 3D holographic display: a feasibility study of 3D ultrasound data converted to a holographic screen. Int J Cardiovasc Imaging;31(2):323-8. https://doi.org/10.1007/s10554-014-0564-z

Ballocca Flavia MLM, Ladha Karim HJ, Qua, Horlick Eric M, Meineri Massimiliano (2019) Validation of quantitative 3-Dimensional Transesophageal Echocardiography Mitral Valve Analysis using Stereoscopic Display. J Cardiothorac Vasc Anesth;33(3):732-41. https://doi.org/10.1053/j.jvca.2018.08.013

Wheeler G, Deng S, Pushparajah K, Schnabel JA, Simpson JM, Gomez A (2019) Virtual linear measurement system for accurate quantification of medical images. Healthc Technol Lett 6(6):220–225. https://doi.org/10.1049/htl.2019.0074

Narang A, Hitschrich N, Mor-Avi V, Schreckenberg M, Schummers G, Tiemann K et al (2020) Virtual reality analysis of three-dimensional echocardiographic and Cardiac Computed Tomographic Data Sets. J Am Soc Echocardiogr 33(11):1306–1315. https://doi.org/10.1016/j.echo.2020.06.018

Aly AH, Gorman RC, Stauffer M, Patel PA, Gorman JH, Pouch AM (2019) Virtual reality for visualization and assessment of mitral valve geometry in structural heart disease. Circulation;140:Abstract 16182. https://doi.org/10.1161/circ.140.suppl_1.16182

Xue H, Sun K, Chen G, Hong W, Wu L, Yu J et al (2010) Three-dimensional echocardiographic virtual endoscopy for the diagnosis of congenital heart disease in children. Int J Cardiovasc Imaging;26(8):851-9. DOI: https://doi.org/10.1007/s10554-010-9649-5

Bruckheimer E, Rotschild C, Dagan T, Amir G, Kaufman A, Gelman S et al (2016) Computer-generated real-time digital holography: first time use in clinical medical imaging. European heart journal cardiovascular Imaging. Eur Heart J Cardiovasc Imaging;17(8):845-9. https://doi.org/10.1093/ehjci/jew087

Brun H, Bugge RAB, Suther LKR, Birkeland S, Kumar R, Pelanis E et al (2019) Mixed reality holograms for heart surgery planning: first user experience in congenital heart disease. Eur Heart J Cardiovasc Imaging;20(8):883-8. https://doi.org/10.1093/ehjci/jey184

Chan F, Aguirre S, Bauser-Heaton H, Hanley F, Perry SB. (2013).Head Tracked Stereoscopic Pre-surgical Evaluation of Major Aortopulmonary Collateral Arteries in the Newborns. Radiological Society of North America 2013 Scientific Assembly and Annual Meeting. Available from: https://archive.rsna.org/2013/13024673.html

Kamiya K, Matsubayashi Y, Mori Y, Wakisaka H, Lee J, Minamidate N et al (2021) A virtual-reality imaging analysis of the dynamic aortic root anatomy. Ann Thorac Surg;112(6):2077-83. https://doi.org/10.1016/j.athoracsur.2021.06.038

Raimondi F, Vida V, Godard C, Bertelli F, Reffo E, Boddaert N et al (2021) Fast-track virtual reality for cardiac imaging in congenital heart disease. J Card Surg;36(7):2598-602. https://doi.org/10.1111/jocs.15508

Kim B, Loke Y-H, Mass P, Irwin MR, Capeland C, Olivieri L et al (2020) A novel virtual reality Medical Image Display System for Group discussions of congenital heart disease: development and usability testing. JMIR Cardio;4(1):e20633. https://doi.org/10.2196/20633

Lau I, Gupta A, Sun Z (2021) Clinical value of virtual reality versus 3D Printing in congenital heart disease. Biomolecules;11(6):884. https://doi.org/10.3390/biom11060884

Insights FB (2022) Virtual Reality Market Size, Share and Covid-19 Impact Analysis, by component, by device type, by industry, and regional forecast, 2022–2029. Contract No.: FBI101378

Deng S, Wheeler G, Toussaint N, Munroe L, Bhattacharya S, Sajith G et al (2021) A virtual reality system for improved image-based planning of Complex Cardiac Procedures. J Imaging;7(8). https://doi.org/10.3390/jimaging7080151

Ayoub A, Pulijala Y (2019) The application of virtual reality and augmented reality in oral & maxillofacial surgery. BMC Oral Health 19(1):238. https://doi.org/10.1186/s12903-019-0937-8

Casari FA, Navab N, Hruby LA, Kriechling P, Nakamura R, Tori R et al (2021) Augmented reality in orthopedic surgery is emerging from Proof of Concept towards Clinical Studies: a Literature Review explaining the technology and current state of the art. Curr Rev Musculoskelet Med 14(2):192–203. https://doi.org/10.1007/s12178-021-09699-3

Mishra R, Narayanan MDK, Umana GE, Montemurro N, Chaurasia B, Deora H (2022) Virtual reality in neurosurgery: beyond Neurosurgical Planning. Int J Environ Res Public Health 19(3):1719. https://doi.org/10.3390/ijerph19031719

Roberts S, Desai A, Checcucci E, Puliatti S, Taratkin M, Kowalewski KF et al (2022) “Augmented reality” applications in urology: a systematic review. Minerva Urol Nephrol 74(5):528–537. https://doi.org/10.23736/s2724-6051.22.04726-7

Beams R, Brown E, Cheng W-C, Joyner JS, Kim AS, Kontson K et al (2022) Evaluation challenges for the application of extended reality Devices in Medicine. J Digit Imaging;35:1409-18. https://doi.org/10.1007/s10278-022-00622-x

IEC (2019) IEC 63145-20-20:2019. Eyewear display - Part 20–20: Fundamental measurement methods - Image quality IEC

BS EN ISO 9241 – 303:2011. Ergonomics of human-system interaction. Requirements for electronic visual displays. BSI (British Standards Institute)

Wheeler G, Deng S, Toussaint N, Pushparajah K, Schnabel JA, Simpson JM et al (2018) Virtual interaction and visualisation of 3D medical imaging data with VTK and Unity. Healthc Technol Lett;5(5):148-53. https://doi.org/10.1049/htl.2018.5064