User Requirements Analysis on Augmented Reality-Based Maintenance in Manufacturing

Joel Murithi Runji1, Yun‐Ju Lee1, Chih‐Hsing Chu1
1Department of Industrial Engineering and Engineering Management, National Tsing Hua University, Hsinchu 30013, Taiwan

Tóm tắt

Abstract Maintenance of technical equipment in manufacturing is inevitable for sustained productivity with minimal downtimes. Elimination of unscheduled interruptions as well as real-time monitoring of equipment health can potentially benefit from adopting augmented reality (AR) technology. How best to employ this technology in maintenance demands a fundamental comprehension of user requirements for production planners. Despite AR applications being developed to assist various manufacturing operations, no previous study has examined how these user requirements in maintenance have been fulfilled and the potential opportunities that exist for further development. Reviews on maintenance have been general on all industrial fields rather than focusing on a specific industry. In this regard, a systematic literature review was performed on previous studies on AR applications in the maintenance of manufacturing entities from 2017 to 2021. Specifically, the review examines how user requirements have been addressed by these studies and identifies gaps for future research. The user requirements are drawn from the challenges encountered during AR-based maintenance in manufacturing following a similar approach to usability engineering methodologies. The needs are identified as ergonomics, communication, situational awareness, intelligence sources, feedback, safety, motivation, and performance assessment. Contributing factors to those needs are cross-tabulated with the requirements and their results presented as trends, prior to drawing insights and providing possible future suggestions for the made observations.

Từ khóa


Tài liệu tham khảo

Cholasuke, 2004, The Status of Maintenance Management in UK Manufacturing Organisations: Results From a Pilot Survey, J. Qual. Maint. Eng., 10, 5, 10.1108/13552510410526820

Bamber, 1999, Factors Affecting Successful Implementation of Total Productive Maintenance: A UK Manufacturing Case Study Perspective, J. Qual. Maint. Eng., 5, 162, 10.1108/13552519910282601

Bottani, 2019, Augmented Reality Technology in the Manufacturing Industry: A Review of the Last Decade, IISE Trans., 51, 284, 10.1080/24725854.2018.1493244

Opiyo, 2021, A Review on Teleoperation of Mobile Ground Robots: Architecture and Situation Awareness, Int. J. Control Autom. Syst., 19, 1384, 10.1007/s12555-019-0999-z

Elia, 2016, Evaluating the Application of Augmented Reality Devices in Manufacturing From a Process Point of View: An AHP Based Model, Expert Syst. Appl., 63, 187, 10.1016/j.eswa.2016.07.006

Sahu, 2020, Artificial Intelligence (AI) in Augmented Reality (AR)-Assisted Manufacturing Applications: A Review, Int. J. Prod. Res., 59, 4903, 10.1080/00207543.2020.1859636

Egger, 2020, Augmented Reality in Support of Intelligent Manufacturing—A Systematic Literature Review, Comput. Ind. Eng., 140, 106195, 10.1016/j.cie.2019.106195

Ong, 2008, Augmented Reality Applications in Manufacturing: A Survey, Int. J. Prod. Res., 46, 2707, 10.1080/00207540601064773

Wang, 2020, A Comprehensive Survey of AR/MR-Based Co-Design in Manufacturing, Eng. Comput., 36, 1715, 10.1007/s00366-019-00792-3

Baroroh, 2020, Systematic Literature Review on Augmented Reality in Smart Manufacturing: Collaboration Between Human and Computational Intelligence, J. Manuf. Syst., 61, 696, 10.1016/j.jmsy.2020.10.017

Gattullo, 2020, Design Preferences on Industrial Augmented Reality: A Survey With Potential Technical Writers, 172

Ahuja, 2011, Total Productive Maintenance Practices in Manufacturing Organisations: Literature Review, Int. J. Technol. Policy Manage., 11, 117, 10.1504/IJTPM.2011.040401

Ahuja, 2008, Total Productive Maintenance: Literature Review and Directions, Int. J. Qual. Reliab. Manage., 25, 709, 10.1108/02656710810890890

Budai, 2008, Complex Systems Maintenance Handbook, 321, 10.1007/978-1-84800-011-7_13

Hami, 2020, A Review of Sustainable Maintenance in the Manufacturing Companies, Int. J. Supply Chain Manage., 9, 935

Franciosi, 2020, Measuring Maintenance Impacts on Sustainability of Manufacturing Industries: From a Systematic Literature Review to a Framework Proposal, J. Clean. Prod., 260, 121065, 10.1016/j.jclepro.2020.121065

Jin, 2016, The Present Status and Future Growth of Maintenance in US Manufacturing: Results From a Pilot Survey, Manuf. Rev., 3, 1, 10.1051/mfreview/2016005

Dilanthi, 2013, Improving Productivity With Maintenance Function in Manufacturing Industry of Sri Lanka: Literature Review, Int. J. Educ. Res., 1, 1, 10.12735/ier.v1i4p01

Silvestri, 2020, Maintenance Transformation Through Industry 4.0 Technologies: A Systematic Literature Review, Comput. Ind., 123, 103335, 10.1016/j.compind.2020.103335

Lee, 1995, Modern Computer-Aided Maintenance of Manufacturing Equipment and Systems: Review and Perspective, Comput. Ind. Eng., 28, 793, 10.1016/0360-8352(95)00012-P

Palmarini, 2018, A Systematic Review of Augmented Reality Applications in Maintenance, Robot. Comput. Integr. Manuf., 49, 215, 10.1016/j.rcim.2017.06.002

del Amo, 2018, A Systematic Review of Augmented Reality Content-Related Techniques for Knowledge Transfer in Maintenance Applications, Comput. Ind., 103, 47, 10.1016/j.compind.2018.08.007

Gattullo, 2020, What, How, and Why Are Visual Assets Used in Industrial Augmented Reality? A Systematic Review and Classification in Maintenance, Assembly, and Training (From 1997 to 2019), IEEE Trans. Vis. Comput. Graph., 28, 1443, 10.1109/tvcg.2020.3014614

Byvaltsev, 2020, Review of the Features of Augmented Reality Application in the Training of Operators and Maintenance Staff, 10.1088/1757-899X/966/1/012121

Koh, 2020, A Review on Augmented Reality Tracking Methods for Maintenance of Robots, J. Teknol., 83, 37, 10.11113/jurnalteknologi.v83.14907

Princle, 2018, [Poster] Using an Industry-Ready AR HMD on a Real Maintenance Task: AR Benefits Performance on Certain Task Steps More Than Others, 236

Henderson, 2009, Evaluating the Benefits of Augmented Reality for Task Localization in Maintenance of an Armored Personnel Carrier Turret, 135

Sanna, 2015, Using Handheld Devices to Support Augmented Reality-Based Maintenance and Assembly Tasks, 178

Rowen, 2019, Impacts of Wearable Augmented Reality Displays on Operator Performance, Situation Awareness, and Communication in Safety-Critical Systems, Appl. Ergon., 80, 17, 10.1016/j.apergo.2019.04.013

Davis, 1982, Strategies for Information Requirements Determination, IBM Syst. J., 21, 4, 10.1147/sj.211.0004

Lindgaard, 2006, User Needs Analysis and Requirements Engineering: Theory and Practice, Interact. Comput., 18, 47, 10.1016/j.intcom.2005.06.003

Karwowski, 2006, Handbook of Human Factors and Ergonomics, 3

Aromaa, 2016, Use of Wearable and Augmented Reality Technologies in Industrial Maintenance Work, 235

Endsley, 1995, Measurement of Situation Awareness in Dynamic Systems, Hum. Factors, 37, 65, 10.1518/001872095779049499

Oliveira, 2014, Human-Centered Interfaces for Situation Awareness in Maintenance, 193

Tatić, 2017, The Application of Augmented Reality Technologies for the Improvement of Occupational Safety in an Industrial Environment, Comput. Ind., 85, 1, 10.1016/j.compind.2016.11.004

Neumann, 1998, Cognitive, Performance, and Systems Issues for Augmented Reality Applications in Manufacturing and Maintenance, 4

Oliveira, 2015, A Head-up Display With Augmented Reality and Gamification for an E-Maintenance System: Using Interfaces and Gamification to Motivate Workers in Procedural Tasks, 10.1007/978-3-319-20618-9_50

Jetter, 2018, Augmented Reality Tools for Industrial Applications: What Are Potential Key Performance Indicators and Who Benefits?, Comput. Human Behav., 87, 18, 10.1016/j.chb.2018.04.054

Lorenz, 2018, Industrial Augmented Reality: Requirements for an Augmented Reality Maintenance Worker Support System, 151

Schlagowski, 2018, Design of an Assistant System for Industrial Maintenance Tasks and Implementation of a Prototype Using Augmented Reality, 294

Nikolic, 2006, A Monitor-Based AR System as a Support Tool for Industrial Maintenance, IFAC Proc., 39, 902, 10.3182/20060912-3-DE-2911.00155

Mourtzis, 2021, Integrated and Adaptive AR Maintenance and Shop-Floor Rescheduling, Comput. Ind., 125, 10.1016/j.compind.2020.103383

Kollatsch, 2021, Efficient Integration Process of Production Data Into Augmented Reality Based Maintenance of Machine Tools, Prod. Eng., 15, 311, 10.1007/s11740-021-01026-6

Ivascu, 2021, Augmented Reality Production Monitoring Control Room In: Auer M., May D. (eds) Cross Reality and Data Science in Engineering. REV 2020. Advances in Intelligent Systems and Computing

Morillo, 2020, Comparative Study of AR Versus Video Tutorials for Minor Maintenance Operations, Multimed. Tools Appl., 79, 7073, 10.1007/s11042-019-08437-9

Mourtzis, 2020, Real-Time Remote Maintenance Support Based on Augmented Reality (AR), Appl. Sci., 10, 1855, 10.3390/app10051855

Fleck, 2020, Maui: Tele-Assistance for Maintenance of Cyber-Physical Systems, 800

Park, 2020, Deep Learning-Based Smart Task Assistance in Wearable Augmented Reality, Robot. Comput. Integr. Manuf., 63, 10.1016/j.rcim.2019.101887

Mourtzis, 2020, A Framework for Automatic Generation of Augmented Reality Maintenance & Repair Instructions Based on Convolutional Neural Networks, Procedia CIRP, 93, 977, 10.1016/j.procir.2020.04.130

Vorraber, 2020, Assessing Augmented Reality in Production: Remote-Assisted Maintenance With HoloLens, Procedia CIRP, 88, 139, 10.1016/j.procir.2020.05.025

Coscetti, 2020, Factory Maintenance Application Using Augmented Reality, 10.1145/3378184.3378218

Illankoon, 2020, Collaborating AI and Human Experts in the Maintenance Domain, AI Soc., 36, 817, 10.1007/s00146-020-01076-x

Vignali, 2019, Development of a 4.0 Industry Application for Increasing Occupational Safety: Guidelines for a Correct Approach, 10.1109/ICE.2019.8792814

Burova, 2020, Utilizing VR and Gaze Tracking to Develop AR Solutions for Industrial Maintenance, 10.1145/3313831.3376405

Dey, 2016, Augmented Reality Based Integrated Intelligent Maintenance System for Production Line, 126

Ortega, 2021, Mantra: An Effective System Based on Augmented Reality and Infrared Thermography for Industrial Maintenance, Appl. Sci., 11, 1, 10.3390/app11010385

Brauner, 2014, Evaluation of Different Feedback Conditions on Worker’s Performance in an Augmented Reality-Based Support System for Carbon Fiber Reinforced Plastic Manufacturing, 5087

Brice, 2020, Augmented Reality, Virtual Reality, and Computer Graphics. AVR 2020. Lecture Notes in Computer Science

Nguyen, 2020, Advances in Agile and User-Centred Software Engineering. LASD 2019, MIDI 2019. Lecture Notes in Business Information Processing

Gattullo, 2020, Towards Next Generation Technical Documentation in Augmented Reality Using a Context-Aware Information Manager, Appl. Sci., 10, 780, 10.3390/app10030780

Wang, 2021, The Role of User-Centered AR Instruction in Improving Novice Spatial Cognition in a High-Precision Procedural Task, Adv. Eng. Inform., 47, 101250, 10.1016/j.aei.2021.101250

He, 2019, A Mobile Solution for Augmenting a Manufacturing Environment With User-Generated Annotations, Information, 10, 1, 10.3390/info10020060

Khalil, 2019, IIoT-Based Fatigue Life Indication Using Augmented Reality, 746

Freddi, 2020, Design for Disassembly and Augmented Reality Applied to a Tailstock, Actuators, 9, 1, 10.3390/act9040102

Izquierdo-Domenech, 2020, Supporting Interaction in Augmented Reality Assisted Industrial Processes Using a CNN-Based Semantic Layer, 27

Kim, 2020, Industrial Augmented Reality: Concepts and User Interface Designs for Augmented Reality Maintenance Worker Support Systems, 67

Lorenz, 2020, Industrial Augmented Reality: 3D-Content Editor for Augmented Reality Maintenance Worker Support System, 203

Prouzeau, 2020, Corsican Twin: Authoring In Situ Augmented Reality Visualisations in Virtual Reality, 10.1145/3399715.3399743

Quint, 2017, The Challenge of Introducing AR in Industry—Results of a Participative Process Involving Maintenance Engineers, Procedia Manuf., 11, 1319, 10.1016/j.promfg.2017.07.260

Vogel-Heuser, 2020, Assisted Safety Test Execution Linking E-CAD and Test Management With Augmented Reality, 196

Alves, 2020, Deployment of a Smart and Predictive Maintenance System in an Industrial Case Study, 493, 10.1109/ISIE45063.2020.9152441

Serras, 2020, Dialogue Enhanced Extended Reality: Interactive System for the Operator 4.0, Appl. Sci., 10, 3960, 10.3390/app10113960

Obermair, 2020, Maintenance With Augmented Reality Remote Support in Comparison to Paper-Based Instructions: Experiment and Analysis, 942

Patibandla, 2020, Augmented Reality for Assistive Maintenance and Real-Time Failure Analysis in Industries, 149

Gao, 2020, Advanced Manufacturing and Automation IX. IWAMA 2019. Lecture Notes in Electrical Engineering

Ariansyah, 2020, Smart Maintenance: A Wearable Augmented Reality Application Integrated With CMMS to Minimize Unscheduled Downtime, Comput. Aided Des. Appl., 17, 740, 10.14733/cadaps.2020.740-751

Erkoyuncu, 2020, Olfactory-Based Augmented Reality Support for Industrial Maintenance, IEEE Access, 8, 30306, 10.1109/ACCESS.2020.2970220

Yigitbas, 2020, On the Development of Context-Aware Augmented Reality Applications, 10.1007/978-3-030-46540-7_11

Horejsi, 2020, A Smart Factory in a Smart City: Virtual and Augmented Reality in a Smart Assembly Line, IEEE Access, 8, 94330, 10.1109/ACCESS.2020.2994650

Eder, 2020, On the Application of Augmented Reality in a Learning Factory Working Environment, Procedia Manuf., 45, 7, 10.1016/j.promfg.2020.04.030

Schmiedinger, 2020, Augmented Reality as a Tool for Providing Informational Content in Different Production Domains, Procedia Manuf., 45, 423, 10.1016/j.promfg.2020.04.047

Boonbrahm, 2020, The Use of Marker-Based Augmented Reality in Space Measurement, Procedia Manuf., 42, 337, 10.1016/j.promfg.2020.02.081

Ivaschenko, 2020, Advances in Automation. RusAutoCon 2019. Lecture Notes in Electrical Engineering

Barreiros, 2020, Adaptive Instructional Systems. HCII 2020. Lecture Notes in Computer Science

Zhukovskiy, 2019, Electrical Equipment Maintenance System With Elements of Augmented Reality Technology, 10.1088/1757-899X/643/1/012024

Coscetti, 2019, Augmented Reality for Tissue Converting Maintenance, 585

Cachada, 2019, Using AR Interfaces to Support Industrial Maintenance Procedures, 3795

Sun, 2019, Towards Industrial IoT-AR Systems Using Deep Learning-Based Object Pose Estimation, 10.1109/IPCCC47392.2019.8958753

Zhao, 2019, Research on Maintenance Guiding System Based on Augmented Reality, 944

Gao, 2019, Research on Visual Monitoring and Auxiliary Maintenance Technology of Equipment Based on Augmented Reality, 93

Siew, 2019, Improving Maintenance Efficiency With an Adaptive AR-Assisted Maintenance System, 74

Kuster, 2019, A Distributed Architecture for Modular and Dynamic Augmented Reality Processes, 663

Fullen, 2019, Machine Learning for Assistance Systems: Pattern-Based Approach to Online Step Recognition, 296

Prajapat, 2019, A Framework for Next Generation Interactive and Immersive Des Models, 671

Zubizarreta, 2019, A Framework for Augmented Reality Guidance in Industry, Int. J. Adv. Manuf. Technol., 102, 4095, 10.1007/s00170-019-03527-2

Kostolani, 2019, Intelligent Predictive Maintenance Control Using Augmented Reality, 131

Manuri, 2019, A State Validation System for Augmented Reality Based Maintenance Procedures, Appl. Sci., 9, 2115, 10.3390/app9102115

Costal, 2019, Modeling of Video Projectors in OpenGL for Implementing a Spatial Augmented Reality Teaching System for Assembly Operations

Tran, 2019, Mapping a Virtual View to the Physical World to Guide the Completion of Complex Task Sequences, 10.1109/SYSCON.2019.8836724

Rajan, 2019, Machine Fault Diagnostics and Condition Monitoring Using Augmented Reality and IoT, 910

Dhiman, 2019, Worker Assistance in Smart Production Environments Using Pervasive Technologies, 95

Liu, 2019, Advanced Manufacturing and Automation VIII. IWAMA 2018. Lecture Notes in Electrical Engineering

Aschenbrenner, 2019, Comparing Human Factors for Augmented Reality Supported Single-User and Collaborative Repair Operations of Industrial Robots, Front. Robot. AI, 6, 1, 10.3389/frobt.2019.00037

De Pace, 2019, A Comparison Between Two Different Approaches for a Collaborative Mixed-Virtual Environment in Industrial Maintenance, Front. Robot. AI, 6, 1, 10.3389/frobt.2019.00051

Phillips, 2019, Artificial Intelligence and Mobile Services – AIMS 2019. AIMS 2019. Lecture Notes in Computer Science

Mohan, 2019, Building Jarvis—A Learner-Aware Conversational Trainer

Hamidane, 2018, Implementation of a Preventive Maintenance System Based on Augmented Reality, 10.1109/PAIS.2018.8598510

Häkkilä, 2018, Exploring Head Mounted Display Based Augmented Reality for Factory Workers, 499

Cachada, 2018, Maintenance 4.0: Intelligent and Predictive Maintenance System Architecture, 139

Kammerer, 2018, Towards Context-Aware Process Guidance in Cyber-Physical Systems With Augmented Reality, 44

Zhou, 2018, Poster: Pose-Assisted Active Visual Recognition in Mobile Augmented Reality, 756

Aromaa, 2018, Human Factors and Ergonomics Evaluation of a Tablet Based Augmented Reality System in Maintenance Work, 118

Princle, 2018, Using an Industry-Ready AR HMD on a Real Maintenance Task: AR Benefits Performance on Certain Task Steps More Than Others, 236

Huang, 2018, Augmented 3D Hands: A Gesture-Based Mixed Reality System for Distributed Collaboration, J. Multimodal User Interfaces, 12, 77, 10.1007/s12193-017-0250-2

Crivellaro, 2018, Robust 3D Object Tracking From Monocular Images Using Stable Parts, IEEE Trans. Pattern Anal. Mach. Intell., 40, 1465, 10.1109/TPAMI.2017.2708711

Seitz, 2018, Augmenting the Industrial Internet of Things With Emojis, 240

Vignali, 2018, Design and Testing of an Augmented Reality Solution to Enhance Operator Safety in the Food Industry, Int. J. Food Eng., 14, 1, 10.1515/ijfe-2017-0122

Aromaa, 2018, Advances in Usability and User Experience. AHFE 2017. Advances in Intelligent Systems and Computing

Neges, 2018, Intelligent Systems in Production Engineering and Maintenance – ISPEM 2017. ISPEM 2017. Advances in Intelligent Systems and Computing

Leutert, 2018, Projector-Based Augmented Reality for Telemaintenance Support, IFAC-PapersOnLine, 51, 502, 10.1016/j.ifacol.2018.08.368

Setti, 2018, Intelligent Systems and Applications. IntelliSys 2016. Studies in Computational Intelligence

Rabah, 2018, Towards Improving the Future of Manufacturing Through Digital Twin and Augmented Reality Technologies, Procedia Manuf., 17, 460, 10.1016/j.promfg.2018.10.070

Mourtzis, 2018, Maintenance Assistance Application of Engineering to Order Manufacturing Equipment: A Product Service System (PSS) Approach, IFAC-PapersOnLine, 51, 217, 10.1016/j.ifacol.2018.08.263

Heinz, 2018, Machine Learning and Knowledge Extraction. CD-MAKE 2018. Lecture Notes in Computer Science

Mourtzis, 2018, Mobile Apps for Providing Product-Service Systems and Retrieving Feedback Throughout Their Lifecycle: A Robotics Use Case, Int. J. Prod. Lifecycle Manage., 11, 116, 10.1504/IJPLM.2018.092821

Vincent, 2017, Inoovas—Industrial Ontology for Operation in Virtual and Augmented Scene: The Architecture, 300

Dvorak, 2017, Object State Recognition for Automatic AR-Based Maintenance Guidance, 1244

Wolfartsberger, 2017, Towards an Augmented Reality and Sensor-Based Assistive System for Assembly Tasks, 230

Mourtzis, 2017, Cloud-Based Augmented Reality Remote Maintenance Through Shop-Floor Monitoring: A Product-Service System Approach, ASME J. Manuf. Sci. Eng. Trans., 139, 061011, 10.1115/1.4035721

Hao, 2017, The Role of Wearable Devices in Meeting the Needs of Cloud Manufacturing: A Case Study, Robot. Comput. Integr. Manuf., 45, 168, 10.1016/j.rcim.2015.10.001

Kranzer, 2017, An Intelligent Maintenance Planning Framework Prototype for Production Systems, 1124

Jayaweera, 2017, Demo Abstract: Enhanced Real-Time Machine Inspection With Mobile Augmented Reality for Maintenance and Repair, 287

Matsuzoe, 2017, Intuitive Visualization Method for Locating Off-Screen Objects Inspired by Motion Perception in Peripheral Vision, 10.1145/3041164.3041198

Lamberti, 2017, Using Semantics to Automatically Generate Speech Interfaces for Wearable Virtual and Augmented Reality Applications, IEEE Trans. Hum.–Mach. Syst., 47, 152, 10.1109/THMS.2016.2573830

Perla, 2017, InspectAR: An Augmented Reality Inspection Framework for Industry, 355

Yew, 2017, Immersive Augmented Reality Environment for the Teleoperation of Maintenance Robots, Procedia CIRP, 61, 305, 10.1016/j.procir.2016.11.183

Abramovici, 2017, Context-Aware Maintenance Support for Augmented Reality Assistance and Synchronous Multi-User Collaboration, Procedia CIRP, 59, 18, 10.1016/j.procir.2016.09.042

Mourtzis, 2017, Advances in Production Management Systems. The Path to Intelligent, Collaborative and Sustainable Manufacturing. APMS 2017. IFIP Advances in Information and Communication Technology

Deac, 2017, Using Augmented Reality in Smart Manufacturing, 727

Hiraoka, 2017, Replacement of Parts by Part Agents to Promote Reuse of Mechanical Parts, 10.1007/978-3-319-72905-3_35