Continuous design control for machine learning in certified medical systems

Software Quality Journal - 2023
Vlad Stirbu1, Tuomas Granlund2, Tommi Mikkonen3
1CompliancePal, Tampere, Finland
2Solita, Tampere, Finland
3University of Jyväskylä, Jyväskylä, Finland

Tóm tắt

AbstractContinuous software engineering has become commonplace in numerous fields. However, in regulating intensive sectors, where additional concerns need to be taken into account, it is often considered difficult to apply continuous development approaches, such as devops. In this paper, we present an approach for using pull requests as design controls, and apply this approach to machine learning in certified medical systems leveraging model cards, a novel technique developed to add explainability to machine learning systems, as a regulatory audit trail. The approach is demonstrated with an industrial system that we have used previously to show how medical systems can be developed in a continuous fashion.

Từ khóa


Tài liệu tham khảo

Aho, T., Sievi-Korte, O., Kilamo, T., Yaman, S., Mikkonen, T. (2020). Demystifying data science projects: A look on the people and process of data science today. In: International Conference on Product-focused Software Process Improvement (PROFES’20), pp. 153–167. Springer.

AWS Solutions. (2021). AWS MLOps Framework. https://docs.aws.amazon.com/solutions/ latest/aws-mlops-framework/welcome.html. Retrieved 14 March 2021.

Bass, L., Weber, I., Zhu, L. (2015). DevOps: A Software Architect’s Perspective. Addison-Wesley Professional.

Baylor, D., Breck, E., Cheng, H.-T., Fiedel, N., Foo, C. Y., Haque, Z., Haykal, S., Ispir, M., Jain, V., Koc, L., Koo, C. Y., Lew, L., Mewald, C., Modi, A. N., Polyzotis, N., Ramesh, S., Roy, S., Whang, S. E., Wicke, M., Wilkiewicz, J., Zhang, X., Zinkevich, M. (2017). Tfx: A tensorflow-based production-scale machine learning platform. In: Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’17, pp. 1387–1395. Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3097983.3098021

Debois, P. (2011). DevOps: A software revolution in the making. Cutter IT Journal 24(8).

Deloitte. (2017). Managing algorithmic risks – Safeguarding the use of complex algorithms and machine learning. https://www2.deloitte.com/us/en/pages/risk/articles/algorithmic-machine-learning-risk-management.html

der Benannten Stellen für Medizinprodukte in Deutschland (IG-NB), I. (2021). Fragenkatalog Künstliche Intelligenz bei Medizinprodukten. https://www.ig-nb.de/dok_view?oid=861877. Retrieved 29 December 2021.

Drvar, M., Turner, J., Piechocki, M., Stiegeler, E., Münch, D. (2020). The future of data collection and data management: Agile RegOps for digitalizing the regulatory value chain. BearingPoint Software Solutions GmbH, Frankfurt.

Ebert, C., Gallardo, G., Hernantes, J., & Serrano, N. (2016). DevOps. IEEE Software, 33(3), 94–100.

European Parliament and the Council. (2017). Regulation

(EU) 2017/745 on medical devices. https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:02017R0745-20200424#tocId168. Retrieved 21 November 2021.

European Parliament and the Council. (2017). Regulation

(EU) 2017/746 on in vitro diagnostic medical devices. https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:02017R0746-20170505#tocId157. Retrieved 21 November 2021.

FDA - Center for Devices and Radiological Health. (1997). Design Control Guidance for Medical Device Manufacturers.

Fitzgerald, B., & Stol, K.-J. (2017). Continuous software engineering: A roadmap and agenda. Journal of Systems and Software, 123, 176–189.

Food and Drug Administration. (2021). Good Machine Learning Practice for Medical Device Development Guiding Principles. https://www.regulations.gov/document/FDA-2019-N-1185-0156

Google Cloud Solutions. (2021). MLOps: Continuous delivery and automation pipelines in machine learning. Google Cloud. https://cloud.google.com/solutions/machine-learning/mlops-continuous-delivery-and-automation-pipelines-in-machine-learning. Accessed 14 March 2021.

Granlund, T., Stirbu, V., & Mikkonen, T. (2022). Medical software needs calm compliance. IEEE Software, 39(1), 19–28. https://doi.org/10.1109/MS.2021.3117292

Granlund, T., Stirbu, V., & Mikkonen, T. (2021). Towards regulatory-compliant MLOps: Oravizio’s journey from a machine learning experiment to a deployed certified medical product. SN Computer Science, 2(5), 342. https://doi.org/10.1007/s42979-021-00726-1

Granlund, T., Vedenpää, J., Stirbu, V., Mikkonen, T. (2021). On medical device cybersecurity compliance in eu. In: 2021 IEEE/ACM 3rd International Workshop on Software Engineering for Healthcare (SEH), pp. 20–23. https://doi.org/10.1109/SEH52539.2021.00011

Humble, J., Farley, D. (2010). Continuous Delivery: Reliable Software Releases Through Build, Test, and Deployment Automation. Pearson Education.

International Electrotechnical Commission. (2015). IEC 62304:2006/A1:2015. Medical device software - Software life-cycle processes.

International Electrotechnical Commission. (2016). IEC 82304-1:2016. Health software - Part 1: General requirements for product safety.

International Medical Device Regulators Forum. (2022). Machine Learning-enabled Medical Devices: Key Terms and Definitions.

International Organization for Standardization. (2016). ISO 13485:2016. Medical devices - Quality management systems - Requirements for regulatory purposes.

International Organization for Standardization. (2019). ISO 14971:2019. Medical devices - Application of risk management to medical devices.

John, M. M., Olsson, H. H., Bosch, J. (2021). Towards MLOps: A framework and maturity model. In: 2021 47th Euromicro Conference on Software Engineering and Advanced Applications (SEAA’21), pp. 1–8. IEEE.

Jørgensen, N. (2001). Putting it all in the trunk: Incremental software development in the FreeBSD open source project. Information Systems Journal, 11(4), 321–336.

Lipton, Z. C. (2018). The mythos of model interpretability: In machine learning, the concept of interpretability is both important and slippery. Queue, 16(3), 31–57.

Medical Device Coordination Group. (2021). Ongoing guidance development and deliverables of MDCG Subgroups - October 2021. https://ec.europa.eu/health/sites/default/files/md_sector/docs/ mdcg_ongoing_guidancedocs_en.pdf. Retrieved 29 December 2021.

Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., Gebru, T. (2019). Model cards for model reporting. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 220–229.

Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., Gebru, T. (2019). Model cards for model reporting. In: Proceedings of the Conference on Fairness, Accountability, and Transparency. FAT* ’19, pp. 220–229. Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3287560.3287596

Myrbakken, H., Colomo-Palacios, R. (2017). DevSecOps: A multivocal literature review. In: International Conference on Software Process Improvement and Capability Determination, pp. 17–29. Springer.

Rajkumar, M., Pole, A. K., Adige, V. S., Mahanta, P. (2016). Devops culture and its impact on cloud delivery and software development. In: 2016 International Conference on Advances in Computing, Communication, & Automation (ICACCA)(Spring), pp. 1–6. IEEE.

Sato, D., Wilder, A., Windheuser, C. (2019). Continuous Delivery for Machine Learning. https://martinfowler.com/articles/cd4ml.html Retrieved 21 December 2020.

Sculley, D., Holt, G., Golovin, D., Davydov, E., Phillips, T., Ebner, D., Chaudhary, V., Young, M., Crespo, J.-F., Dennison, D. (2015). Hidden technical debt in machine learning systems. In: Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 2. NIPS’15, pp. 2503–2511. MIT Press, Cambridge, MA, USA.

Stirbu, V., Mikkonen, T. (2021). Introducing traceability in github for medical software development. In: Product-Focused Software Process Improvement (PROFES’21). Springer

Stirbu, V., Mikkonen, T. (2018). Towards agile yet regulatory-compliant development of medical software. In: 2018 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW), pp. 337–340. IEEE.

Taivalsaari, A., Mikkonen, T., Ingalls, D., Palacz, K. (2008). Web browser as an application platform. In: 2008 34th Euromicro Conference Software Engineering and Advanced Applications, pp. 293–302. IEEE.

Toivakka, H., Granlund, T., Poranen, T., Zhang, Z. (2021). Towards RegOps: A DevOps Pipeline for Medical Device Software. In: Product Focused Software Improvement (PROFES’21). Springer.

Treveil, M., Omont, N., Stenac, C., Lefevre, K., Phan, D., Zentici, J., Lavoillotte, A., Miyazaki, M., Heidmann, L. (2020). Introducing MLOps. O’Reilly Media, Inc.

U.S. Department of Health and Human Services. (2021). Federal Food, Drug, and Cosmetic Act. https://www.fda.gov/regulatory-information/laws-enforced-fda/federal-food-drug-and-cosmetic-act-fdc-act. Retrieved 21 November 2021.

U.S. Food and Drug Administration (FDA). (2021). Artificial Intelligence/Machine Learning (AI/ML)-Based Software as a Medical Device (SaMD) Action Plan. https://www.fda.gov/media/145022/download

Vogel, D. A. (2011). Medical Device Software Verification, Validation and Compliance. Artech House, Boston/London.

Thông tin
Thông tin xuất bản

Software Quality Journal

Thông tin tác giả