Project and Development of a Reinforcement Learning Based Control Algorithm for Hybrid Electric Vehicles

Applied Sciences - Tập 12 Số 2 - Trang 812
Claudio Maino1, Antonio Mastropietro2,3, Luca Sorrentino2, Enrico Busto2, Daniela Anna Misul1, Ezio Spessa1
1Politecnico di Torino, Department of Energetics, Interdepartmental Center for Automotive Research and Sustainable Mobility (CARS@PoliTO), Corso Duca Degli Abruzzi 24, 10129 Turin, Italy
2Addfor Industriale Srl, P.zza Solferino 7, 10121 Turin, Italy
3Politecnico di Torino, Department of Mathematical Sciences, corso Duca degli Abruzzi 24, 10129 Turin, Italy

Tóm tắt

Hybrid electric vehicles are, nowadays, considered as one of the most promising technologies for reducing on-road greenhouse gases and pollutant emissions. Such a goal can be accomplished by developing an intelligent energy management system which could lead the powertrain to exploit its maximum energetic performances under real-world driving conditions. According to the latest research in the field of control algorithms for hybrid electric vehicles, Reinforcement Learning has emerged between several Artificial Intelligence approaches as it has proved to retain the capability of producing near-optimal solutions to the control problem even in real-time conditions. Nevertheless, an accurate design of both agent and environment is needed for this class of algorithms. Within this paper, a detailed plan for the complete project and development of an energy management system based on Q-learning for hybrid powertrains is discussed. An integrated modular software framework for co-simulation has been developed and it is thoroughly described. Finally, results have been presented about a massive testing of the agent aimed at assessing for the change in its performance when different training parameters are considered.

Từ khóa


Tài liệu tham khảo

Onori, S., Serrao, L., and Rizzoni, G. (2016). Hybrid Electric Vehicles, Springer.

Cardoso, 2020, A review of micro and mild hybrid systems, Energy Rep., 6, 385, 10.1016/j.egyr.2019.08.077

Enang, 2017, Modelling and control of hybrid electric vehicles (A comprehensive review), Renew. Sustain. Energy Rev., 74, 1210, 10.1016/j.rser.2017.01.075

Rizzo, 2020, A survey on through-the-road hybrid electric vehicles, Electronics, 9, 1, 10.3390/electronics9050879

Singh, 2019, A comprehensive review on hybrid electric vehicles: Architectures and components, J. Mod. Transp., 27, 77, 10.1007/s40534-019-0184-3

Torreglosa, 2020, Analyzing the improvements of energy management systems for hybrid electric vehicles using a systematic literature review: How far are these controls from rule-based controls used in commercial vehicles?, Appl. Sci., 10, 1, 10.3390/app10238744

Martinez, 2017, Energy Management in Plug-in Hybrid Electric Vehicles: Recent Progress and a Connected Vehicles Perspective, IEEE Trans. Veh. Technol., 66, 4534, 10.1109/TVT.2016.2582721

Biswas, 2019, Energy management systems for electrified powertrains: State-of-the-art review and future trends, IEEE Trans. Veh. Technol., 68, 6453, 10.1109/TVT.2019.2914457

Hofman, T., van Druten, R.M., Steinbuch, M., and Serrarens, A.F.A. (2008). Rule-Based Equivalent Fuel Consumption Minimization Strategies for Hybrid Vehicles, IFAC. IFAC Proceedings Volumes 2008.

Finesso, 2016, Robust equivalent consumption-based controllers for a dual-mode diesel parallel HEV, Energy Convers. Manag., 127, 124, 10.1016/j.enconman.2016.08.021

Lin, 2003, Power management strategy for a parallel hybrid electric truck, IEEE Trans Control Syst Technol. IEEE Trans. Control Syst. Technol., 11, 839, 10.1109/TCST.2003.815606

Huang, 2017, Model predictive control power management strategies for HEVs: A review, J. Power Sources, 341, 91, 10.1016/j.jpowsour.2016.11.106

Harold, 2020, Powertrain Control for Hybrid-Electric Vehicles Using Supervised Machine Learning, Vehicles, 2, 267, 10.3390/vehicles2020015

Finesso, 2016, An unsupervised machine-learning technique for the definition of a rule-based control strategy in a complex HEV, SAE Int. J. Altern. Powertrains, 5, 308, 10.4271/2016-01-1243

Ganesh, 2022, A review of reinforcement learning based energy management systems for electrified powertrains: Progress, challenge, and potential solution, Renew. Sustain. Energy Rev., 154, 111833, 10.1016/j.rser.2021.111833

Sutton, R.S., and Barto, A.G. (2018). Reinforcement Learning: An Introduction, MIT Press.

Zhu, Z., Liu, Y., and Canova, M. (2020, January 1–3). Energy Management of Hybrid Electric Vehicles via Deep Q-Networks. Proceedings of the 2020 American Control Conference (ACC), Denver, CO, USA.

Han, 2019, Energy management based on reinforcement learning with double deep Q-learning for a hybrid electric tracked vehicle, Appl. Energy, 254, 113708, 10.1016/j.apenergy.2019.113708

Li, 2019, Energy management for a power-split hybrid electric bus via deep reinforcement learning with terrain information, Appl. Energy, 255, 113762, 10.1016/j.apenergy.2019.113762

Liu, 2015, Reinforcement Learning of Adaptive Energy Management With Transition Probability for a Hybrid Electric Tracked Vehicle, IEEE Trans. Ind. Electron., 62, 7837, 10.1109/TIE.2015.2475419

Liu, 2017, Reinforcement Learning Optimized Look-Ahead Energy Management of a Parallel Hybrid Electric Vehicle, IEEE/ASME Trans. Mechatron., 22, 1497, 10.1109/TMECH.2017.2707338

Mittal, 2020, Optimization of Energy Management Strategy for Range-Extended Electric Vehicle Using Reinforcement Learning and Neural Network, SAE Tech. Pap., 2020, 1

Biswas, A., Anselma, P.G., and Emadi, A. (2019, January 19–21). Real-Time Optimal Energy Management of Electrified Powertrains with Reinforcement Learning. Proceedings of the 2019 IEEE Transportation Electrification Conference and Expo (ITEC), Detroit, MI, USA.

Xu, 2020, Parametric study on reinforcement learning optimized energy management strategy for a hybrid electric vehicle, Appl. Energy, 259, 114200, 10.1016/j.apenergy.2019.114200

Ehsani, M., Gao, Y., Longo, S., and Ebrahimi, K.M. (2018). Modern Electric Hybrid Electric and Fuel Cell Vehicles, CRC Press.

Sundström, O., and Guzzella, L. (2009, January 8–10). A generic dynamic programming Matlab function. Proceedings of the 2009 IEEE Control Applications, (CCA) & Intelligent Control, (ISIC), St. Petersburg, Russia.

Finesso, 2016, Cost-optimized design of a dual-mode diesel parallel hybrid electric vehicle for several driving missions and market scenarios, Appl. Energy, 177, 366, 10.1016/j.apenergy.2016.05.080

Finesso, R., Misul, D., Spessa, E., and Venditti, M. (2018). Optimal design of power-split HEVs based on total cost of ownership and CO2 emission minimization. Energies, 11.

Maino, 2021, Optimal mesh discretization of the dynamic programming for hybrid electric vehicles, Appl. Energy, 292, 116920, 10.1016/j.apenergy.2021.116920

Spaan, M.T.J. (2012). Partially Observable Markon Decision Processes. Reinforcement Learning. Adaptation, Learning, and Optimization, Springer.

Watkins, C.J.C.H. (1989). Learning from Delayed Rewards. [PhD Thesis, King’s College].

Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. (2016). OpenAI Gym. arXiv.

Van Rossum, G. (2020). The Python Lybrary Reference, Release 3.8.5, Python Software Foundation.

Finesso, R., Spessa, E., and Venditti, M. (2014). Optimization of the layout and control strategy for parallel through-the-road hybrid electric vehicles. SAE Tech. Pap., 1.

Anselma, P.G., Belingardi, G., Falai, A., Maino, C., Miretti, F., Misul, D., and Spessa, E. (2019, January 2–4). Comparing parallel hybrid electric vehicle powertrains for real-world driving. Proceedings of the 2019 AEIT International Conference of Electrical and Electronic Technologies for Automotive (AEIT AUTOMOTIVE), Turin, Italy.

Maino, 2021, A deep neural network based model for the prediction of hybrid electric vehicles carbon dioxide emissions, Energy AI, 5, 100073, 10.1016/j.egyai.2021.100073

Fusco, 2018, Experimental analyses and clustering of travel choice behaviours by floating car big data in a large urban area, IET Intell. Transp. Syst., 12, 270, 10.1049/iet-its.2018.0015