Application and Evaluation of the Reinforcement Learning Approach to Eco-Driving at Intersections under Infrastructure-to-Vehicle Communications
Tóm tắt
Eco-driving behavior is able to improve vehicles’ fuel consumption efficiency and minimize exhaust emissions, especially with the presence of infrastructure-to-vehicle (I2V) communications for connected vehicles. Several techniques such as dynamic programming and neural networks have been proposed to study eco-driving behavior. However, most techniques need a complicated problem-solving process and cannot be applied to dynamic traffic conditions. Comparatively, reinforcement learning (RL) presents great potential for self-learning to take actions in a complicated environment to achieve the optimal mapping between traffic conditions and the corresponding optimal control action of a vehicle. In this paper, a vehicle was treated as an agent to select its maneuver, that is, acceleration, cruise speed, and deceleration, according to dynamic conditions while approaching a signalized intersection equipped with I2V communication. An improved cellular automation model was utilized as the simulation platform. Three parameters, including the distance between the vehicle and the intersection, signal status, and instant vehicle speeds, were selected to characterize real-time traffic state. The total CO2 emitted by the vehicle on the approach to the intersection serves as a measure of reward policy that informs the vehicle how good its operation was. The Q-learning algorithm was utilized to optimize vehicle driving behaviors for eco-driving. Vehicle exhaust emissions and traffic performance (travel time, stop duration, and stop rate) were evaluated in two cases: (1) an isolated intersection, and (2) a medium-scale realistic network. Simulation results showed that the eco-driving behavior obtained by RL can not only reduce emissions but also optimize traffic performance.
Từ khóa
Tài liệu tham khảo
United States Environmental Protection Agency, 2015, U.S. Greenhouse Gas Inventory Report: 1990-2014
Ministry of Environmental Protection of the People’s Republic of China, 2015, China Vehicle Emission Control Annual Report: 2015
Qian G., 2011, Proc., Australian Transport Research Forum 2011
Mensing F., Bideaux E., Trigui R., Ribet J., Jeanneret B. Eco-driving: An Economic or Ecologic Driving Style? Transportation Research Part C: Emerging Technologies, Vol. 38, 2014, pp. 110–121. https://doi.org/10.1016/j.trc.2013.10.013
Zarkadoula M., Zoidis G., Tritopoulou E. Training Urban Bus Drivers to Promote Smart Driving: A Note on a Greek Eco-driving Pilot Program. Transportation Research Part D: Transport and Environment, Vol. 12, No. 6, 2007, pp. 449–451. https://doi.org/10.1016/j.trd.2007.05.002
Ho S. H., Wong Y. D., Chang V. W. C. What Can Eco-driving Do for Sustainable Road Transport? Perspectives from a City (Singapore) Eco-driving Programme. Sustainable Cities and Society, Vol. 14, 2015, pp. 82–88. https://doi.org/10.1016/j.scs.2014.08.002
Xu Y., Li H., Liu H., Rodgers M. O., Guensler R. L. Eco-driving for Transit: An Effective Strategy to Conserve Fuel and Emissions. Applied Energy, Vol. 194, 2017, pp. 784–797. https://doi.org/10.1016/j.apenergy.2016.09.101
Li J., Dridi M., El-Moudni A. A Cooperative Traffic Control of Vehicle-intersection (CTCVI) for the Reduction of Traffic Delays and Fuel Consumption. Sensors, Vol. 16, No. 12, 2016, pp. 2175–2175. https://doi.org/10.3390/s16122175
Barth M., Boriboonsomsin K. Energy and Emissions Impacts of a Freeway-Based Dynamic Eco-driving System. Transportation Research Part D: Transport and Environment, Vol. 14, No. 6, 2009, pp. 400–410. https://doi.org/10.1016/j.trd.2009.01.004
Schall D. L., Mohnen A. Incentivizing Energy-Efficient Behavior at Work: An Empirical Investigation Using a Natural Field Experiment on Eco-driving. Applied Energy, Vol. 185, 2017, pp. 1757–1768. https://doi.org/10.1016/j.apenergy.2015.10.163
Li J., Dridi M., El-Moudni A. A Cooperative Traffic Control of Vehicle-Intersection (CTCVI) for the Reduction of Traffic Delays and Fuel Consumption. Sensors, Vol. 16, No. 12, 2016, pp. 2175–2175. https://doi.org/10.3390/s16122175
Milesich T., Bucha J., Gulan L., Danko J. The Possibility of Applying Neural Networks to Influence Vehicle Energy Consumption by Eco Driving. Proc., International Conference Mechatronics, Springer, Cham. 2017, pp. 372–379. https://doi.org/10.1007/978-3-319-65960-2_46
Jiang Y., Zanon M., Hult R., Houska B. Distributed Algorithm for Optimal Vehicle Coordination at Traffic Intersections. IFAC World Congress, Vol. 50, No. 1, 2017, pp. 12082–12087. https://doi.org/10.1016/j.ifacol.2017.08.1511
Sutton R. S., 1998, Reinforcement Learning: An introduction
Abdulhai B., Pringle R., Karakoulas G. J. Reinforcement Learning for True Adaptive Traffic Signal Control. Journal of Transportation Engineering, Vol. 129, No. 3, 2003, pp. 278–285. https://doi.org/10.1061/(ASCE)0733-947X(2003)129:3(278)
Walraven E., Spaan M. T. J., Bakker B. Traffic Flow Optimization: A Reinforcement Learning Approach. Engineering Applications of Artificial Intelligence, Vol. 52, 2016, pp. 203–212. https://doi.org/10.1016/j.engappai.2016.01.001
Zolfpour-Arokhlo M., Selamat A., Hashim S. Z. M., Afkhami H. Modeling of Route Planning System Based on Q Value-Based Dynamic Programming with Multi-Agent Reinforcement Learning algorithms. Engineering Applications of Artificial Intelligence, Vol. 29, 2014, pp. 163–177. https://doi.org/10.1016/j.engappai.2014.01.001
Watkins C. J., Dayan P. Q-Learning. Machine Learning, Vol. 8, No. 3–4, 1992, pp. 279–292. https://doi.org/10.1007/BF00992698
United States Environmental Protection Agency, 2008, Average Annual Emissions and Fuel Consumption for Passenger Cars and Light Trucks
Jimenez-Palacios J. L., 1998, Understanding and Quantifying Motor Vehicle Emissions with Vehicle Specific Power and TILDAS Remote Sensing
Frey H. C., 2013, Development and Evaluation of a Simplified Version of MOVES for Coupling with a Traffic Simulation Model
Nagel K., Schreckenberg M. A Cellular Automaton Model for Freeway Traffic. Journal De Physique I, Vol. 2, No. 12, 1992, pp. 2221–2229. https://doi.org/10.1051/jp1:1992277
Wang J., Rakha H. A. Fuel Consumption Model for Conventional Diesel Buses. Applied Energy, Vol. 170, 2016, pp. 394–402. https://doi.org/10.1016/j.apenergy.2016.02.124
Shi J. Q., Hu Y. J., Li S. L., Zhang X. H., Mao C. Y. Simulation and Analysis of Road Construction Traffic Flow in Urban Road Networks. Advances in Mechanical Engineering, Vol. 7, No. 11, 2015, pp. 1–6. https://doi.org/10.1177/1687814015618176