Application and Evaluation of the Reinforcement Learning Approach to Eco-Driving at Intersections under Infrastructure-to-Vehicle Communications

Transportation Research Record - Tập 2672 Số 25 - Trang 89-98 - 2018
Junqing Shi1, Fengxiang Qiao2, Qing Li2, Lei Yu3,4, Yongju Hu
1Department of Traffic and Transportation, College of Engineering, Zhejiang Normal University, Jinhua, Zhejiang, China
2Innovative Transportation Research Institute, Texas Southern University, Houston, TX
3Texas Southern University, Houston, TX
4Yangtze River Scholar and Adjunct Professor, Xuchang University and Beijing Jiaotong University

Tóm tắt

Eco-driving behavior is able to improve vehicles’ fuel consumption efficiency and minimize exhaust emissions, especially with the presence of infrastructure-to-vehicle (I2V) communications for connected vehicles. Several techniques such as dynamic programming and neural networks have been proposed to study eco-driving behavior. However, most techniques need a complicated problem-solving process and cannot be applied to dynamic traffic conditions. Comparatively, reinforcement learning (RL) presents great potential for self-learning to take actions in a complicated environment to achieve the optimal mapping between traffic conditions and the corresponding optimal control action of a vehicle. In this paper, a vehicle was treated as an agent to select its maneuver, that is, acceleration, cruise speed, and deceleration, according to dynamic conditions while approaching a signalized intersection equipped with I2V communication. An improved cellular automation model was utilized as the simulation platform. Three parameters, including the distance between the vehicle and the intersection, signal status, and instant vehicle speeds, were selected to characterize real-time traffic state. The total CO2 emitted by the vehicle on the approach to the intersection serves as a measure of reward policy that informs the vehicle how good its operation was. The Q-learning algorithm was utilized to optimize vehicle driving behaviors for eco-driving. Vehicle exhaust emissions and traffic performance (travel time, stop duration, and stop rate) were evaluated in two cases: (1) an isolated intersection, and (2) a medium-scale realistic network. Simulation results showed that the eco-driving behavior obtained by RL can not only reduce emissions but also optimize traffic performance.

Từ khóa


Tài liệu tham khảo

United States Environmental Protection Agency, 2015, U.S. Greenhouse Gas Inventory Report: 1990-2014

Ministry of Environmental Protection of the People’s Republic of China, 2015, China Vehicle Emission Control Annual Report: 2015

Qian G., 2011, Proc., Australian Transport Research Forum 2011

Mensing F., Bideaux E., Trigui R., Ribet J., Jeanneret B. Eco-driving: An Economic or Ecologic Driving Style? Transportation Research Part C: Emerging Technologies, Vol. 38, 2014, pp. 110–121. https://doi.org/10.1016/j.trc.2013.10.013

Zarkadoula M., Zoidis G., Tritopoulou E. Training Urban Bus Drivers to Promote Smart Driving: A Note on a Greek Eco-driving Pilot Program. Transportation Research Part D: Transport and Environment, Vol. 12, No. 6, 2007, pp. 449–451. https://doi.org/10.1016/j.trd.2007.05.002

Ho S. H., Wong Y. D., Chang V. W. C. What Can Eco-driving Do for Sustainable Road Transport? Perspectives from a City (Singapore) Eco-driving Programme. Sustainable Cities and Society, Vol. 14, 2015, pp. 82–88. https://doi.org/10.1016/j.scs.2014.08.002

Xu Y., Li H., Liu H., Rodgers M. O., Guensler R. L. Eco-driving for Transit: An Effective Strategy to Conserve Fuel and Emissions. Applied Energy, Vol. 194, 2017, pp. 784–797. https://doi.org/10.1016/j.apenergy.2016.09.101

Li J., Dridi M., El-Moudni A. A Cooperative Traffic Control of Vehicle-intersection (CTCVI) for the Reduction of Traffic Delays and Fuel Consumption. Sensors, Vol. 16, No. 12, 2016, pp. 2175–2175. https://doi.org/10.3390/s16122175

Barth M., Boriboonsomsin K. Energy and Emissions Impacts of a Freeway-Based Dynamic Eco-driving System. Transportation Research Part D: Transport and Environment, Vol. 14, No. 6, 2009, pp. 400–410. https://doi.org/10.1016/j.trd.2009.01.004

Schall D. L., Mohnen A. Incentivizing Energy-Efficient Behavior at Work: An Empirical Investigation Using a Natural Field Experiment on Eco-driving. Applied Energy, Vol. 185, 2017, pp. 1757–1768. https://doi.org/10.1016/j.apenergy.2015.10.163

Li J., Dridi M., El-Moudni A. A Cooperative Traffic Control of Vehicle-Intersection (CTCVI) for the Reduction of Traffic Delays and Fuel Consumption. Sensors, Vol. 16, No. 12, 2016, pp. 2175–2175. https://doi.org/10.3390/s16122175

Milesich T., Bucha J., Gulan L., Danko J. The Possibility of Applying Neural Networks to Influence Vehicle Energy Consumption by Eco Driving. Proc., International Conference Mechatronics, Springer, Cham. 2017, pp. 372–379. https://doi.org/10.1007/978-3-319-65960-2_46

Jiang Y., Zanon M., Hult R., Houska B. Distributed Algorithm for Optimal Vehicle Coordination at Traffic Intersections. IFAC World Congress, Vol. 50, No. 1, 2017, pp. 12082–12087. https://doi.org/10.1016/j.ifacol.2017.08.1511

Sutton R. S., 1998, Reinforcement Learning: An introduction

Abdulhai B., Pringle R., Karakoulas G. J. Reinforcement Learning for True Adaptive Traffic Signal Control. Journal of Transportation Engineering, Vol. 129, No. 3, 2003, pp. 278–285. https://doi.org/10.1061/(ASCE)0733-947X(2003)129:3(278)

Walraven E., Spaan M. T. J., Bakker B. Traffic Flow Optimization: A Reinforcement Learning Approach. Engineering Applications of Artificial Intelligence, Vol. 52, 2016, pp. 203–212. https://doi.org/10.1016/j.engappai.2016.01.001

Zolfpour-Arokhlo M., Selamat A., Hashim S. Z. M., Afkhami H. Modeling of Route Planning System Based on Q Value-Based Dynamic Programming with Multi-Agent Reinforcement Learning algorithms. Engineering Applications of Artificial Intelligence, Vol. 29, 2014, pp. 163–177. https://doi.org/10.1016/j.engappai.2014.01.001

Watkins C. J., Dayan P. Q-Learning. Machine Learning, Vol. 8, No. 3–4, 1992, pp. 279–292. https://doi.org/10.1007/BF00992698

United States Environmental Protection Agency, 2008, Average Annual Emissions and Fuel Consumption for Passenger Cars and Light Trucks

Jimenez-Palacios J. L., 1998, Understanding and Quantifying Motor Vehicle Emissions with Vehicle Specific Power and TILDAS Remote Sensing

Frey H. C., 2013, Development and Evaluation of a Simplified Version of MOVES for Coupling with a Traffic Simulation Model

Nagel K., Schreckenberg M. A Cellular Automaton Model for Freeway Traffic. Journal De Physique I, Vol. 2, No. 12, 1992, pp. 2221–2229. https://doi.org/10.1051/jp1:1992277

Wang J., Rakha H. A. Fuel Consumption Model for Conventional Diesel Buses. Applied Energy, Vol. 170, 2016, pp. 394–402. https://doi.org/10.1016/j.apenergy.2016.02.124

Shi J. Q., Hu Y. J., Li S. L., Zhang X. H., Mao C. Y. Simulation and Analysis of Road Construction Traffic Flow in Urban Road Networks. Advances in Mechanical Engineering, Vol. 7, No. 11, 2015, pp. 1–6. https://doi.org/10.1177/1687814015618176

Shi J. Q., Cheng L., Long J. C., Liu Y. L. A New Cellular Automaton Model for Urban Two-Way Road Networks. Computational Intelligence & Neuroscience, Vol. 2014, pp. 685047. https://doi.org/10.1155/2014/685047

Thông tin
Thông tin xuất bản

Transportation Research Record

Tập 2672 Số 25

89-98

Thông tin tác giả