Improving coordination in small-scale multi-agent deep reinforcement learning through memory-driven communication

Ahilan, S., & Dayan, P. (2019). Feudal multi-agent hierarchies for cooperative reinforcement learning. arXiv preprint arXiv:1901.08492 Brosig, J., Ockenfels, A., & Weimann, J., et al. (2003). Information and communication in sequential bargaining. Citeseer Caicedo, J. C., & Lazebnik, S. (2015). Active object localization with deep reinforcement learning. In: Proceedings of the IEEE international conference on computer vision (pp. 2488–2496). Chen, M. G. (2009). Communication, coordination, and camaraderie in world of warcraft. Games and Culture, 4(1), 47–73. Chu, X., & Ye, H. (2017). Parameter sharing deep deterministic policy gradient for cooperative multi-agent reinforcement learning. arXiv preprint arXiv:1710.00336 Comfort, L. K. (2007). Crisis management in hindsight: Cognition, communication, coordination, and control. Public Administration Review, 67, 189–197. Cooper, R., DeJong, D. V., Forsythe, R., & Ross, T. W. (1989). Communication in the battle of the sexes game: Some experimental results. The RAND Journal of Economics, 20(4), 568. Cooper, R., De Jong, D. V., Forsythe, R., & Ross, T. W. (1992). Forward induction in coordination games. Economics Letters, 40(2), 167–172. Cortes, J., Martinez, S., Karatas, T., & Bullo, F. (2002). Coverage control for mobile sensing networks. In: Proceedings of IEEE international conference on robotics and automation, 2002. ICRA’02, IEEE, (Vol. 2, pp. 1327–1332) Crites, R. H., & Barto, A. G. (1998). Elevator group control using multiple reinforcement learning agents. Machine Learning, 33(2–3), 235–262. Das, A., Gervet, T., Romoff, J., Batra, D., Parikh, D., Rabbat, M., & Pineau, J. (2018). Tarmac: Targeted multi-agent communication. arXiv preprint arXiv:1810.11187 De Ruiter, J. P., Noordzij, M. L., Newman-Norlund, S., Newman-Norlund, R., Hagoort, P., Levinson, S. C., et al. (2010). Exploring the cognitive infrastructure of communication. Interaction Studies, 11(1), 51–77. Degris, T., White, M., & Sutton, R. S. (2012). Off-policy actor-critic. arXiv preprint arXiv:1205.4839 Demichelis, S., & Weibull, J. W. (2008). Language, meaning, and games: A model of communication, coordination, and evolution. American Economic Review, 98(4), 1292–1311. Evans, R., & Gao, J . (2016) . Deepmind ai reduces google data centre cooling bill by 40 Foerster, J., Assael, I. A., de Freitas, N., & Whiteson, S. (2016). Learning to communicate with deep multi-agent reinforcement learning. In: Advances in neural information processing systems (pp. 2137–2145). Foerster, J., Farquhar, G., Afouras, T., Nardelli, N., & Whiteson, S. (2017). Counterfactual multi-agent policy gradients. arXiv preprint arXiv:1705.08926 Foerster, J. N., Song, F., Hughes, E., Burch, N., Dunning, I., Whiteson, S., Botvinick, M., & Bowling, M. (2018). Bayesian action decoder for deep multi-agent reinforcement learning. arXiv preprint arXiv:1811.01458 Fox, D., Burgard, W., Kruppa, H., & Thrun, S. (2000). A probabilistic approach to collaborative multi-robot localization. Autonomous Robots, 8(3), 325–344. French, A., Macedo, M., Poulsen, J., Waterson, T., & Yu, A. (2008). Multivariate analysis of variance (manova). San Francisco State University Fusaroli, R., Bahrami, B., Olsen, K., Roepstorff, A., Rees, G., Frith, C., et al. (2012). Coming to terms: Quantifying the benefits of linguistic coordination. Psychological Science, 23(8), 931–939. Galantucci, B. (2005). An experimental study of the emergence of human communication systems. Cognitive Science, 29(5), 737–767. Garrod, S., Fay, N., Rogers, S., Walker, B., & Swoboda, N. (2010). Can iterated learning explain the emergence of graphical symbols? Interaction Studies, 11(1), 33–50. Guestrin, C., Lagoudakis, M., & Parr, R. (2002). Coordinated reinforcement learning. ICML, Citeseer, 2, 227–234. Gupta, J. K., Egorov, M., & Kochenderfer, M. (2017). Cooperative multi-agent control using deep reinforcement learning. In: International conference on autonomous agents and multiagent systems (pp. 66–83). Springer Hernandez-Leal, P., Kaisers, M., Baarslag, T., de Cote, E. M. (2017). A survey of learning in multiagent environments: Dealing with non-stationarity. arXiv preprint arXiv:1707.09183 Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 9(8), 1735–1780. Iqbal, S., & Sha, F. (2019). Actor-attention-critic for multi-agent reinforcement learning. ICML Itō, T., Zhang, M., Robu, V., Fatima, S., Matsuo, T., & Yamaki, H. (2011). Innovations in agent-based complex automated negotiations. Berlin: Springer. Jang, E., Gu, S., & Poole, B. (2016). Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.01144 Jarrassé, N., Charalambous, T., & Burdet, E. (2012). A framework to describe, analyze and generate interactive motor behaviors. PloS One, 7(11), e49945. Jiang, J., & Lu, Z. (2018). Learning attentional communication for multi-agent cooperation. arXiv preprint arXiv:1805.07733 Kearns, M. (2012). Experiments in social computation. Communications of the ACM, 55(10), 56–67. Kim, D., Moon, S., Hostallero, D., Kang, W. J., Lee, T., Son, K., & Yi, Y. (2019). Learning to schedule communication in multi-agent reinforcement learning. ICLR Kingma, D. P., & Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 Kong, X., Xin, B., Liu, F., & Wang, Y. (2017). Revisiting the master-slave architecture in multi-agent deep reinforcement learning. arXiv preprint arXiv:1712.07305 Kraemer, L., & Banerjee, B. (2016). Multi-agent reinforcement learning as a rehearsal for decentralized planning. Neurocomputing, 190, 82–94. Lasswell, H. D. (1948). The structure and function of communication in society. The Communication of Ideas, 37(1), 136–39. Lauer, M., & Riedmiller, M. (2000). An algorithm for distributed reinforcement learning in cooperative multi-agent systems. In: Proceedings of the seventeenth international conference on machine learning, Citeseer Laurent, G. J., Matignon, L., Fort-Piat, L., et al. (2011). The world of independent learners is not markovian. International Journal of Knowledge-based and Intelligent Engineering Systems, 15(1), 55–64. Lazaridou, A., Peysakhovich, A., Baroni, M. (2016). Multi-agent cooperation and the emergence of (natural) language. arXiv preprint arXiv:1612.07182 LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. Nature, 521(7553), 436. Li, Y. (2017). Deep reinforcement learning: An overview. arXiv preprint arXiv:1701.07274 Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., & Wierstra, D. (2015). Continuous control with deep reinforcement learning. CoRR abs/1509.02971 Littman, M. L. (1994). Markov games as a framework for multi-agent reinforcement learning. In: Machine learning proceedings 1994 (pp. 157–163). Elsevier Lowe, R., Wu, Y., Tamar, A., Harb, J., Abbeel, O. P., & Mordatch, I. (2017). Multi-agent actor-critic for mixed cooperative-competitive environments. In: Advances in neural information processing systems (pp. 6379–6390) Matignon, L., Laurent, G., & Le Fort-Piat, N. (2007). Hysteretic q-learning: an algorithm for decentralized reinforcement learning in cooperative multi-agent teams. In: IEEE/RSJ international conference on intelligent robots and systems (pp. 157–163) IROS’07.x Miller, J. H., & Moser, S. (2004). Communication and coordination. Complexity, 9(5), 31–40. Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., & Riedmiller, M. (2013). Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., et al. (2015). Human-level control through deep reinforcement learning. Nature, 518(7540), 529. Mordatch, I., & Abbeel, P. (2017). Emergence of grounded compositional language in multi-agent populations. arXiv preprint arXiv:1703.04908 Olfati-Saber, R., Fax, J. A., & Murray, R. M. (2007). Consensus and cooperation in networked multi-agent systems. Proceedings of the IEEE, 95(1), 215–233. Oliehoek, F. A., & Vlassis, N. (2007). Q-value functions for decentralized pomdps. In: Proceedings of the 6th international joint conference on autonomous agents and multiagent systems. ACM Ono, N., & Fukumoto, K. (1996). Multi-agent reinforcement learning: A modular approach. In: Second international conference on multiagent systems (pp. 252–258). Panait, L., & Luke, S. (2005). Cooperative multi-agent learning: The state of the art. Autonomous Agents and Multi-agent Systems, 11(3), 387–434. Parker, D. C., Manson, S. M., Janssen, M. A., Hoffmann, M. J., & Deadman, P. (2003). Multi-agent systems for the simulation of land-use and land-cover change: A review. Annals of the Association of American Geographers, 93(2), 314–337. Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., & Lerer, A. (2017). Automatic differentiation in pytorch Peng, P., Yuan, Q., Wen, Y., Yang, Y., Tang, Z., Long, H., & Wang, J. (2017). Multiagent bidirectionally-coordinated nets for learning to play starcraft combat games. arXiv preprint arXiv:1703.10069 Peshkin, L., Kim, K. E., Meuleau, N., & Kaelbling, L. P. (2000). Learning to cooperate via policy search. In: Proceedings of the sixteenth conference on uncertainty in artificial intelligence (pp. 489–496). Morgan Kaufmann Publishers Inc. Petrillo, A., Salvi, A., Santini, S., & Valente, A. S. (2018). Adaptive multi-agents synchronization for collaborative driving of autonomous vehicles with multiple communication delays. Transportation Research Part C: Emerging Technologies, 86, 372–392. Pipattanasomporn, M., Feroze, H., & Rahman, S. (2009). Multi-agent systems in a distributed smart grid: Design and implementation. In: Power systems conference and exposition (2009). PSCE’09 (pp. 1–8). IEEE: IEEE/PES. Ren, W., & Sorensen, N. (2008). Distributed coordination architecture for multi-robot formation control. Robotics and Autonomous Systems, 56(4), 324–333. Scardovi, L., & Sepulchre, R. (2008). Synchronization in networks of identical linear systems. In: 47th IEEE conference on decision and control, 2008. CDC 2008 (pp. 546–551). IEEE Schmidhuber , J. (1996). A general method for multi-agent reinforcement learning in unrestricted environments. In: Adaptation, coevolution and learning in multiagent systems: papers from the 1996 AAAI spring symposium (pp. 84–87) Schmidhuber, J. (2015). Deep learning in neural networks: An overview. Neural Networks, 61, 85–117. Schulman, J., Levine, S., Abbeel, P., Jordan, M., & Moritz, P. (2015). Trust region policy optimization. In: International conference on machine learning (pp. 1889–1897) Selten, R., & Warglien, M. (2007). The emergence of simple languages in an experimental coordination game. Proceedings of the National Academy of Sciences, 104(18), 7361–7366. Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D., & Riedmiller, M. (2014). Deterministic policy gradient algorithms. In: ICML Singh, A., Jain, T., & Sukhbaatar, S. (2019). Learning when to communicate at scale in multiagent cooperative and competitive tasks. ICLR Singh, S. P., Jaakkola, T., & Jordan, M. I.(1994). Learning without state-estimation in partially observable markovian decision processes. In: Proceedings of machine learning 1994 (pp. 284–292). Elsevier Stone, P., & Veloso, M. (1998). Towards collaborative and adversarial learning: A case study in robotic soccer. International Journal of Human-Computer Studies, 48(1), 83–104. Sukhbaatar, S., & Fergus, R., et al. (2016). Learning multiagent communication with backpropagation. In: Advances in neural information processing systems (pp. 2244–2252) Sutton, R. S., & Barto, A. G. (1998). Introduction to reinforcement learning (Vol. 135). Cambridge: MIT Press. Számadó, S. (2010). Pre-hunt communication provides context for the evolution of early human language. Biological Theory, 5(4), 366–382. Tampuu, A., Matiisen, T., Kodelja, D., Kuzovkin, I., Korjus, K., Aru, J., et al. (2017). Multiagent cooperation and competition with deep reinforcement learning. PloS One, 12(4), e0172395. Tan, M. (1993). Multi-agent reinforcement learning: Independent vs. cooperative agents. In: Proceedings of the tenth international conference on machine learning (pp. 330–337) Theisen, C. A., Oberlander, J., & Kirby, S. (2010). Systematicity and arbitrariness in novel communication systems. Interaction Studies, 11(1), 14–32. Tuyls, K., & Weiss, G. (2012). Multiagent learning: Basics, challenges, and prospects. Ai Magazine, 33(3), 41. Uhlenbeck, G. E., & Ornstein, L. S. (1930). On the theory of the brownian motion. Physical Review, 36(5), 823. Van Rossum, G., & Drake, F. L, Jr. (1995). Python Tutorial. The Netherlands: Centrum voor Wiskunde en Informatica Amsterdam. Vorobeychik, Y., Joveski, Z., & Yu, S. (2017). Does communication help people coordinate? PloS One, 12(2), e0170780. Wen, G., Duan, Z., Yu, W., & Chen, G. (2012). Consensus in multi-agent systems with communication constraints. International Journal of Robust and Nonlinear Control, 22(2), 170–182. Wen, Y., Yang, Y., Luo, R., Wang, J., & Pan, W. (2019). Probabilistic recursive reasoning for multi-agent reinforcement learning. arXiv preprint arXiv:1901.09207 Wharton, T. (2003). Natural pragmatics and natural codes. Mind & Language, 18(5), 447–477. Wunder, M., Littman, M., & Stone, M. (2009). Communication, credibility and negotiation using a cognitive hierarchy model. In: Workshop# 19: MSDM 2009, p 73 You, K., & Xie, L. (2011). Network topology and communication data rate for consensusability of discrete-time multi-agent systems. IEEE Transactions on Automatic Control, 56(10), 2262.