Personalized federated learning based on multi-head attention algorithm

Shanshan Jiang1, Meixia Lu2, Kai Hu2, Jiasheng Wu2, Yaogen Li2, Liguo Weng2, Min Xia2, Haifeng Lin3
1School of Management Science and Engineering, Nanjing University of Information Science and Technology, Nanjing, China
2Jiangsu Provincial Collaborative Innovation Center for Atmospheric Environment and Equipment Technology, Nanjing University of Information Science and Technology, Nanjing, China
3College of Information Science and Technology, Nanjing Forestry University, Nanjing, China

Tóm tắt

Federated Learning (FL) is an algorithm for the encrypted exchange of model parameters while ensuring the independence of participants. Classic federated learning does not take into account the correlation between features, nor does it take into account the data differences caused by the reasonable personalization of each client. Therefore, this paper proposes a personalized federated learning algorithm based on a multi-head attention mechanism. First, in order to improve the personalization of local models, attention mechanism is used to capture the relevance of local features. Then, when aggregating local models, the weight $$\lambda$$ is generated for local models based on the differences between models, and finally aggregate them into a new global model. Finally, the multi-head attention is proposed to calculate the importance score of the global model parameters on the current local model, and assign it to the local model as the attention coefficient, so as to realize personalized federated learning. Through experiments on MNIST, SVHN and STL10 datasets, the validity of Personalized Federated Learning is verified, and the rationality of hyperparameter setting is discussed through visualizing results.

Tài liệu tham khảo

Honghao Gao, Wanqiu Huang, Tong Liu, Yuyu Yin, Youhuizi Li (2022) Ppo2: Location privacy-oriented task offloading to edge computing using reinforcement learning for intelligent autonomous transport systems. IEEE Trans Transp Syst. https://doi.org/10.1109/TITS.2022.3169421 Xiao Junsheng, Huahu Xu, Gao Honghao, Bian Minjie, Li Yang (2021) A weakly supervised semantic segmentation network by aggregating seed cues: the multi-object proposal generation perspective. ACM Trans Multimed Comput Commun Appl 17:15 Honghao Gao, Binyang Qiu, Barroso Ramon J, Duran Hussain Walayat, Yueshen Xu, Xinheng Wang (2022) Tsmae: a novel anomaly detection approach for internet of things time series data using memory-augmented autoencoder. IEEE Trans Netw Sci Eng. https://doi.org/10.1109/TNSE.2022.3163144 Brendan Mcmahan H, Eider Moore, Daniel Ramage, Seth Hampson, Blaise Arcas (2016) Communication-efficient learning of deep networks from decentralized data. PMLR 54:1273–1282 Mcmahan H Brendan, Moore Eider, Ramage Daniel, Arcas Blaise (2017) Federated learning of deep networks using model averaging. arXiv preprint arXiv:1602.05629 Ke Guolin, Meng Qi, Thomas Finley (2017) A highly efficient gradient boosting decision tree. NIPS 30:3149–3157 Chen Lu, Xia Min, Lin Haifeng (2022) Multi-scale strip pooling feature aggregation network for cloud and cloud shadow segmentation. Neural Comput Appl 34:6149–6162 T Xu M (2021) Research on long and short-term neural network recommendation model based on self-attention mechanism Zhu Qiannan, Zhou Xiaofei, Song Zeliang, Tan Jianlong, Guo Li (2019) Dan: deep attention neural network for news recommendation. AAAI 33:5973–5980 An Mingxiao Wu, Chuhan Fangzhao, Wu, Kun Zhang, Zheng Liu, Xing Xie (2019) Neural news recommendation with long-and short-term user representations. ACL 57:336–345 Smith Virginia, Chiang Chaokai, Sanjabi Maziar, Talwalkar Ameet (2017) Federated multi-task learning. NIPS:4427–4437 Liu Yang, Chen Tianjian, Yang Qiang (2018) Secure federated transfer learning. arXiv preprintarXiv:1812.03337 Kewei Cheng, Tao Fan, Yilun Jin, Yang Liu, Tianjian Chen, Qiang Yang (2019) SecureBoost: A lossless federated learning framework. IEEE Intell Sys 36:7–98 Deng Yuyang, Kamani, Mohammad Mahdi, Mahdavi Mehrdad (2020) Adaptive personalized federated learning. arXiv preprint arXiv:2003.13461 Yang Qiang, Liu Yang, Cheng Yong (2019) Federated machine learning: concept and applications. ACM 10:1–19 Zhuo Hankz Hankui, Feng Wenfeng, Lin Yufeng, Xu Qian, Yang Qiang (2019) Federated deep reinforcement learning. arXiv preprintarXiv:1812.03337 Arivazhagan Manoj Ghuhan, Aggarwal Vinay, Singh Aaditya Kumar, Choudhary Sunav (2019) Federated learning with personalization layers. arXiv preprint arXiv:1901.08277v1 Jiang Yihan, Konečný Jakub, Rush Keith, Kannan Sreeram (2019) Improving federated learning personalization via model agnostic meta learning. arXiv preprint arXiv:1909.12488 Dinh Canh T, TranNguyen H, Nguyen Tuan Dung (2020) Personalized federated learning with moreau envelopes. arXiv preprintarXiv:2006.08848 Fallah Alireza, Mokhtari Aryan, Ozdaglar Asuman (2020) Personalized federated learning: a meta-learning approach. arXiv preprint arXiv:2002.07948 Nichol Alex, Achiam Joshua, Schulman John (2018) On first-order meta-learning algorithms. arXiv preprint arXiv:1803.02999 Wang Jialei, Kolar Mladen, Srerbo Nathan (2016) Distributed multi-task learning. Artif Intell Stat 51:751–760 Tan Alysa Ziying, Yu Han, Cui Lizhen, Yang Qiang (2021) Towards personalized federated learning. arXiv preprintarXiv:2103.00710 Liu Yang, Yang Qiang, Chen Tianjian (2019) Tutorial on federated learning and transfer learning for privacy, security and confidentiality.AAAI’19 Kairouz Peter, McMahan H. Brendan, Avent Brendan, Bellet Aurélien, Bennis Mehdi, Bhagoji Arjun Nitin, Bonawitz Kallista, et al Charles Zachary (2019) Advances and open problems in federated learning Lecun Yann, Bottou Leon (1998) Gradient-based learning applied to document recognition. Proc IEEE 86:2278–2324 Song Lei, Xia Min, Weng Liguo, Lin Haifeng, Qian Min, Chen Bingyu (2023) Axial cross attention meets cnn: bibranch fusion network for change detection. IEEE J Sel Top Appl Earth Observ Remote Sens 16:32–43 Hao Yu, Yang Sen (2018) and Shenghuo Zhu. Demystifying why model averaging works for deep learning, parallel restarted sgd with faster convergence and less communication Cho Kyunghyun, van Merrienboer Bart, Bahdanau Dzmitry, Yoshua Bengio (2014) Encoder-decoder approaches on the properties of neural machine translation. arXiv preprint arXiv:1409.1259 Krizhevsky Alex (2009) Learning multiple layers of features from tiny images. Tech Report. 1–60 Miao Shoukuan, Xia Min, Qian Ming, Zhang Yonghong, Liu Jia, Lin Haifeng (2022) Cloud/shadow segmentation based on multi-level feature enhanced network for remote sensing imagery. Int J Remote Sens 43(15–16):5940–5960 Yi Qu, Xia Min, Zhang Yonghong (2021) Strip pooling channel spatial attention network for the segmentation of cloud and cloud shadow. Comput Geosci 157:104940 Chen Bingyu, Xia Min, Qian Ming, Huang Junqing (2022) Manet: a multi-level aggregation network for semantic segmentation of high-resolution remote sensing images. Int J Remote Sens 43(15–16):5874–5894 Min Xia Xu, Zhang Wan’an Liu, Weng Liguo, Yiqing Xu (2020) Multi-stage feature constraints learning for age estimation. IEEE Trans Inf Forensics Secur 15:2417–2428 Wang Zhiwei, Xia Min, Min Lu, Pan Lingling, Liu Jun (2022) Parameter identification in power transmission systems based on graph convolution network. IEEE Trans Power Deliv 37(4):3155–3163 Liu Jingjing, Liu Yefeng, Zhang Qichun (2022) A weight initialization method based on neural network with asymmetric activation function. Neurocomputing 483:171–182 Gao Jiahong, Weng Liguo, Xia Min, Lin Haifeng (2022) MLNet: multichannel feature fusion lozenge network for land segmentation. J Appl Remote Sens 16(1):1–19 Gao Liang, Fu Huazhu, Li Li, Chen Yingwen, Xu Ming, Xu Cheng-Zhong (2022) Feddc: Federated learning with non-iid data via local drift decoupling and correction. IEEE/CVF Conference on Computer Vision and Pattern Recognition, page 22094283 Reddi Sashank, Charles Zachary, Zaheer Manzil, Garrett Zachary, Rush Keith, Konecny Jakub, Kumar Sanjiv, McMahan H Brendan (2021) Adaptive federated optimization. International Conference on Representation Learning, page 13 Wei Zhao, Benyou Wang, Min Yang, Jianbo Ye, Zhou Zhao, Xiaojun Chen (2019) Leveraging long and short-term information in content-aware movie recommendation via adversarial training. IEEE Trans Cybern 50:4680–4693 Robin C (2017) Geyer, Tassilo Klein, and Moin Nabi. A client level perspective. NIPS, differentially private federated learning Mukund Deshpande, George Karypis (2004) Item-based top-n recommendation algorithms. ACM Trans Inf Syst 22:143–147