Personalized federated learning based on multi-head attention algorithm
Tóm tắt
Federated Learning (FL) is an algorithm for the encrypted exchange of model parameters while ensuring the independence of participants. Classic federated learning does not take into account the correlation between features, nor does it take into account the data differences caused by the reasonable personalization of each client. Therefore, this paper proposes a personalized federated learning algorithm based on a multi-head attention mechanism. First, in order to improve the personalization of local models, attention mechanism is used to capture the relevance of local features. Then, when aggregating local models, the weight
$$\lambda$$
is generated for local models based on the differences between models, and finally aggregate them into a new global model. Finally, the multi-head attention is proposed to calculate the importance score of the global model parameters on the current local model, and assign it to the local model as the attention coefficient, so as to realize personalized federated learning. Through experiments on MNIST, SVHN and STL10 datasets, the validity of Personalized Federated Learning is verified, and the rationality of hyperparameter setting is discussed through visualizing results.
Tài liệu tham khảo
Honghao Gao, Wanqiu Huang, Tong Liu, Yuyu Yin, Youhuizi Li (2022) Ppo2: Location privacy-oriented task offloading to edge computing using reinforcement learning for intelligent autonomous transport systems. IEEE Trans Transp Syst. https://doi.org/10.1109/TITS.2022.3169421
Xiao Junsheng, Huahu Xu, Gao Honghao, Bian Minjie, Li Yang (2021) A weakly supervised semantic segmentation network by aggregating seed cues: the multi-object proposal generation perspective. ACM Trans Multimed Comput Commun Appl 17:15
Honghao Gao, Binyang Qiu, Barroso Ramon J, Duran Hussain Walayat, Yueshen Xu, Xinheng Wang (2022) Tsmae: a novel anomaly detection approach for internet of things time series data using memory-augmented autoencoder. IEEE Trans Netw Sci Eng. https://doi.org/10.1109/TNSE.2022.3163144
Brendan Mcmahan H, Eider Moore, Daniel Ramage, Seth Hampson, Blaise Arcas (2016) Communication-efficient learning of deep networks from decentralized data. PMLR 54:1273–1282
Mcmahan H Brendan, Moore Eider, Ramage Daniel, Arcas Blaise (2017) Federated learning of deep networks using model averaging. arXiv preprint arXiv:1602.05629
Ke Guolin, Meng Qi, Thomas Finley (2017) A highly efficient gradient boosting decision tree. NIPS 30:3149–3157
Chen Lu, Xia Min, Lin Haifeng (2022) Multi-scale strip pooling feature aggregation network for cloud and cloud shadow segmentation. Neural Comput Appl 34:6149–6162
T Xu M (2021) Research on long and short-term neural network recommendation model based on self-attention mechanism
Zhu Qiannan, Zhou Xiaofei, Song Zeliang, Tan Jianlong, Guo Li (2019) Dan: deep attention neural network for news recommendation. AAAI 33:5973–5980
An Mingxiao Wu, Chuhan Fangzhao, Wu, Kun Zhang, Zheng Liu, Xing Xie (2019) Neural news recommendation with long-and short-term user representations. ACL 57:336–345
Smith Virginia, Chiang Chaokai, Sanjabi Maziar, Talwalkar Ameet (2017) Federated multi-task learning. NIPS:4427–4437
Liu Yang, Chen Tianjian, Yang Qiang (2018) Secure federated transfer learning. arXiv preprintarXiv:1812.03337
Kewei Cheng, Tao Fan, Yilun Jin, Yang Liu, Tianjian Chen, Qiang Yang (2019) SecureBoost: A lossless federated learning framework. IEEE Intell Sys 36:7–98
Deng Yuyang, Kamani, Mohammad Mahdi, Mahdavi Mehrdad (2020) Adaptive personalized federated learning. arXiv preprint arXiv:2003.13461
Yang Qiang, Liu Yang, Cheng Yong (2019) Federated machine learning: concept and applications. ACM 10:1–19
Zhuo Hankz Hankui, Feng Wenfeng, Lin Yufeng, Xu Qian, Yang Qiang (2019) Federated deep reinforcement learning. arXiv preprintarXiv:1812.03337
Arivazhagan Manoj Ghuhan, Aggarwal Vinay, Singh Aaditya Kumar, Choudhary Sunav (2019) Federated learning with personalization layers. arXiv preprint arXiv:1901.08277v1
Jiang Yihan, Konečný Jakub, Rush Keith, Kannan Sreeram (2019) Improving federated learning personalization via model agnostic meta learning. arXiv preprint arXiv:1909.12488
Dinh Canh T, TranNguyen H, Nguyen Tuan Dung (2020) Personalized federated learning with moreau envelopes. arXiv preprintarXiv:2006.08848
Fallah Alireza, Mokhtari Aryan, Ozdaglar Asuman (2020) Personalized federated learning: a meta-learning approach. arXiv preprint arXiv:2002.07948
Nichol Alex, Achiam Joshua, Schulman John (2018) On first-order meta-learning algorithms. arXiv preprint arXiv:1803.02999
Wang Jialei, Kolar Mladen, Srerbo Nathan (2016) Distributed multi-task learning. Artif Intell Stat 51:751–760
Tan Alysa Ziying, Yu Han, Cui Lizhen, Yang Qiang (2021) Towards personalized federated learning. arXiv preprintarXiv:2103.00710
Liu Yang, Yang Qiang, Chen Tianjian (2019) Tutorial on federated learning and transfer learning for privacy, security and confidentiality.AAAI’19
Kairouz Peter, McMahan H. Brendan, Avent Brendan, Bellet Aurélien, Bennis Mehdi, Bhagoji Arjun Nitin, Bonawitz Kallista, et al Charles Zachary (2019) Advances and open problems in federated learning
Lecun Yann, Bottou Leon (1998) Gradient-based learning applied to document recognition. Proc IEEE 86:2278–2324
Song Lei, Xia Min, Weng Liguo, Lin Haifeng, Qian Min, Chen Bingyu (2023) Axial cross attention meets cnn: bibranch fusion network for change detection. IEEE J Sel Top Appl Earth Observ Remote Sens 16:32–43
Hao Yu, Yang Sen (2018) and Shenghuo Zhu. Demystifying why model averaging works for deep learning, parallel restarted sgd with faster convergence and less communication
Cho Kyunghyun, van Merrienboer Bart, Bahdanau Dzmitry, Yoshua Bengio (2014) Encoder-decoder approaches on the properties of neural machine translation. arXiv preprint arXiv:1409.1259
Krizhevsky Alex (2009) Learning multiple layers of features from tiny images. Tech Report. 1–60
Miao Shoukuan, Xia Min, Qian Ming, Zhang Yonghong, Liu Jia, Lin Haifeng (2022) Cloud/shadow segmentation based on multi-level feature enhanced network for remote sensing imagery. Int J Remote Sens 43(15–16):5940–5960
Yi Qu, Xia Min, Zhang Yonghong (2021) Strip pooling channel spatial attention network for the segmentation of cloud and cloud shadow. Comput Geosci 157:104940
Chen Bingyu, Xia Min, Qian Ming, Huang Junqing (2022) Manet: a multi-level aggregation network for semantic segmentation of high-resolution remote sensing images. Int J Remote Sens 43(15–16):5874–5894
Min Xia Xu, Zhang Wan’an Liu, Weng Liguo, Yiqing Xu (2020) Multi-stage feature constraints learning for age estimation. IEEE Trans Inf Forensics Secur 15:2417–2428
Wang Zhiwei, Xia Min, Min Lu, Pan Lingling, Liu Jun (2022) Parameter identification in power transmission systems based on graph convolution network. IEEE Trans Power Deliv 37(4):3155–3163
Liu Jingjing, Liu Yefeng, Zhang Qichun (2022) A weight initialization method based on neural network with asymmetric activation function. Neurocomputing 483:171–182
Gao Jiahong, Weng Liguo, Xia Min, Lin Haifeng (2022) MLNet: multichannel feature fusion lozenge network for land segmentation. J Appl Remote Sens 16(1):1–19
Gao Liang, Fu Huazhu, Li Li, Chen Yingwen, Xu Ming, Xu Cheng-Zhong (2022) Feddc: Federated learning with non-iid data via local drift decoupling and correction. IEEE/CVF Conference on Computer Vision and Pattern Recognition, page 22094283
Reddi Sashank, Charles Zachary, Zaheer Manzil, Garrett Zachary, Rush Keith, Konecny Jakub, Kumar Sanjiv, McMahan H Brendan (2021) Adaptive federated optimization. International Conference on Representation Learning, page 13
Wei Zhao, Benyou Wang, Min Yang, Jianbo Ye, Zhou Zhao, Xiaojun Chen (2019) Leveraging long and short-term information in content-aware movie recommendation via adversarial training. IEEE Trans Cybern 50:4680–4693
Robin C (2017) Geyer, Tassilo Klein, and Moin Nabi. A client level perspective. NIPS, differentially private federated learning
Mukund Deshpande, George Karypis (2004) Item-based top-n recommendation algorithms. ACM Trans Inf Syst 22:143–147