Identification of Multilingual Offense and Troll from Social Media Memes Using Weighted Ensemble of Multimodal Features

Agrawal, 2015, Vqa: Visual question answering, Int. J. Comput. Vision, 123, 4, 10.1007/s11263-016-0966-6

Andrew, J.J., 2021. JudithJeyafreedaAndrew@DravidianLangTech-EACL2021: offensive language detection for Dravidian code-mixed YouTube comments. In: Proceedings of the First Workshop on Speech and Language Technologies for Dravidian Languages, Association for Computational Linguistics, Kyiv, pp. 169–174. URL:/ https://www.aclweb.org/anthology/2021.dravidianlangtech-1.22

Aroyehun, S.T., Gelbukh, A., 2018. Aggression detection in social media: using deep neural networks, data augmentation, and pseudo labeling. In: Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC-2018). Association for Computational Linguistics, Santa Fe, New Mexico, USA, pp. 90–97. URL:/ https://www.aclweb.org/anthology/W18-4411

Bahdanau, D., Cho, K., Bengio, Y., 2015. Neural machine translation by jointly learning to align and translate. In: 3rd International Conference on Learning Representations, ICLR 2015.

Bhardwaj, M., Akhtar, M.S., Ekbal, A., Das, A., Chakraborty, T., 2020. Hostility detection dataset in hindi. arXiv:2011.03588.

Bonheme, L., Grzes, M., 2020. SESAM at SemEval-2020 task 8: investigating the relationship between image and text in sentiment analysis of memes. In: Proceedings of the Fourteenth Workshop on Semantic Evaluation, International Committee for Computational Linguistics, Barcelona (online), pp. 804–816. URL:/ https://aclanthology.org/2020.semeval-1.102

Chakravarthi, B.R., Priyadharshini, R., Jose, N., Kumar, A.M., Mandl, T., Kumaresan, P.K., Ponnusamy, R., Hariharan, McCrae, J., Sherly, E., 2021. Findings of the shared task on offensive language identification in Tamil, Malayalam, and Kannada. In: Proceedings of the First Workshop on Speech and Language Technologies for Dravidian Languages, Association for Computational Linguistics, Kyiv, pp. 133–145. URL:/ https://www.aclweb.org/anthology/2021.dravidianlangtech-1.17

Chollet, 2017, Xception: Deep learning with depthwise separable convolutions, 1800

Conneau, 2020, Unsupervised cross-lingual representation learning at scale, ACL

Connie, 2017, Smart content recognition from images using a mixture of convolutional neural networks, Lect. Notes Electr. Eng., 11

Das, 2020, Detecting hate speech in multi-modal memes, CoRR abs/2012.14891

Davidson, T., Warmsley, D., Macy, M., Weber, I., May 2017. Automated hate speech detection and the problem of offensive language. In: Proceedings of the International AAAI Conference on Web and Social Media 11 (1). URL:/ https://ojs.aaai.org/index.php/ICWSM/article/view/14955

Gandhi, S., Kokkula, S., Chaudhuri, A., Magnani, A., Stanley, T., Ahmadi, B., Kandaswamy, V., Ovenc, O., Mannor, S., 2019. Image matters: detecting offensive and non-compliant content/ logo in product images, CoRR abs/1905.02234. arXiv:1905.02234. URL:/ http://arxiv.org/abs/1905.02234.

Gomez, R., Gibert, J., Gomez, L., Karatzas, D., 2020. Exploring hate speech detection in multimodal publications. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV).

He, 2016, Deep residual learning for image recognition, 770

Hegde, S.U., Hande, A., Priyadharshini, R., Thavareesan, S., Chakravarthi, B.R., 2021. UVCE-IIITT@DravidianLangTech-EACL2021: Tamil troll meme classification: You need to pay more attention. In: Proceedings of the First Workshop on Speech and Language Technologies for Dravidian Languages, Association for Computational Linguistics, Kyiv. pp. 180–186. URL: https://www.aclweb.org/anthology/2021.dravidianlangtech-1.24.

Hossain, E., Sharif, O., Hoque, M., Sarker, I.H., 2020. Sentilstm: A deep learning approach for sentiment analysis of restaurant reviews. In: HIS.

Hossain, E., Sharif, O., Hoque, M.M., 2021. NLP-CUET@DravidianLangTech-EACL2021: investigating visual and textual features to identify trolls from multimodal social media memes. In: Proceedings of the First Workshop on Speech and Language Technologies for Dravidian Languages, Association for Computational Linguistics, Kyiv, pp. 300–306. URL:/ https://www.aclweb.org/anthology/2021.dravidianlangtech-1.43

Hossain, E., Sharif, O., Hoque, M.M., 2021. NLP-CUET@LT-EDI-EACL2021: Multilingual code-mixed hope speech detection using cross-lingual representation learner. In: Proceedings of the First Workshop on Language Technology for Equality, Diversity and Inclusion, Association for Computational Linguistics, Kyiv. pp. 168–174. URL: https://aclanthology.org/2021.ltedi-1.25.

Hosseinmardi, 2016, Prediction of cyberbullying incidents in a media-based social network, 2016 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM), 186

Huang, 2019, Attention on attention for image captioning, 4633

Hudson, D.A., Manning, C.D., 2019. Gqa: a new dataset for real-world visual reasoning and compositional question answering. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

Illendula, 2019, Multimodal emotion classification

Kiela, D., Firooz, H., Mohan, A., Goswami, V., Singh, A., Ringshia, P., Testuggine, D., 2020. The hateful memes challenge: detecting hate speech in multimodal memes. In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M.F., Lin, H. (Eds.), Advances in Neural Information Processing Systems, vol. 33, Curran Associates Inc, pp. 2611–2624. URL: https://proceedings.neurips.cc/paper/2020/file/1b84c4cee2b8b3d823b30e2d604b1878-Paper.pdf.

Kumar, R., Ojha, A.K., Malmasi, S., Zampieri, M., 2020. Evaluating aggression identification in social media. In: Proceedings of the Second Workshop on Trolling, Aggression and Cyberbullying, European Language Resources Association (ELRA), Marseille, France, pp. 1–5. URL:/ https://www.aclweb.org/anthology/2020.trac-1.1

Kumari, K., Singh, J.P., Dwivedi, Y.K., Rana, N.P., 2020. Towards cyberbullying-free social media in smart cities: a unified multi-modal approach. Soft Comput. 24(15), 11059–11070. doi:10.1007/s00500-019-04550-x.

Li, 2016, Efficient hyperparameter optimization and infinitely many armed bandits, CoRR abs/1603.06560

Li, L.H., Yatskar, M., Yin, D., Hsieh, C., Chang, K., 2019. Visualbert: a simple and performant baseline for vision and language, CoRR abs/1908.03557. arXiv:1908.03557.

Lippe, P., Holla, N., Chandra, S., Rajamanickam, S., Antoniou, G., Shutova, E., Yannakoudakis, H., 2020. A multimodal framework for the detection of hateful memes, CoRR abs/2012.12871. arXiv:2012.12871. URL:/ https://arxiv.org/abs/2012.12871

Liu, 2020, Very deep transformers for neural machine translation, ArXiv abs/2008.07772

Lukovnikov, 2019, Pretrained transformers for simple question answering over knowledge graphs, International Semantic Web Conference, Springer, 470

Manoj, B. Chinmaya, 2021. TrollMeta@DravidianLangTech-EACL2021: meme classification using deep learning. In: Proceedings of the First Workshop on Speech and Language Technologies for Dravidian Languages, Association for Computational Linguistics, Kyiv, pp. 277–280. URL:/ https://www.aclweb.org/anthology/2021.dravidianlangtech-1.39.

Mishra, P., Yannakoudakis, H., Shutova, E., 2019. Tackling online abuse: a survey of automated abuse detection methods, CoRR abs/1908.06024. arXiv:1908.06024. URL:/ http://arxiv.org/abs/1908.06024

Mishra, S., Prasad, S., Mishra, S., 2020. Multilingual joint fine-tuning of transformer models for identifying trolling, aggression and cyberbullying at TRAC 2020. In: Proceedings of the Second Workshop on Trolling, Aggression and Cyberbullying, European Language Resources Association (ELRA), Marseille, France. pp. 120–125. URL: https://www.aclweb.org/anthology/2020.trac-1.19.

Mojica de la Vega, L.G., Ng, V., 2018. Modeling trolling in social media conversations. In: Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), European Language Resources Association (ELRA), Miyazaki, Japan. URL:/ https://www.aclweb.org/anthology/L18-1585.

Morency, L.-P., Baltrušaitis, T., 2017. Multimodal machine learning: integrating language, vision and speech. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts, Association for Computational Linguistics, Vancouver, Canada, pp. 3–5. URL:/ https://www.aclweb.org/anthology/P17-5002

Morishita, T., Morio, G., Horiguchi, S., Ozaki, H., Miyoshi, T., 2020. Hitachi at SemEval-2020 task 8: simple but effective modality ensemble for meme emotion recognition. In: Proceedings of the Fourteenth Workshop on Semantic Evaluation, International Committee for Computational Linguistics, Barcelona (online), pp. 1126–1134. URL:/ https://aclanthology.org/2020.semeval-1.149.

Mut Altin, L.S., Bravo, A., Saggion, H., 2020. LaSTUS/TALN at TRAC - 2020 trolling, aggression and cyberbullying, in: Proceedings of the Second Workshop on Trolling, Aggression and Cyberbullying, European Language Resources Association (ELRA), Marseille, France, pp. 83–86. URL:/ https://www.aclweb.org/anthology/2020.trac-1.13.

Nakamura, 2020, Fakeddit: A new multimodal benchmark dataset for fine-grained fake news detection, 6149

Nojavanasghari, 2016, Deep multimodal fusion for persuasiveness prediction, 284

Pavlopoulos, 2019, ConvAI at SemEval-2019 task 6: offensive language identification and categorization with perspective and BERT, 571

Perifanos, K., Goutsos, D., 2021. Multimodal hate speech detection in greek social media. In: Preprints. URL:/ https://www.preprints.org/manuscript/202103.0390/v1

Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al., 2021. Learning transferable visual models from natural language supervision. In: International Conference on Machine Learning, PMLR. pp. 8748–8763.

Safi Samghabadi, N., Patwa, P., PYKL, S., Mukherjee, P., Das, A., Solorio, T., 2020. Aggression and misogyny detection using BERT: a multi-task approach. In: Proceedings of the Second Workshop on Trolling, Aggression and Cyberbullying, European Language Resources Association (ELRA), Marseille, France, pp. 126–131. URL:/ https://www.aclweb.org/anthology/2020.trac-1.20.

Sandulescu, 2020, Detecting hateful memes using a multimodal deep ensemble, CoRR abs/2012.13235

Sanh, V., Debut, L., Chaumond, J., Wolf, T., 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. ArXiv abs/1910.01108.

Sharif, 2021, Identification and classification of textual aggression in social media: Resource creation and evaluation, 1

Sharif, O., Hossain, E., Hoque, M.M., 2021. NLP-CUET@DravidianLangTech-EACL2021: offensive language detection from multilingual code-mixed text using transformers. In: Proceedings of the First Workshop on Speech and Language Technologies for Dravidian Languages, Association for Computational Linguistics, Kyiv, pp. 255–261. URL:/ https://www.aclweb.org/anthology/2021.dravidianlangtech-1.35

Sharma, C., Bhageria, D., Scott, W., PYKL, S., Das, A., Chakraborty, T., Pulabaigari, V., Gambäck, B., 2020. SemEval-2020 task 8: memotion analysis- the visuo-lingual metaphor!. In: Proceedings of the Fourteenth Workshop on Semantic Evaluation, International Committee for Computational Linguistics, Barcelona (online), pp. 759–773. URL:/ https://aclanthology.org/2020.semeval-1.99

Simonyan, 2015, Very deep convolutional networks for large-scale image recognition, CoRR abs/1409.1556

Solieman, 2021, The detection of depression using multimodal models based on text and voice quality features, 1843

Song, 2021, A multimodal fake news detection model based on crossmodal attention residual and multichannel convolutional neural networks, Inf. Process. Manage., 58, 10.1016/j.ipm.2020.102437

Suhr, A., Zhou, S., Zhang, I., Bai, H., Artzi, Y., 2018. A corpus for reasoning about natural language grounded in photographs, CoRR abs/1811.00491. arXiv:1811.00491. URL:/ http://arxiv.org/abs/1811.00491

Sun, 2019, How to fine-tune BERT for text classification?, CoRR abs/1905.05583

Suryawanshi, S., Chakravarthi, B.R., 2021. Findings of the shared task on troll meme classification in Tamil. In: Proceedings of the First Workshop on Speech and Language Technologies for Dravidian Languages, Association for Computational Linguistics, Kyiv. pp. 126–132. URL: https://www.aclweb.org/anthology/2021.dravidianlangtech-1.16.

Suryawanshi, S., Chakravarthi, B.R., Arcan, M., Buitelaar, P., 2020. Multimodal meme dataset (MultiOFF) for identifying offensive content in image and text. In: Proceedings of the Second Workshop on Trolling, Aggression and Cyberbullying, European Language Resources Association (ELRA), Marseille, France, pp. 32–41. URL:/ https://www.aclweb.org/anthology/2020.trac-1.6.

Szegedy, 2015, Rethinking the inception architecture for computer vision, CoRR abs/1512.00567

Tulkens, S., Hilte, L., Lodewyckx, E., Verhoeven, B., Daelemans, W., 2016. A dictionary-based approach to racism detection in dutch social media, CoRR abs/1608.08738. arXiv:1608.08738. URL:/ http://arxiv.org/abs/1608.08738

Vaswani, 2017, Attention is all you need, 6000

Velioglu, R., Rose, J., 2020. Detecting hate speech in memes using multimodal deep learning approaches: Prize-winning solution to hateful memes challenge. CoRR abs/2012.12975. arXiv:2012.12975.

Wang, S., Liu, J., Ouyang, X., Sun, Y., 2020. Galileo at SemEval-2020 task 12: Multi-lingual learning for offensive language identification using pre-trained language models. In: Proceedings of the Fourteenth Workshop on Semantic Evaluation, International Committee for Computational Linguistics, Barcelona (online), pp. 1448–1455. URL:/ https://www.aclweb.org/anthology/2020.semeval-1.189

Williams, 2016, Racial microaggressions and perceptions of internet memes, Comput. Hum. Behav., 63, 424, 10.1016/j.chb.2016.05.067

Xue, 2021, Detecting fake news by exploring the consistency of multimodal data, Inf. Process. Manage., 58, 10.1016/j.ipm.2021.102610

Zhang, 2020, Hateful memes detection via complementary visual and linguistic networks, CoRR abs/2012.04977

Zhou, X., Sap, M., Swayamdipta, S., Smith, N.A., Choi, Y., 2021. Challenges in automated debiasing for toxic language detection. arXiv:2102.00086.