Multimodal analysis of user behavior and browsed content under different image search intents

Arapakis I, Konstas, I, Jose, JM (2009) Using facial expressions and peripheral physiological signals as implicit indicators of topical relevance. In: Proceedings of the 17th ACM international conference on multimedia, MM ’09. ACM, New York, pp 461–470

Bird S (2006) NLTK: the natural language toolkit. In: Proceedings of the COLING/ACL on interactive presentation sessions, pp 69–72

Blanchard N, Bixler R, Joyce T, D’Mello S (2014) Automated physiological-based detection of mind wandering during learning. In: Proceedings of ITS, Springer, Berlin, pp 55–60 (2014)

Borth D, Chen T, Ji R, Chang SF (2013) Sentibank: large-scale ontology and classifiers for detecting sentiment and emotions in visual content. In: Proceedings of the 21st ACM International Conference on Multimedia, MM ’13. ACM, New York, pp 459–460

Bradley MM, Miccoli L, Escrig MA, Lang PJ (2008) The pupil as a measure of emotional arousal and autonomic activation. Psychophysiology 45(4):602–607

Breiman L (2001) Random forests. Mach Learn 45(1):5–32

Broder A (2002) A taxonomy of web search. SIGIR Forum 36(2):3–10

Calvo RA, D’Mello S (2010) Affect detection: an interdisciplinary review of models, methods, and their applications. IEEE Trans Affect Comput 1(1):18–37

Chatzichristofis S, Boutalis Y, Lux M (2009) Selection of the proper compact composite descriptor for improving content based image retrieval. In: Proceedings of IASTED Int’l conference on signal processing, pattern recognition and applications, vol 134643, p 64

Cortes C, Vapnik V (1995) Support vector machine. Mach Learn 20(3):273–297

Deng J, Dong W, Socher R, Li LJ, Li K, Fei-Fei L (2009) Imagenet: A large-scale hierarchical image database. In: IEEE conference on computer vision and pattern recognition 2009. CVPR 2009. IEEE, pp 248–255

D’Mello S, Olney A, Williams C, Hays P (2012) Gaze tutor: a gaze-reactive intelligent tutoring system. Int J Human Comput Stud 70(5):377–398

Ekman P, Friesen W (1978) The Facial Action Coding System (FACS). Consulting Psychologists Press, Stanford University, Palo Alto

Elsweiler D, Mandl S, Kirkegaard Lunn B (2010) Understanding casual-leisure information needs: A diary study in the context of television viewing. In: Symposium on information interaction in context, IIiX ’10. ACM, New York, pp 25–34

Fidel R (1997) The image retrieval task: implications for the design and evaluation of image databases. New Rev Hypermed Multimed 3(1):181–199

Gialampoukidis I, Moumtzidou A, Liparas D, Vrochidis S, Kompatsiaris I (2016) A hybrid graph-based and non-linear late fusion approach for multimedia retrieval. In: Proceedings of the 14th international workshop on content-based multimedia indexing (CBMI), pp 1–6

Gunatilaka AH, Baertlein BA (2001) Feature-level and decision-level fusion of noncoincidently sampled sensors for land mine detection. IEEE Trans Pattern Anal Mach Intell 23(6):577–589

Hanjalic A, Kofler C, Larson M (2012) Intent and its discontents: The user at the wheel of the online video search engine. In: Proceedings of the 20th ACM international conference on multimedia, MM’12. ACM, New York, pp 1239–1248

Haralick RM (1979) Statistical and structural approaches to texture. Proc IEEE 67(5):786–804

Howarth P, Rüger S (2004) Evaluation of texture features for content-based image retrieval. In: Enser P, Kompatsiaris Y, O’Connor NE, Smeaton AF, Smeulders AWM (eds) Image and video retrieval. Springer, Berlin, Heidelberg, pp 326–334. https://doi.org/10.1007/978-3-540-27814-6_40

Jaques N, Conati C, Harley JM, Azevedo R (2014) Predicting affect from gaze data during interaction with an intelligent tutoring system. In: Proceedings of ITS, vol 8474. LNCS, pp 29–38

Jou B, Chang SF (2016) Deep cross residual learning for multitask visual recognition. In: Proceedings of the 2016 ACM on multimedia conference, MM ’16. ACM, New York, pp 998–1007

Kofler C, Larson M, Hanjalic A (2016) User intent in multimedia search: a survey of the state of the art and future challenges. ACM Comput Surv 49(2):36:1–36:37

Kreibig SD (2010) Autonomic nervous system activity in emotion: a review. Biol Psychol 84(3):394–421

Lagger C, Lux M, Marques O (2017) What makes people watch online videos: an exploratory study. Comput Entertain 15(2):6:1–6:31. https://doi.org/10.1145/3034706

Lallé S, Conati C, Carenini G (2016) Predicting confusion in information visualization from eye tracking and interaction data. In: Proceedings of the twenty-fifth international joint conference on artificial intelligence, IJCAI’16. AAAI Press, pp 2529–2535

Lux M, Kofler C, Marques O (2010) A classification scheme for user intentions in image search. In: CHI’10 extended abstracts on human factors in computing systems, pp 3913–3918

Lux M, Taschwer M, Marques O (2012) A closer look at photographers’ intentions: a test dataset. In: Proceedings of the ACM multimedia 2012 workshop on crowdsourcing for multimedia, CrowdMM ’12. ACM, New York, pp. 17–18

Maniu S, O’Hare N, Aiello LM, Chiarandini L, Jaimes A (2013) Search behaviour on photo sharing platforms. In: 2013 IEEE international conference on multimedia and Expo (ICME). IEEE, pp 1–6

McDuff D, Mahmoud A, Mavadati M, Amr M, Turcot J, Kaliouby Re (2016) Affdex SDK: a cross-platform real-time multi-face expression recognition toolkit. In: CHI Conference extended abstracts on human factors in computing systems, pp 3723–3726

Miller GA (1995) Wordnet: a lexical database for english. Commun ACM 38(11):39–41

Moshfeghi Y, Jose JM (2013) On cognition, emotion, and interaction aspects of search tasks with different search intentions. International conference on World Wide Web–WWW ’13. ACM Press, New York, pp 931–942

O’Hare N, de Juan P, Schifanella R, He Y, Yin D, Chang Y (2016) Leveraging user interaction signals for web image search. In: Proceedings of the 39th international ACM SIGIR conference on research and development in information retrieval—SIGIR ’16. ACM Press, New York, pp 559–568

Park JY, O’Hare N, Schifanella R, Jaimes A, Chung CW (2015) A large-scale study of user image search behavior on the web. In: Proceedings of the 33rd annual ACM conference on human factors in computing systems—CHI ’15. ACM Press, New York, pp. 985–994

Poddar A, Ruthven I (2010) The emotional impact of search tasks. In: Proceedings of the 3rd symposium on information interaction in context, IIiX ’10, pp 35–44

Riegler M, Larson M, Lux M, Kofler C (2014) How ’how’ reflects what’s what: content-based exploitation of how users frame social images. In: Proceedings of the 22Nd ACM international conference on multimedia, MM ’14. ACM, New York, NY, pp 397–406

Snoek CGM, Worring M, Smeulders AWM (2005) Early versus late fusion in semantic video analysis. In: Proceedings of the 13th Annual ACM international conference on multimedia, MULTIMEDIA ’05. ACM, New York, pp 399–402

Soleymani M, Riegler M, Halvorsen P (2017) Multimodal analysis of image search intent—intent recognition in image search from user behavior and visual content. In: Proceedings of the ACM international conference on multimedia retrieval (ICMR), pp 251–259

Soleymani M, Villaro-Dixon F, Pun T, Chanel G (2017) Toolbox for emotional feature extraction from physiological signals (teap). Front ICT 4:1

Tamura H, Mori S, Yamawaki T (1978) Textural features corresponding to visual perception. IEEE Trans Syst Man Cybern 8(6):460–473

Teevan J, Adar E, Jones R, Potts MAS (2007) Information re-retrieval: repeat queries in yahoo’s logs. In: Proceedings of ACM SIGIR conference on research and development in information retrieval, SIGIR ’07, pp 151–158

Wold S, Esbensen K, Geladi P (1987) Principal component analysis. Chemom Intell Lab Syst 2(1–3):37–52