Airwriting: a wearable handwriting recognition system
Tóm tắt
We present a wearable input system which enables interaction through 3D handwriting recognition. Users can write text in the air as if they were using an imaginary blackboard. The handwriting gestures are captured wirelessly by motion sensors applying accelerometers and gyroscopes which are attached to the back of the hand. We propose a two-stage approach for spotting and recognition of handwriting gestures. The spotting stage uses a support vector machine to identify those data segments which contain handwriting. The recognition stage uses hidden Markov models (HMMs) to generate a text representation from the motion sensor data. Individual characters are modeled by HMMs and concatenated to word models. Our system can continuously recognize arbitrary sentences, based on a freely definable vocabulary. A statistical language model is used to enhance recognition performance and to restrict the search space. We show that continuous gesture recognition with inertial sensors is feasible for gesture vocabularies that are several orders of magnitude larger than traditional vocabularies for known systems. In a first experiment, we evaluate the spotting algorithm on a realistic data set including everyday activities. In a second experiment, we report the results from a nine-user experiment on handwritten sentence recognition. Finally, we evaluate the end-to-end system on a small but realistic data set.
Tài liệu tham khảo
Amft O, Amstutz R, Smailagic A, Siewiorek D, Tröster G (2009) Gesture-controlled user input to complete questionnaires on wrist-worn watches. In: Human–computer interaction. Novel interaction methods and techniques. Lecture Notes in Computer Science, vol 5611. Springer, Heidelberg, pp 131–140
Amma C, Gehrig D, Schultz T (2010) Airwriting recognition using wearable motion sensors. In: Proceedings of the 1st augmented human international conference (AH’10). doi:
10.1145/1785455.1785465
Amma C, Georgi M, Schultz T (2012) Airwriting: hands-free mobile text input by spotting and continuous recognition of 3D-space handwriting with inertial sensors. In: IEEE 16th international symposium on wearable computers (ISWC), pp 52–59
Amma C, Schultz T (2012) Airwriting: demonstrating mobile text input by 3D-space handwriting. In: Proceedings of the ACM international conference on intelligent user interfaces (IUI’12)
Bang WC, Chang W, Kang KH, Choi ES, Potanin A, Kim DY (2003) Self-contained spatial input device for wearable computers. In: Proceedings of the IEEE international symposium on wearable computers (ISWC’03)
Chen M, AlRegib G, Juang B (2012) 6D motion gesture recognition using spatio-temporal features. In: IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, pp 2341–2344
citation_journal_title=IEEE Trans Inf Theory; citation_title=Hidden markov processes; citation_author=Y Ephraim, N Merhav; citation_volume=48; citation_issue=6; citation_publication_date=2002; citation_pages=1518-1569; citation_doi=10.1109/TIT.2002.1003838; citation_id=CR7
Gustafson S, Bierwirth D, Baudisch P (2010) Imaginary interfaces: spatial interaction with empty hands and without visual feedback. In: Proceedings of the 23nd annual ACM symposium on user interface software and technology (UIST’10)
Hein A, Hoffmeyer A, Kirste T (2009) Utilizing an accelerometric bracelet for ubiquitous gesture-based interaction. In: Universal access in human–computer interaction. Intelligent and ubiquitous interaction environments. Lecture Notes in Computer Science, vol 5615. Springer, Heidelberg, pp 519–527
citation_title=Spoken language processing; citation_publication_date=2001; citation_id=CR10; citation_author=X Huang; citation_author=A Acero; citation_author=H Hon; citation_publisher=Prentice Hall
citation_journal_title=Pattern Recognit; citation_title=Gesture spotting with body-worn inertial sensors to detect user activities; citation_author=H Junker, O Amft, P Lukowicz, G Tröster; citation_volume=41; citation_issue=6; citation_publication_date=2008; citation_pages=2010-2024; citation_doi=10.1016/j.patcog.2007.11.016; citation_id=CR11
Kallio S, Kela J, Mantyjarvi J (2003) Online gesture recognition system for mobile interaction. In: Proceedings of the IEEE international conference on systems, man and cybernetics (ICSMC’03)
Kim D, Choi H, Kim J (2006) 3D space handwriting recognition with ligature model. In: Ubiquitous computing systems. Lecture Notes in Computer Science, vol 4239. Springer, Heidelberg, pp 41–56
citation_journal_title=IEEE Trans Pattern Anal Mach Intell; citation_title=An hmm-based threshold model approach for gesture recognition; citation_author=HK Lee, J Kim; citation_volume=21; citation_issue=10; citation_publication_date=1999; citation_pages=961-973; citation_doi=10.1109/34.799904; citation_id=CR14
Lyons K, Starner T, Plaisted D, Fusia J, Lyons A, Drew A, Looney EW (2004) Twiddler typing: one-handed chording text entry for mobile phones. In: Proceedings of the SIGCHI conference on human factors in computing systems (CHI’04)
MacKenzie IS, Soukoreff RW, Helga J (2011) 1 thumb, 4 buttons, 20 words per minute: design and evaluation of h4-writer. In: Proceedings of the 24th annual ACM symposium on user interface software and technology (UIST’11)
McGuire R, Hernandez-Rebollar J, Starner T, Henderson V, Brashear H, Ross D (2004) Towards a one-way american sign language translator. In: Proceedings of the sixth IEEE international conference on automatic face and gesture recognition (FGR’04)
Mistry P, Maes P, Chang L (2009) Wuw-wear ur world: a wearable gestural interface. In: Proceedings of the 27th international conference extended abstracts on human factors in computing systems (CHI EA ’09)
citation_journal_title=IEEE Trans Syst Man Cybern Part C Appl Rev; citation_title=Gesture recognition: a survey; citation_author=S Mitra, T Acharya; citation_volume=37; citation_publication_date=2007; citation_pages=311-324; citation_doi=10.1109/TSMCC.2007.893280; citation_id=CR19
Odell J, Valtchev V, Woodland P, Young S (1994) A one pass decoder design for large vocabulary recognition. In: Proceedings of the workshop on human language technology. Association for Computational Linguistics, pp 405–410
citation_journal_title=IEEE Trans Pattern Anal Mach Intell; citation_title=Online and off-line handwriting recognition: a comprehensive survey; citation_author=R Plamondon, S Srihari; citation_volume=22; citation_issue=1; citation_publication_date=2000; citation_pages=63-84; citation_doi=10.1109/34.824821; citation_id=CR21
citation_journal_title=Proc IEEE; citation_title=A tutorial on hidden markov models and selected applications in speech recognition; citation_author=L. Rabiner; citation_volume=77; citation_issue=2; citation_publication_date=1989; citation_pages=257-286; citation_id=CR22
Raffa G, Lee J, Nachman L, Song J (2010) Don’t slow me down: bringing energy efficiency to continuous gesture recognition. In: Proceedings of the international symposium on wearable computers (ISWC’10)
Schultz T (2002) GlobalPhone: a multilingual speech and text database developed at Karlsruhe University. In: Proceedings of the international conference on spoken language processing (ICSLP’02)
Soltau H, Metze F, Fügen C, Waibel A (2001) A one-pass decoder based on polymorphic linguistic context assignment. In: IEEE workshop on automatic speech recognition and understanding (ASRU ’01)
citation_journal_title=IEEE Pervasive Comput; citation_title=Wearable activity tracking in car manufacturing; citation_author=T Stiefmeier, D Roggen, G Tröster, G Ogris, P Lukowicz; citation_volume=7; citation_issue=2; citation_publication_date=2008; citation_pages=42; citation_doi=10.1109/MPRV.2008.40; citation_id=CR26
Stolcke A (2002) Srilm—an extensible language modeling toolkit. In: International conference on spoken language processing
Tamaki E, Miyaki T, Rekimoto J (2009) Brainy hand: an ear-worn hand gesture interaction device. In: Proceedings of the 27th international conference extended abstracts on human factors in computing systems (CHI EA ’09)
Woodman OJ (2007) An introduction to inertial navigation. Technical report. University of Cambridge