Retrieving geometric information from images: the case of hand-drawn diagrams
Tóm tắt
This paper addresses the problem of retrieving meaningful geometric information implied in image data. We outline a general algorithmic scheme to solve the problem in any geometric domain. The scheme, which depends on the domain, may lead to concrete algorithms when the domain is properly and formally specified. Taking plane Euclidean geometry
$${\mathbb {E}}$$
as an example of the domain, we show how to formally specify
$${\mathbb {E}}$$
and how to concretize the scheme to yield algorithms for the retrieval of meaningful geometric information in
$${\mathbb {E}}$$
. For images of hand-drawn diagrams in
$${\mathbb {E}}$$
, we present concrete algorithms to retrieve typical geometric objects and geometric relations, as well as their labels, and demonstrate the feasibility of our algorithms with experiments. An example is presented to illustrate how nontrivial geometric theorems can be generated from retrieved geometric objects and relations and thus how implied geometric knowledge may be discovered automatically from images.
Tài liệu tham khảo
An W, Chen X, Wang D (2016) Searching for geometric theorems using features retrieved from diagrams. In: Kotsireas IS, Rump SM, Yap CK (eds) Mathematical aspects of computer and information sciences, LNCS 9582. Springer, Heidelberg, pp 383–397
ADG (1996–2014) Proceedings of international workshop on automated deduction in geometry. http://dblp.uni-trier.de/db/conf/adg/index.html. Accessed 4 Aug 2016
Balbiani P, Dugat V, Fariñas del Cerro L, Lopez A (1994) Eléments de géométrie mécanique. Hermès, Paris
Ballard DH (1981) Generalizing the Hough transform to detect arbitrary shapes. Pattern Recogn 13(2):111–122
Banerjee B, Chandrasekaran B (2010) A constraint satisfaction framework for executing perceptions and actions in diagrammatic reasoning. J Artif Intell Res 39(1):373–427
Beeson M, Wos L (2014) OTTER proofs in Tarskian geometry. In: Demri S, Kapur D, Weidenbach C (eds) Automated reasoning, LNAI 8562. Springer, Heidelberg, pp 495–510
Belongie S, Malik J, Puzicha J (2002) Shape matching and object recognition using shape contexts. IEEE Trans Pattern Anal Mach Intell 24(4):509–522
Bhatt M, Lee JH, Schultz CPL (2011) CLP(QS): a declarative spatial reasoning framework. In: Egenhofer M, Giudice N, Moratz R, Worboys M (eds) Spatial information theory, LNCS 6899. Springer, Heidelberg, pp 210–230
Buchberger B (1985) Gröbner bases: an algorithmic method in polynomial ideal theory. In: Bose NK (ed) Multidimensional systems theory. D. Reidel, Dordrecht, pp 184–232
Bulko WC (1988) Understanding text with an accompanying diagram. In: Ali M (ed) Proceedings of the 1st international conference on industrial and engineering applications of aritificial intelligence and expert systems, vol 2. ACM Press, New York, pp 894–898
Chen TC, Chung KL (2001) A new randomized algorithm for detecting lines. Real-Time Imaging 7(6):473–481
Chen TC, Chung KL (2001) An efficient randomized algorithm for detecting circles. Comput Vis Image Underst 83(2):172–191
Chen X, Wang D (2012) Management of geometric knowledge in textbooks. Data Knowl Eng 73:43–57
Chen X, Wang D (2013) Formalization and specification of geometric knowledge objects. Math Comput Sci 7(4):439–454
Chen X (2014) Representation and automated transformation of geometric statements. J Syst Sci Complex 27(2):382–412
Chen X, Song D, Wang D (2015) Automated generation of geometric theorems from images of diagrams. Ann Math Artif Intell 74(3–4):333–358
Chou S-C, Gao X-S, Zhang J-Z (1994) Machine proofs in geometry: automated production of readable proofs for geometry theorems. World Scientific, Singapore
Chou S-C, Gao X-S (2001) Automated reasoning in geometry. In: Handbook of automated reasoning, chap 10, vol I. MIT Press, Cambridge, pp 623–665
Cohn AG, Hazarika SM (2001) Qualitative spatial representation and reasoning: an overview. Fundam Inf 46(1–2):1–29
Dehlinger C, Dufourd J-F, Schreck P (2001) Higher-order intuitionistic formalization and proofs in Hilberts elementary geometry. In: Richter-Gebert J, Wang D (eds) Automated deduction in geometry, LNAI 2061. Springer, Heidelberg, pp 306–323
Duda RO, Hart PE (1972) Use of the Hough transformation to detect lines and curves in pictures. Commun ACM 15(1):11–15
Ferguson RW, Forbus KD (1998) Telling juxtapositions: using repetition and alignable difference in diagram understanding. In: Holyoak K, Gentner D, Kokinov B (eds) Advances in analogy research: integration of theory and data from the cognitive, computational, and neural sciences, NBU Series in Cognitive Science. New Bulgarian University, Sofia, pp 109–117
Forbus K, Usher J, Lovett A, Lockwood K, Wetzel J (2011) CogSketch: sketch understanding for cognitive science research and for education. Top Cogn Sci 3(4):648–666
Gelernter H (1995) Realization of a geometry theorem proving machine. In: Feigenbaum EA, Feldman J (eds) Computers and thought. MIT Press, Cambridge, pp 134–152
GeoGebra. http://www.geogebra.org/cms/. Accessed 4 Aug 2016
GEOTHER. http://www-polsys.lip6.fr/wang/GEOTHER/. Accessed 4 Aug 2016
Ghimire D, Lee J (2013) Geometric feature-based facial expression recognition in image sequences using multi-class AdaBoost and support vector machines. Sensors 13(6):7714–7734
Giaquinto M (2007) Visual thinking in mathematics: an epistemological study. Oxford University Press, New York
Glasgow J, Narayanan NH, Chandrasekaran B (1995) Diagrammatic reasoning: cognitive and computational perspectives. MIT Press, Cambridge
Gonthier G (2008) Formal proof—the four-color theorem. Not AMS 55(11):1382–1393
Goodrum AA (2000) Image information retrieval: an overview of current research. Inf Sci Res Spec Issue Inf Sci 3(2):63–66
Gorman JW, Mitchell OR, Kuhl FP (1988) Partial shape recognition using dynamic programming. IEEE Trans Pattern Anal Mach Intell 10(2):257–266
Hales T, Adams M, Bauer G et al (2015) A formal proof of the Kepler conjecture. arXiv preprint arXiv:1501.02155
Hammond T, Davis R (2002) Tahuti: a geometrical sketch recognition system for UML class diagrams. In: Proceeding of the special interest group on computer graphics 2006 courses, Article No. 25. ACM Press, New York
Hong H, Wang D, Winkler F (eds) (1995) Algebraic approaches to geometric reasoning. Special issue of annals of mathematics and artificial intelligence, vol 13. Baltzer, Basel, pp 1–2
Johnson LE (2015) Automated elementary geometry theorem discovery via inductive diagram manipulation. Master thesis, Massachusetts Institute of Technology
Jorge JA, Fonseca MJ (2000) A simple approach to recognise geometric shapes interactively. In: Chhabra AK, Dori D (eds) Graphics recognition recent advances, LNCS 1941. Springer, Heidelberg, pp 266–274
Jung CR, Schramm R (2004) Rectangle detection based on a windowed Hough transform. In: Proceedings of the 17th Braizilian symposium on computer graphics and image processing. IEEE Computer Society, New York, pp 113–120
Kapur D, Mundy JL (eds) (1989) Geometric reasoning. Special issue of artificial intelligence, vol 37. MIT Press, Cambridge, pp 1–3
Lafarge F, Descombes X (2010) Geometric feature extraction by a multi-marked point process. IEEE Trans Pattern Anal Mach Intell 32(9):1597–1609
Li K, Lu X, Ling H, Liu L, Feng T, Tang Z (2013) Detection of overlapped quadrangles in plane geometric figures. In: Proceedings of the 12th international conference on document analysis and recognition. IEEE Computer Society, Washington, pp 260–264
Matas J, Galambos C, Kittler J (2000) Robust detection of lines using the progressive probabilistic Hough transform. Comput Vis Image Underst 78(1):119–137
Materlik D (2003) Using sketch recognition to enhance the human-computer interface of geometry software. Diploma thesis, Free University of Berlin
McCharen JD, Overbeek RA, Wos L (1976) Problems and experiments for and with automated theorem-proving programs. IEEE Trans Comput 25(8):773–782
Montalvo FS (1990) Diagram understanding: the symbolic descriptions behind the scenes. In: Ichikawa T, Jungert E, Korfhage RR (eds) Visual languages and applications, series of languages and information systems. Plenum Press, New York, pp 5–27
Moon H, Chellappa R, Rosenfeld A (2002) Optimal edge-based shape detection. IEEE Trans Image Process 11(11):1209–1227
Mori S, Nishida H, Yamada H (1999) Optical character recognition. Wiley, New York
Nixon MS, Aguado AS (2008) Feature extraction & image processing, 2nd edn. Academic Press, Oxford, pp 87–88
Novak G (1995) Diagrams for solving physical problems. In: Glasgow J, Narayanan NH, Chandrasekaran B (eds) Diagrammatic reasoning: cognitive and computational perspectives. AAAI Press, Palo Alto, pp 753–774
Quaife A (1992) Automated development of fundamental mathematical theories. Kluwer, Dordrecht
Schultz C, Bhatt M (2015) Spatial symmetry driven pruning strategies for efficient declarative spatial reasoning. In: Fabrikant SI, Raubal M, Bertolotto M, Davies C, Freundschuh S, Bell S (eds) Spatial information theory, LNCS 9368. Springer, Heidelberg, pp 331–353
Seo MJ, Hajishirzi H, Farhadi A, Etzioni O (2014) Diagram understanding in geometry questions. In: Proceedings of the 28th AAAI conference on artificial intelligence. AAAI Press, Palo Alto, pp 2831–2838
Song D, Wang D, Chen X (2015) Discovering geometric theorems from scanned and photographed images of diagrams. In: Botana F, Quaresma P (eds) Automated deduction in geometry, LNAI 9201. Springer, Heidelberg, pp 149–165
Song D, Chen X (2016) Automated generation of keywords from images for geometric information search. In: Prodeedings of the 11th international workshop on automated deduction in geometry (ADG 2016) (Strasbourg, France, June 27–29, 2016), pp 172–189
Sonka M, Hlavac V, Boyle R (2014) Image processing, analysis, and machine vision, 4th edn. Cengage Learning, Stamford
Takahashi K (2012) PLCA: a framework for qualitative spatial reasoning based on connection patterns of regions. In: Qualitative spatio-temporal representation and reasoning: trends and future directions, chap 2. IGI Global, Hershey, pp 63–96
Tangelder JWH, Veltkamp RC (2008) A survey of content based 3D shape retrieval methods. Multimed Tools Appl 39(3):441–471
Tesseract-OCR. http://code.google.com/p/tesseract-ocr/. Accessed 4 Aug 2016
Ulgen F, Flavell AC, Akamatsu N (1995) Geometric shape recognition with fuzzy filtered input to a backpropagation neural network. IEICE Trans Inf Syst 78(2):174–183
Wang D (1996) Geometry machines: from AI to SMC. In: Calmet J, Campbell JA, Pfalzgraf J (eds) Artificial intelligence and symbolic mathematical computation, LNCS 1138. Springer, Heidelberg, pp 213–239
Wang D (2001) Elimination methods. Springer, New York
Wang D (2003) Automated generation of diagrams with Maple and Java. In: Joswig M, Takayama N (eds) Algebra, geometry, and software systems. Springer, Heidelberg, pp 277–287
Wu W-T (1994) Mechanical theorem proving in geometries: basic principles (translated from the Chinese by X. Jin and D. Wang). Springer, New York
Yuen HK, Princen J, Illingworth J, Kittler J (1990) Comparative study of Hough transform methods for circle finding. Image Vis Comput 8(1):71–77
Zhang TY, Suen CY (1984) A fast parallel algorithm for thinning digital patterns. Commun ACM 27(3):236–239