Dealing with temporal variation in patent categorization
Tóm tắt
In this paper, we quantify the existence of concept drift in patent data, and examine its impact on classification accuracy. When developing algorithms for classifying incoming patent applications with respect to their category in the International Patent Classification (IPC) hierarchy, a temporal mismatch between training data and incoming documents may deteriorate classification results. We measure the effect of this temporal mismatch and aim to tackle it by optimal selection of training data. To illustrate the various aspects of concept drift on IPC class level, we first perform quantitative analyses on a subset of English abstracts extracted from patent documents in the CLEF-IP 2011 patent corpus. In a series of classification experiments, we then show the impact of temporal variation on the classification accuracy of incoming applications. We further examine what training data selection method, combined with our classification approach yields the best classifier; and how combining different text representations may improve patent classification. We found that using the most recent data is a better strategy than static sampling but that extending a set of recent training data with older documents does not harm classification performance. In addition, we confirm previous findings that using 2-skip-2-grams on top of the bag of unigrams structurally improves patent classification. Our work is an important contribution to the research into concept drift for text classification, and to the practice of classifying incoming patent applications.
Tài liệu tham khảo
Benzineb, K., & Guyot, J. (2011). Automated patent classification. In M. Lupu, K. Mayer, J. Tait, & A. J. Trippe (Eds.), Current challenges in patent information retrieval (Vol. 29, pp. 239–261). Berlin: Springer.
Carmona-Cejudo, J. M., Baena-García, M., Bueno, R. M., Gama, J., & Bifet, A. (2011). Using gnusmail to compare data stream mining methods for on-line email classification. Journal of Machine Learning Research-Proceedings Track, 17, 12–18.
Cohen, A., Bhupatiraju, R., & Hersh, W. (2004). Feature generation, feature selection, classifiers, and conceptual drift for biomedical document triage. In Proceedings of the thirteenth text retrieval conference-TREC.
Dagan, I., Karov, Y., Roth, D. (1997). Mistake-driven learning in text categorization. In Proceedings of 2nd conference on empirical methods in NLP, Providence, pp. 55–63.
D’hondt, E., Verberne, S., Weber, N., Koster, K., & Boves, L. (2012). Using skipgrams and pos-based feature selection for patent classification. Computational Linguistics in the Netherlands Journal, 2, 52–70.
D’hondt, E., Verberne, S., Koster, C., & Boves, L. (2013). Text representations for patent classification. Computational Linguistics, 39(3), 755–775.
Fawcett, T. (2003). “In vivo” spam filtering: A challenge problem for KDD. ACM SIGKDD Explorations Newsletter, 5(2), 140–148.
Forman, G. (2004). A pitfall and solution in multi-class feature selection for text classification. In Proceedings of the twenty-first international conference on machine learning, ICML ’04 (pp. 38–45). New York, NY: ACM.
Forman, G. (2006). Tackling concept drift by temporal inductive transfer. In Proceedings of the 29th annual international ACM SIGIR conference on research and development in information retrieval, SIGIR ’06 (pp. 252–259). New York, NY: ACM.
Frantzi, K., Ananiadou, S., & Tsujii, J. (1998). The C-value/NC-value method of automatic recognition for multi-word terms. In Proceedings of the second European conference on research and advanced technology for digital libraries, ECDL ’98 (pp. 585–604). London: Springer.
Galavotti, L., Sebastiani, F., & Simi, M. (2000). Experiments on the use of feature selection and negative evidence in automated text categorization. In Proceedings of research and advanced technology for digital libraries, 4th European conference, Lisbon, pp. 59–68.
Ja, Gama, Medas, P., Castillo, G., & Rodrigues, P. (2004). Learning with drift detection. In A. Bazzan & S. Labidi (Eds.), Advances in artificial intelligence SBIA 2004, lecture notes in computer science (Vol. 3171, pp. 286–295). Berlin: Springer.
Joachims, T. (1999). Making large-scale support vector machine learning practical. In B. Schölkopf, C. J. C. Burges, & A. J. Smola (Eds.), Advances in Kernel methods (pp. 169–184). Cambridge: MIT Press.
Kelly, M., Hand, D., & Adams, N. (1999). The impact of changing populations on classifier performance. In Proceedings of the fifth ACM SIGKDD international conference on knowledge discovery and data mining, KDD ’99 (pp. 367–371). New York, NY: ACM.
Klimt, B., & Yang, Y. (2004) The enron corpus: A new dataset for email classification research. In Proceedings of the 15th European conference on machine learning, ECML 2004, Vol. 15, p. 217. Berlin: Springer.
Klinkenberg, R. (2004). Learning drifting concepts: Example selection vs. example weighting. Intelligent Data Analysis, 8(3), 281–300.
Koster, C., & Beney, J. (2009). Phrase-based document categorization revisited. In Proceedings of the 2nd international workshop on patent information retrieval, PaIR ’09 (pp. 49–56). New York, NY: ACM.
Koster, C., & Seutter, M., Beney, J. (2003). Multi-classification of patent applications with winnow. In M. Broy, A. V. Zamulin (Eds,). Ershov memorial conference, Lecture Notes in Computer Science, Vol. 2890 (pp. 546–555). Berlin: Springer.
Koster, C., Beney, J., Verberne, S., & Vogel, M. (2011). Phrase-based document categorization. In M. Lupu, K. Mayer, J. Tait, & A. J. Trippe (Eds.), Current Challenges in Patent Information Retrieval (Vol. 29, pp. 263–286). Berlin: Springer.
Koychev, I. (2000). Gradual forgetting for adaptation to concept drift. In Proceedings of ECAI 2000 workshop on current issues in Spatio-Temporal reasoning.
Kuncheva, L. (2004). Classifier ensembles for changing environments. In F. Roli, J. Kittler, & T. Windeatt (Eds.), Multiple classifier systems, lecture notes in computer science (Vol. 3077, pp. 1–15). Berlin: Springer.
Lebanon, G., & Zhao, Y. (2008). Local likelihood modeling of temporal text streams. Proceedings of the 25th international conference on Machine learning—ICML ’08 (pp. 552–559). New York, NY: ACM Press.
Lewis, D. D., Yang, Y., Rose, T. G., & Li, F. (2004). Rcv1: A new benchmark collection for text categorization research. The Journal of Machine Learning Research, 5, 361–397.
Liu, R., & Lu, Y. (2002). Incremental context mining for adaptive document classification. In Proceedings of the eighth ACM SIGKDD international conference on knowledge discovery and data mining (pp. 599–604). New York: ACM.
Ma, C., Lu, B. L., & Utiyama, M. (2009). Incorporating prior knowledge into task decomposition for large-scale patent classification. In W. Yu, H. He, & N. Zhang (Eds.), Advances in neural networks ISNN 2009, lecture notes in computer science (Vol. 5552, pp. 784–793). Berlin: Springer.
Mourão, F., Rocha, L., Araújo, R., Couto, T., Gonçalves, M., & Meira, W. J. (2008). Understanding temporal aspects in document classification. In Proceedings of the 2008 international conference on web search and data mining (WSDM ’08) (pp. 159–170). New York: ACM.
Nanba, H., Fujii, A., Iwayama, M., & Hashimoto, T. (2008). Overview of the patent mining task at the NTCIR-7 workshop. In Proceedings of NTCIR-7 workshop meeting, pp. 325–332.
Nanba, H., Fujii, A., Iwayama, M., & Hashimoto, T. (2010). Overview of the patent mining task at the NTCIR-8 workshop. In Proceedings of NTCIR-7 workshop meeting, pp. 293–302.
Oostdijk, N., Verberne, S.,&Koster, C. (2010). Constructing a broad-coverage lexicon for text mining in the patent domain. In Chair NCC, K. Choukri, B. Maegaard, J. Mariani, J. Odijk, S. Piperidis, M. Rosner, D. Tapias (Eds.), Proceedings of the seventh international conference on language resources and evaluation (LREC’10), Valletta, Malta.
Richter, G., & MacFarlane, A. (2005). The impact of metadata on the accuracy of automated patent classification. World Patent Information, 27(1), 13–26.
Rocha, L., Mourão, F., Mota, H., Salles, T., Gonçalves, M. A., & Meira, W, Jr. (2012). Temporal contexts: Effective text classification in evolving document collections. Information Systems, 38(3), 388–409.
Salles, T., Rocha, L., Pappa, G.L., Mourão, F., Meira, W. Jr, & Gonçalves, M. (2010). Temporally-aware algorithms for document classification. In Proceedings of the 33rd international ACM SIGIR conference on research and development in information retrieval, SIGIR ’10 (pp. 307–314). New York, NY: ACM.
Salton, G., & Buckley, C. (1988). Term-weighting approaches in automatic text retrieval. Information Processing Management, 24(5), 513–523.
SanJuan, E., Dowdall, J., Ibekwe-SanJuan, F., & Rinaldi, F. (2005). A symbolic approach to automatic multiword term structuring. Computer Speech and Language, 19(4), 524–542.
Schlimmer, J., & Granger, R, Jr. (1986). Incremental learning from noisy data. Machine Learning, 1, 317–354.
Scholz, M., & Klinkenberg, R. (2007). Boosting classifiers for drifting concepts. Intelligent Data Analysis, 11(1), 3–28.
Segal, R., & Kephart, J. (1999). Mailcat: An intelligent assistant for organizing e-mail. In Proceedings of the third annual conference on autonomous agents (pp. 276–282). New York, NY: ACM.
Šilić, A., & Dalbelo Bašić, B. (2012). Exploring classification concept drift on a large news text corpus. In Computational linguistics and intelligent text processing, pp. 428–437.
Tsymbal, A. (2004). The problem of concept drift: Definitions and related work. Tech. Rep. TCD-CS-2004-15, Computer Science Department, Trinity College Dublin.
van Halteren, H. (2000). The detection of inconsistency in manually tagged text. In Proceedings of LINC-00.
Verberne, S., Vogel, M., & D’hondt, E. (2010). Patent classification experiments with the linguistic classification system LCS. In Proceedings of the conference on multilingual and multimodal information access evaluation (CLEF 2010), Padua.
Žliobaitė, I. (2009). Learning under concept drift: An overview. Tech. rep.: Vilnius University.